💻 computer science

FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

FrozenDrive is a parameter-free, zero-shot framework that leverages a frozen diffusion model with knowledge-preserving spatio-temporal attention to generate globally consistent, text-guided multi-view driving scenes, significantly enhancing autonomous driving performance in adverse and rare conditions without requiring fine-tuning.

Original authors: Yuhwan Jeong, Hyeonseong Kim, Daehyun We, Seonkyu Song, Jinnyeong Yang, Hyun-Kurl Jang, Youngho Yoon, Kuk-Jin Yoon

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhwan Jeong, Hyeonseong Kim, Daehyun We, Seonkyu Song, Jinnyeong Yang, Hyun-Kurl Jang, Youngho Yoon, Kuk-Jin Yoon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car. To do this, you need to show it millions of different driving scenarios: sunny days, rainy nights, snowy streets, and rare situations like a bus getting stuck in a construction zone.

The problem is that collecting real footage of these rare or dangerous moments is incredibly hard, expensive, and sometimes impossible. So, researchers try to use AI to invent (synthesize) these scenes.

Here is the story of FrozenDrive, a new method for creating these fake driving scenes, explained simply.

The Problem: The "Over-Training" Trap

Think of a powerful AI model (like a famous artist) who has already learned how to paint beautiful, realistic pictures of the world. This artist knows how to paint snow, rain, and night perfectly because they have seen them before.

However, when researchers tried to teach this artist to paint specific driving scenes (like a car on a specific road), they usually had to make the artist "re-learn" everything by fine-tuning their brain.

The Mistake: It's like forcing a master painter to take a crash course in driving. In the process, they might forget how to paint realistic snow or night skies because they are so focused on the new driving rules.
The Result: The new AI can draw a car on a road, but if you ask it to "make it snow," the snow looks fake, or the car looks like a blob. It has lost its original artistic knowledge.

The Solution: The "Frozen" Artist

The FrozenDrive team came up with a clever idea: Don't retrain the artist's brain at all.

Instead of changing the artist's internal knowledge, they built a set of transparent overlays (like stencils or glasses) that sit on top of the artist's work.

The Frozen Backbone: They take the pre-trained AI model and "freeze" it completely. They don't change a single number inside its brain. This preserves all the original knowledge about what snow, rain, and cars actually look like.
The Stencils (ControlNet): They feed the artist a "map" (like a blueprint of the road, where cars should be, and how deep the street is) and a text prompt (like "Make it a rainy night").
The Magic Glasses (Attention): This is the secret sauce. They put special "glasses" on the AI that force it to look at all the camera angles at once and check the previous video frame.
- Multi-View: It ensures that if a car is on the left in the front camera, it's also on the left in the side camera. No more "hallucinations" where the car disappears or changes shape.
- Time-Travel: It ensures the car moves smoothly from one second to the next, not jumping around.

Why "Frozen" is Better

Usually, to make an AI consistent, you have to tweak its brain. But tweaking the brain makes it forget its original training.

FrozenDrive keeps the brain frozen. It only changes how the AI looks at the information.
The Analogy: Imagine a chef who knows how to cook a perfect steak. If you ask them to cook a steak for a specific diet, you don't need to retrain their whole brain on cooking. You just give them a specific recipe card (the text prompt) and a special pan (the driving map). The chef still knows how to cook a steak perfectly, but now they can do it exactly how you asked, even if you asked for a "snowy steak" (a scenario they've never cooked before).

The Results: Better Robots, Safer Roads

Because the AI didn't forget how to paint realistic weather and objects:

Zero-Shot Magic: You can type "Heavy snow" or "Night rain," and the AI generates a realistic scene, even if it was never explicitly trained on snow or rain data. It just uses its general knowledge of snow and rain and applies it to the driving map.
Rare Objects: It gets better at drawing rare things (like a bicycle or a construction vehicle) because it didn't get "confused" by over-training on common things like cars.
Training Real Robots: When they used these fake scenes to train a real self-driving car system (called UniAD and SparseDrive), the robot drove much better in bad weather. It was less confused by rain and night than robots trained on other types of fake data.

Summary

FrozenDrive is like giving a master painter a set of transparent stencils and a text prompt, rather than forcing them to go back to art school. This way, the painter keeps all their original skills (knowing what snow and cars look like) but can still create perfectly consistent, multi-camera driving scenes for any weather condition, even ones they've never seen before. This helps train safer self-driving cars without needing to collect dangerous real-world footage.

Technical Summary: FrozenDrive

Problem Statement

The training of autonomous driving (AD) models is constrained by the high cost and difficulty of collecting large-scale, accurately labeled datasets, particularly for adverse weather conditions (e.g., rain, snow, night) and rare, long-tail scenarios. While synthetic data generation using diffusion models offers a solution, existing approaches face significant limitations:

Knowledge Erosion: Enforcing multi-view and temporal consistency typically requires fine-tuning the backbone or adding learnable layers. This process erodes the rich pre-trained knowledge of the diffusion model, weakening its text-to-image alignment and zero-shot capabilities.
Distributional Bias: Models trained on standard benchmarks often fail to generalize to unseen or adverse conditions because they are heavily bounded by the training distribution.
Fidelity Imbalance: Current methods tend to prioritize frequent classes, leading to poor generation fidelity for rare objects (e.g., bicycles, construction vehicles).

Methodology: FrozenDrive

The authors propose FrozenDrive, a controllable generative framework that synthesizes zero-shot, text-guided, multi-view driving scenes while preserving the pre-trained knowledge of a frozen diffusion backbone.

Core Architecture

Frozen Backbone: Unlike prior works that fine-tune the diffusion model (e.g., Stable Diffusion), FrozenDrive keeps the entire diffusion backbone frozen and parameter-free. This preserves the model's original text alignment and generative priors.
ControlNet Conditioning: The framework utilizes a ControlNet-based pipeline to inject structured driving-stack signals without modifying the backbone weights. Inputs include:
- HD map layouts and 3D bounding boxes.
- Per-view camera indicators.
- Depth maps derived from occupancy grids.
- Relative pose to the previous frame.
- Text prompts for scene description.

Key Technical Innovations

Knowledge-Preserving Spatio-Temporal Attention: To achieve consistency without new trainable parameters, the authors replace standard self-attention layers with two complementary mechanisms that reshape the input context:
- Multi-View Inflated Self-Attention (MISA): Concatenates latent features from all camera views into a single sequence. This allows the frozen attention mechanism to attend across views, enforcing cross-view geometric and semantic consistency in a single pass.
- Temporal Reference Self-Attention (TRSA): Expands the input context to include latent features from the previous frame as a reference. This enables the current frame to attend to temporal cues (appearance, geometry) from the past, ensuring video-style coherence without learning new temporal weights.
- Implementation: These mechanisms operate within the frozen decoder. The projection weights ( $W^Q, W^K, W^V$ ) remain untouched; only the input token sequences are modified to include cross-view and temporal references.
Object-Focused Constraint (Object-Presence Ratio Loss): To address the long-tail distribution problem, the authors introduce a per-pixel weighting loss. This loss upweights underrepresented categories (e.g., bicycles, motorcycles) based on their inverse frequency in the training data. This shifts the learning focus toward rare objects, improving their generation fidelity and downstream detection performance.

Key Contributions

Parameter-Free Consistency: The introduction of a framework that enforces multi-view and temporal consistency on a frozen diffusion backbone via input reshaping (MISA and TRSA) rather than parameter updates, thereby preventing knowledge forgetting.
Zero-Shot Text Guidance: The ability to synthesize diverse, globally coherent driving scenes from text prompts, including unseen combinations of weather, time of day, and rare environments, without task-specific fine-tuning.
Improved Downstream Robustness: The demonstration that data augmented with FrozenDrive significantly improves the perception and planning performance of AD models (specifically UniAD and SparseDrive), particularly in adverse conditions like rain and night.

Experimental Results

The method was evaluated on the nuScenes dataset.

Generation Quality: FrozenDrive achieved state-of-the-art performance among generative baselines (including MagicDrive, Panacea, DriveArena, and MagicDrive-V2) in terms of downstream perception metrics (3D Object Detection mAP/NDS and BEV Segmentation mIoU). It also achieved the lowest FVD (Fréchet Video Distance) among SD-based methods, indicating high temporal and spatial consistency.
Adverse Condition Performance: When used to augment training data for night and rain scenarios, models trained with FrozenDrive data outperformed those trained on real data, rule-based filters, and other generative baselines. For instance, in night conditions, the method achieved a 3DOD mAP of 18.15 compared to 8.89 for DriveArena.
Zero-Shot Capability: The model successfully generated plausible snowy scenes—a condition absent from the training data—whereas fine-tuned baselines failed to reflect the text prompt accurately.
Object Fidelity: The object-presence ratio loss significantly improved the detection AP for rare categories (e.g., +1.07% for motorcycles, +1.00% for bicycles) compared to models without this loss.

Significance and Claims

The paper claims that FrozenDrive addresses the fundamental trade-off between controllability and knowledge preservation in generative driving scene synthesis. By strictly freezing the backbone and utilizing knowledge-preserving attention mechanisms, the method retains the robust zero-shot text alignment of the pre-trained model while achieving high-fidelity multi-view consistency.

The authors emphasize that this approach enables scenario-targeted data generation that strengthens downstream AD robustness. Unlike previous methods that degrade performance in long-tail or adverse conditions due to knowledge erosion, FrozenDrive produces synthetic data that effectively generalizes to unseen weather and rare object configurations, offering a scalable solution for data scarcity in autonomous driving.