💻 computer science

ConsisDrive: Identity-Preserving Driving World Models for Video Generation by Instance Mask

ConsisDrive is an identity-preserving driving world model that utilizes instance-masked attention and instance-masked loss to ensure temporal consistency of objects, achieving state-of-the-art video generation quality and improved performance in autonomous driving tasks.

Original authors: Zhuoran Yang, Yanyong Zhang

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhuoran Yang, Yanyong Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie of a busy city street. In a high-quality movie, if a red sports car drives past a coffee shop, it stays a red sports car the whole time. But in some "glitchy" AI-generated videos, that car might suddenly turn into a blue truck halfway through the scene, or its color might flicker like a broken neon sign.

This paper introduces ConsisDrive, a new AI "world model" designed to fix these glitches specifically for self-driving car simulations.

Here is the breakdown of how they fixed it, using everyday analogies:

1. The Problem: "The Identity Crisis"

Current AI video generators are like artists who have a very short memory. They draw one frame, then they draw the next, but they forget exactly what they drew a second ago.

Category Drift: A bus slowly morphs into a truck (like a shapeshifter).
Color Shift: A car changes from black to red mid-drive (like a chameleon).
Foreground Dilution: The AI focuses so much on the big, easy things (like the sky and buildings) that it "forgets" to pay attention to small, important things (like a pedestrian crossing the street), making them look blurry or ghostly.

2. The Solution: Two "Smart Filters"

To fix this, the researchers added two clever mechanisms to the AI's brain:

A. Instance-Masked Attention (The "VIP Guest List")

Imagine you are hosting a massive, noisy dinner party with 50 guests. In a normal AI, everyone is shouting at everyone else at once—it’s chaos, and information gets mixed up. This is why a car's color might "bleed" into the sidewalk.

ConsisDrive implements a "VIP Guest List" (the Instance-Masked Attention).

The Identity Tag: Every object (a car, a person, a bike) gets a specific ID card. The AI is told: "When you are drawing this specific car, you are only allowed to look at this car's ID card to remember its color and shape." This prevents the "bus-to-truck" shapeshifting.
The Trajectory Path: The AI also gets a "map" of where that object is moving. It’s like giving the car a GPS tracker. The AI is told: "Follow this specific path. Only look at the pixels that belong to this car as it moves from point A to point B." This prevents the color from flickering.

B. Instance-Masked Loss (The "Strict Teacher")

When an AI learns, it’s like a student taking a test. Most AI models use a "uniform" grading system: they get points for getting the sky right and points for getting the car right. But because the sky is huge and the car is small, the AI spends all its energy on the sky and ignores the car.

ConsisDrive uses a "Strict Teacher" (the Instance-Masked Loss).

This teacher says: "I don't care if the clouds look perfect; if that pedestrian is blurry, you fail the test!"
It uses a "probabilistic" approach, meaning it constantly shifts its focus back and forth between the big background and the small, important objects. This ensures the background looks natural, but the "stars of the show" (the cars and people) are rendered with crystal clarity.

3. Why does this matter?

If we want to train self-driving cars in a virtual world (which is much cheaper and safer than driving real cars on real streets), the virtual world must be perfect.

If the AI simulator accidentally turns a red stop sign into a green light, or makes a pedestrian disappear for a split second, a self-driving car trained in that world will learn dangerous habits. ConsisDrive makes these virtual worlds so consistent and realistic that they can be used to train the "brains" of real autonomous vehicles, making them safer for everyone on the road.

Technical Summary: ConsisDrive: Identity-Preserving Driving World Models

ConsisDrive is a novel driving world model designed to address the critical issue of identity drift in generative driving videos. While current diffusion-based world models can synthesize realistic scenes, they often fail to maintain the consistency of specific objects (e.g., a car changing color or a bus morphing into a truck) across temporal sequences. This inconsistency limits the utility of synthetic data for training downstream autonomous driving tasks like perception, tracking, and planning.

1. The Problem: Identity Drift

The authors identify three root causes for the lack of temporal consistency in existing models:

Lack of Explicit Identity Conditions: Models often lack specific signals (category, size, ID) to anchor an object's identity over time.
Non-Instance-Aware Attention: Standard 3D attention mechanisms allow "information leakage," where visual tokens from different objects interact, causing attributes like color to bleed between instances.
Supervision Dilution: Standard training objectives apply uniform loss across the entire frame. Because background pixels (sky, buildings) dominate the scene, the model lacks sufficient gradient signal to focus on fine-grained details of small foreground objects (e.g., pedestrians).

2. Methodology

To solve these issues, ConsisDrive introduces two primary technical components that inject "instance awareness" into the generation process:

A. Instance-Masked Attention (IMA)
This module modifies the attention mechanism using two distinct types of masks derived from 3D bounding box projections:

Instance Identity Mask: This restricts visual tokens so they can only attend to the identity embeddings (category, size, and tracking ID) of their corresponding instance. This prevents semantic shifts (e.g., category changes).
Instance Trajectory Mask: This ensures that tokens belonging to the same object across different frames attend only to each other, while blocking interactions with other instances. This enables the stable propagation of visual attributes like texture and color along an object's path.

B. Instance-Masked Loss (IML)
To combat supervision dilution, the authors propose a training objective that emphasizes foreground regions:

Probabilistic Dynamic Masking: Instead of applying a constant foreground loss (which might cause the model to neglect the background), the model uses a stochastic strategy. With a probability $\alpha$ , it applies a loss focused on foreground instances; otherwise, it applies a standard global reconstruction loss. This balances foreground fidelity with overall scene realism.

3. Key Contributions

Instance-Aware Architecture: A framework that integrates per-instance identity and trajectory constraints directly into the Diffusion Transformer (DiT) backbone.
Dual-Masking Mechanism: The introduction of both Identity and Trajectory masks to decouple instance interactions.
Adaptive Supervision: A probabilistic loss function that prevents background noise from overwhelming foreground learning.

4. Results and Evaluation

The model was evaluated on the nuScenes dataset across several dimensions:

Video Generation Quality: ConsisDrive achieved State-of-the-Art (SOTA) performance, reaching an FID of 3.88 and an FVD of 37.23, outperforming previous models like MagicDrive-V2 and DriveDreamer2.
Perception Tasks (Detection): When used for data augmentation, training a perception model (StreamPETR) on ConsisDrive data led to a significant boost in the nuScenes Detection Score (NDS), reaching 54.6 (a 7.7-point improvement over real data alone).
Multi-Object Tracking (MOT): The model demonstrated superior temporal consistency, significantly reducing ID Switches (IDS) compared to baselines, proving that object identities are successfully propagated across frames.
Ablation Studies: Quantitative results confirmed that removing the Identity Mask leads to category shifts, removing the Trajectory Mask leads to color shifts, and removing the Masked Loss degrades the detection of small objects.

5. Significance

ConsisDrive represents a significant step forward in the development of high-fidelity synthetic training data for autonomous driving. By solving the identity drift problem, it ensures that generated videos are not just visually pleasing, but also semantically and temporally accurate enough to be used for training the "brains" of self-driving cars (perception, tracking, and planning modules). This reduces the reliance on expensive, real-world data collection while providing a more reliable simulation environment.