InstaDrive: Instance-Aware Driving World Models for Realistic and Consistent Video Generation

Technical Summary: InstaDrive

Problem Statement
Autonomous driving systems rely heavily on high-quality, large-scale multi-view driving videos for training downstream tasks such as perception, object tracking, and planning. While "world models" offer a cost-effective alternative to real-world data collection by generating realistic driving scenarios, existing approaches struggle with two critical limitations:

1. Instance-Level Temporal Inconsistency: Current models often fail to maintain stable attributes (e.g., color, texture, identity) of specific objects across frames, leading to flickering or identity switches.
2. Spatial Geometric Fidelity: Generated videos frequently exhibit misalignment with control signals (e.g., bounding boxes deviating from their intended positions) and fail to correctly model occlusion hierarchies (e.g., distant objects incorrectly occluding closer ones).

These deficiencies hinder the applicability of synthetic data in rigorous autonomous driving tasks, which require precise spatial context and temporally stable object tracking.

Methodology
The authors propose InstaDrive, a driving world model built upon the OpenSora V1.1 framework (utilizing a Variational Auto-Encoder, T5 text encoder, and Spatial-Temporal Diffusion Transformer). InstaDrive introduces two novel, instance-aware modules to address the aforementioned challenges:

1. Instance Flow Guider (IFG):

   • Objective: To enforce temporal consistency by tracking and propagating instance features across frames.
   • Mechanism: The module utilizes tracking IDs to capture motion trajectories (instance flow) of surrounding objects. It calculates a 3D displacement vector (offset) between the current frame and the most recent visible frame of an instance.
   • Implementation: This offset is rendered into a "motion map" within the latent space, where x, y, and z offsets are mapped to RGB channels. This motion map is injected into the Diffusion Transformer (DiT) backbone via ControlNet.
   • Effect: The temporal attention layers use this motion condition to retrieve semantic features (category, color, texture) from previous frames and propagate them forward, ensuring that an object's identity remains stable even when reappearing after occlusion.

2. Spatial Geometric Aligner (SGA):

   • Objective: To ensure precise spatial localization and correct occlusion reasoning.
   • Mechanism: The module transforms 3D bounding boxes from the world coordinate system into the camera's First-Person View (FPV) using intrinsic and extrinsic parameters.
   • Implementation:
     • Projection: 3D box corners are projected to 2D pixel coordinates to create explicit spatial constraints in the RGB domain.
     • Depth Ordering: To model occlusion, the system calculates the depth of each corner point relative to the camera's optical center. These depth values are encoded via Fourier embedding and an MLP to create a depth order representation.
     • Fusion: The projected box conditions and depth order representations are fused using cross-attention mechanisms before being injected into the generation pipeline.
   • Effect: This ensures that generated objects adhere strictly to bounding box control signals and that closer objects correctly occlude farther ones, maintaining geometric consistency.

Key Contributions

• Instance Flow Guider: A lightweight module that extracts and propagates instance features across frames, significantly improving temporal consistency without relying on external feature extractors.
• Spatial Geometric Aligner: A mechanism that integrates 3D bounding box projections and explicit depth ordering to align instances with pixel space and resolve occlusion hierarchies.
• Procedural Long-Tail Simulation: A pipeline leveraging CARLA's autopilot to stochastically simulate rare, safety-critical driving scenarios (e.g., sudden braking, cut-ins) across diverse maps, providing rigorous control conditions for video synthesis.
• Unified Framework: An end-to-end approach that combines these modules with ControlNet to handle multiple control signals (road maps, 3D boxes, camera poses) efficiently.

Results
Evaluated on the nuScenes dataset, InstaDrive demonstrates state-of-the-art (SOTA) performance:

• Video Quality: Achieves a FID of 3.96 and FVD of 38.06, outperforming existing methods like MagicDrive-V2, DriveDreamer2, and Panacea.
• Temporal Consistency: In Multi-Object Tracking (MOT) tasks using the StreamPETR model, InstaDrive-generated data reduces ID Switches (IDS) to 532 (compared to 593 for DriveDreamer2 and 687 for real-world baselines), indicating superior instance stability.
• Spatial Fidelity: In perception tasks, models trained on InstaDrive-generated data achieve a nuScenes Detection Score (NDS) of 40.51 on the validation set (86.38% relative performance to real data). When used for data augmentation (mixing synthetic and real data), the perception model's NDS improves by 3.6 points over real-data-only training.
• Qualitative Analysis: Visual comparisons confirm that InstaDrive eliminates color shifts, adheres to bounding box constraints, and correctly renders occlusion relationships, unlike prior works which exhibit deviations in these areas.

Significance
The paper claims that InstaDrive establishes a fine-grained and robust world model that effectively bridges the gap between synthetic generation and real-world autonomous driving requirements. By explicitly addressing instance-level temporal consistency and spatial geometric fidelity, the model not only generates high-quality videos but also produces synthetic data that is directly applicable and beneficial for training downstream perception, tracking, and planning tasks. Furthermore, the integration with CARLA for procedural long-tail scenario generation highlights the system's potential for safety-critical evaluation in autonomous systems.