MoVieDrive: Urban Scene Synthesis with Multi-Modal Multi-View Video Diffusion Transformer

1. Problem Statement

Autonomous driving requires a holistic understanding of urban environments, which necessitates multi-modal data (e.g., RGB, depth maps, semantic maps) rather than just RGB videos. While recent video generation models (like SVD or CogVideoX) excel at generating high-quality single-modal videos, they face three critical limitations when applied to autonomous driving:

1. Lack of Multi-Modal Support: Existing driving scene generators primarily focus on RGB video, ignoring crucial modalities like depth and semantics needed for perception and planning.
2. Inefficiency of Multi-Model Pipelines: Current approaches often use separate models to generate different modalities. This increases deployment complexity and fails to leverage complementary cues between modalities (e.g., how depth information can guide semantic segmentation).
3. Multi-View Consistency: Generating long, controllable videos across multiple camera views (multi-view) with strict spatiotemporal consistency is difficult for standard diffusion models.

MoVieDrive aims to solve these issues by proposing a unified framework capable of generating multi-modal, multi-view driving scene videos simultaneously with high controllability and fidelity.

---

2. Methodology

The core of MoVieDrive is a Unified Diffusion Transformer Model that integrates diverse conditioning inputs and a novel architecture designed for multi-modal learning.

A. Conditioning Inputs Encoding

The model accepts diverse conditions to guide scene generation:

• Text Conditions: Includes Camera Prompts (intrinsic/extrinsic parameters encoded via Fourier embedding and MLP) and Text Prompts (scene descriptions encoded via a frozen text encoder). These guide holistic scene generation.
• Layout Conditions: Includes Box Maps (2D projections of 3D bounding boxes), Road Maps (lane dividers/boundaries), and 3D Occupancy Maps. These provide fine-grained structural control.
  • Innovation: Instead of using multiple encoders, the authors propose a Unified Layout Encoder using causal convolutional layers (causal ResNet blocks) to fuse these conditions into a single embedding before feeding them to the diffusion model.
• Context Reference Conditions: Optional initial frames (from a shared 3D VAE) used for future scene prediction (world modeling).

B. Multi-Modal Multi-View Diffusion Transformer

The architecture decomposes the learning process into Modal-Shared and Modal-Specific components to balance commonality and uniqueness across modalities:

1. Modal-Shared Components:
   • Temporal Layers: Uses 3D full attention (from CogVideoX) to learn temporal consistency across frames and inject text conditions via cross-attention.
   • Multi-View Spatiotemporal Blocks: Designed to enforce consistency across camera views. These blocks include:
     • 3D Spatial Embedding: Encodes 3D occupancy locations into spatial embeddings.
     • 3D Spatial Attention: Learns 3D structure across all camera views.
     • Spatiotemporal Attention: Captures full spatiotemporal information for the multi-view scene.
2. Modal-Specific Components:
   • Cross-Modal Interaction Layers: These layers allow modalities to interact. The query comes from the current modality's latent, while keys and values are derived from the concatenated latents of other modalities. This enables the model to learn modal-specific content while maintaining cross-modal alignment.
   • Projection Heads: Separate linear layers for each modality to estimate the noise and generate clean latents.

C. Training and Inference

• Training: Uses the DDPM noise scheduler with a multi-modal loss function. The model is trained on the nuScenes dataset, where depth and semantic maps are synthesized using off-the-shelf models (Depth-Anything-V2, Mask2Former) since ground-truth multi-modal data is scarce.
• Inference: Uses DDIM sampling and Classifier-Free Guidance (CFG) to balance diversity and condition adherence.

---

3. Key Contributions

1. Unified Framework: Proposes the first unified diffusion transformer model for multi-modal multi-view autonomous driving scene generation, eliminating the need for multiple separate models.
2. Novel Architecture: Introduces a hybrid architecture with Modal-Shared (temporal and multi-view spatiotemporal blocks) and Modal-Specific (cross-modal interaction) layers, effectively leveraging complementary cues between RGB, depth, and semantics.
3. Diverse Conditioning: Develops a Unified Layout Encoder to fuse box, road, and occupancy maps, and integrates text/camera prompts for high controllability.
4. Long Video Generation: Demonstrates the ability to generate long videos (up to 18s+) without reference frames, maintaining scene consistency.

---

4. Experimental Results

The approach was evaluated on the nuScenes and Waymo datasets.

• Video Fidelity (FVD): MoVieDrive achieved an FVD of 46.8 on nuScenes, significantly outperforming state-of-the-art (SOTA) methods like MagicDrive-V2 (112.7) and DriveDreamer-2 (55.7). It also showed a ~22% improvement over a baseline of CogVideoX + SyntheOcc.
• Controllability:
  • 3D Object Detection (mAP): Achieved 22.7, surpassing MagicDrive-V2 (11.5).
  • BEV Segmentation (mIoU): Achieved 35.8, outperforming MagicDrive-V2 (17.4).
• Multi-Modal Quality:
  • Depth: Achieved an AbsRel of 0.110.
  • Semantic: Achieved an mIoU of 37.5.
  • Ablation: The unified multi-modal approach outperformed pipelines that generated RGB first and then used separate models for depth/semantic, proving the benefit of joint training.
• Generalization: On the Waymo dataset, MoVieDrive achieved an FVD of 61.6, outperforming baselines. It successfully generated diverse scenes under different weather (snow, rain) and time conditions (night) via text prompts.

---

5. Significance

MoVieDrive represents a significant leap forward in autonomous driving simulation and data synthesis:

• Efficiency: By unifying multi-modal generation into a single model, it reduces deployment complexity and computational overhead compared to cascaded multi-model pipelines.
• Holistic Understanding: The ability to generate synchronized RGB, depth, and semantic videos simultaneously allows for better training of perception models and safer testing of autonomous driving algorithms in long-tail scenarios.
• Controllability: The framework offers unprecedented control over scene structure (via layout maps) and content (via text), enabling the synthesis of rare or dangerous driving scenarios that are difficult to capture in the real world.
• Scalability: The modular design allows for the inclusion of additional modalities (e.g., LiDAR, thermal) without re-architecting the entire system.