🤖 AI

OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation

This paper introduces DRIVE-CHOREO, an LLM-choreographed multi-agent world model that unifies heterogeneous driving controls and multi-view geometry through a shared latent token sequence and co-compression strategy, achieving state-of-the-art consistency and demonstrating significant downstream utility for autonomous driving tasks.

Original authors: Zijie Meng, Yufei Liu, Chengqian Ma, Zhiyu Li, Jiyuan Liu, Wenhua Nie, Bingcai Wei, Shuqin Chen, Weichen Xu, Jiquan Yuan, Miao Zhang

Published 2026-06-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zijie Meng, Yufei Liu, Chengqian Ma, Zhiyu Li, Jiyuan Liu, Wenhua Nie, Bingcai Wei, Shuqin Chen, Weichen Xu, Jiquan Yuan, Miao Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to drive a car by showing it videos of the road. The robot needs to understand three things at once: what the driver is saying (language), where the cars and roads are (geometry), and what the cameras actually see (pixels).

For a long time, AI researchers have struggled to get these three things to work together smoothly. It's like trying to conduct an orchestra where the violinists are reading sheet music, the drummers are listening to a podcast, and the singers are improvising without a conductor. The result is often a messy performance where the car might suddenly teleport, the sky might change color between cameras, or the robot ignores the traffic lights.

The paper introduces OMNIDRIVE, a new system that acts like a master film director to fix this mess. Here is how it works, broken down into simple concepts:

1. The Problem: The "Disconnected" Crew

Current AI models for driving videos usually handle the "what" (language) and the "where" (maps) separately from the "what it looks like" (video).

The Issue: They try to glue these separate parts together after the video is mostly made. This is like trying to paste a map onto a movie screen after the movie has already started playing. The result is often "drift," where the left camera sees a tree, but the right camera sees a building, or the car jumps weirdly.

2. The Solution: The "Three-Agent" Director

OMNIDRIVE replaces the messy glue with a team of three specialized AI agents (powered by a large language model) that work together before and during the video creation. Think of them as a film production crew:

The Architect (The Screenwriter): You give it a simple prompt like "Drive through a rainy city at night." The Architect translates this into a strict, structured script called a WORLDSCRIPT. It lists exactly what the weather is, where the ego-car (your car) is going, and where other cars are.
The Cartographer (The Set Designer): This agent takes the script and draws a "blueprint" or a sparse map. It doesn't draw the whole video; it just draws the lines of the road, the lanes, and the boxes around other cars. It turns the text into a visual map that matches the camera angles.
The Auditor (The Quality Control Inspector): While the video is being made, this agent watches the different camera views. If the front camera sees a red car but the side camera sees a blue one, the Auditor shouts, "Hey, that doesn't match!" and tells the system to fix it immediately.

3. The Secret Sauce: "Latent Co-Compression"

This is the paper's most technical but crucial innovation. Usually, AI looks at each of the six cameras on a car separately, like looking through six different windows one by one.

OMNIDRIVE does something different. It takes the video from all six cameras and the "blueprint" from the Cartographer, and stitches them all together into one single, long strip of data before the AI even starts generating the video.

The Analogy: Imagine you have six different puzzle pieces. Instead of trying to fit them together after you've painted them, you tape them all onto one big board first. Then, you paint the whole board at once. Because they are physically connected on the board, the paint (the video) naturally flows from one camera to the next without gaps or errors.
The Result: The AI "sees" the 3D world as a single, unified object rather than six separate 2D pictures. This ensures that if a car is on the left, it is also on the right, exactly where it should be.

4. The Outcome: A Perfectly Aligned Movie

Because the language, the map, and the video are all "choreographed" together from the very first step:

No More Ghosting: Cars don't disappear or teleport between camera views.
Perfect Consistency: The lighting and shadows match perfectly across all six cameras.
Real-World Use: The paper shows that if you train a self-driving car's brain only on videos made by OMNIDRIVE, the car actually drives better in the real world than cars trained on real footage alone.

Summary

OMNIDRIVE is like a super-conductor for self-driving video generation. Instead of letting the language, the maps, and the cameras argue with each other, it forces them to sit at the same table, speak the same language, and move in perfect sync. The result is a driving simulation that is so realistic and consistent that it can actually teach real cars how to drive safely.

Technical Summary: OMNIDRIVE

1. Problem Statement

Autonomous driving world models face two persistent, unresolved tensions that hinder the generation of high-fidelity, controllable multi-view videos:

Heterogeneous Control Injection: Existing systems struggle to unify fundamentally different control modalities. Geometric cues (HD-maps, 3D boxes, camera poses) are spatially anchored to the pixel grid, while semantic cues (free-form text, weather, style) are globally diffuse. Current approaches typically graft separate branches (e.g., ControlNet for geometry, cross-attention for text) that remain disconnected until the diffusion process attempts to reconcile them post-hoc.
Post-Hoc Cross-View Fusion: Most methods compress each camera view independently using separate 3D VAEs and only fuse them later via cross-view attention. This results in camera-local latents that lack global 3D geometric awareness, leading to persistent issues such as cross-view drift, photometric flicker, and object teleportation, even when individual view quality metrics (FID) are competitive.

The authors argue that the root cause of both failures is the absence of a shared symbolic interlingua that aligns language, geometry, and pixels at the latent-token level.

2. Methodology

OMNIDRIVE reframes controllable multi-view video generation as a latent choreography problem, orchestrated by a multi-agent system and unified through a novel compression strategy.

A. LLM-Choreographed Multi-Agent System

Instead of a monolithic prompt-to-encoder pipeline, OMNIDRIVE employs three specialized Qwen2.5-VL agents that collectively author a single, position-aware token sequence:

ARCHITECT: Parses free-form user prompts into a structured WORLDSCRIPT (JSON). This script encodes global state (weather, time), ego trajectory, agent lists (classes, positions, behaviors), and road topology. The output is embedded as semantic tokens ( $C_{sem}$ ).
CARTOGRAPHER: Grounds the WORLDSCRIPT into spatially anchored geometry. It programmatically renders sparse layout images (rasterized HD-maps, projected 3D boxes, ego ribbons) for each camera and time step. These layouts are encoded into geometric tokens ( $C_{geo}$ ) that share the exact positional indices as the visual tokens.
AUDITOR: During training, this agent consumes decoded multi-view crops and the WORLDSCRIPT to generate structured critiques (consistency scores and failure mode tags) between camera pairs. These are embedded as review tokens ( $C_{rev}$ ) to provide auxiliary supervision.

B. Latent Co-Compression

To solve the cross-view consistency issue, OMNIDRIVE introduces Latent Co-Compression, which replaces independent per-camera encoding with a unified process:

View-Time Permutation ( $\Pi$ ): The six-camera $\times$ time video cube is flattened into a pseudo-temporal stream where the temporal dimension is defined as $\tilde{t} = (n-1)T + t$ . This permutation is applied identically to both the RGB video and the CARTOGRAPHER's geometric layout images.
Unified 3D VAE: A single 3D VAE (pretrained on generic videos) encodes the permuted tensor. Because the permutation is metric-preserving and orthogonal, it maintains the Lipschitz constant of the encoder.
View-Block-Aware Padding (VBP): To prevent the convolutional receptive field from straddling different camera views (which would mix unrelated spatial contexts), a learnable masking operator is applied. This ensures that temporal kernels only aggregate information from the same instant across views, reducing inter-camera variance.
Shared Noise Endpoint: For views captured at the same physical instant, the noise endpoint ( $z_1$ ) in the flow-matching formulation is shared, further reducing photometric variance.

C. Choreographed Generation (MM-DiT)

The co-compressed latent is patchified and concatenated with the agent-authored tokens ( $C_{sem}, C_{geo}, C_{tmp}, C_{rev}$ ) into a unified sequence $S$ .

Positional Schema: Geometric tokens ( $C_{geo}$ ) share the same 3D positional indices $(\tilde{t}, i, j)$ as the visual tokens ( $X$ ), enforcing pixel-level geometric grounding. Semantic tokens are offset, and temporal tokens are sinusoidal.
Biased Attention: A controllable bias is applied to the multi-modal attention mechanism, specifically tightening the attention between visual tokens and geometric tokens ( $X$ and $C_{geo}$ ) via a scalar gain $\gamma_{geo}$ . This collapses multi-branch cross-attention into a single self-attention mechanism with explicit positional alignment.

D. Training and Inference

Loss Function: The model is trained using Conditional Flow Matching ( $L_{CFM}$ ) combined with an equivariance-aware smoothness term ( $L_{sm}$ ) to penalize non-physical latent jumps, and an AUDITOR loss ( $L_{rev}$ ) that aligns velocity disagreements between views with the critic's consistency scores.
Inference: Generation is performed via a rectified-flow ODE solver (Heun steps). The AUDITOR can optionally be invoked at test time to score intermediate samples and trigger a single refinement step if consistency scores drop below a threshold.

3. Key Contributions

OMNIDRIVE Architecture: The first multi-agent driving world model that unifies language understanding, geometric layout, and pixel generation into a single position-aware token grid.
Latent Co-Compression: A novel view-time permutation strategy that transforms inter-camera 3D constraints into local convolutional dependencies within a shared 3D VAE, eliminating the need for post-hoc fusion.
LLM-Choreographed Conditioning: A framework where three specialized agents (ARCHITECT, CARTOGRAPHER, AUDITOR) author a conditioning sequence bound to the visual grid at the token level, enabling fine-grained, disentangled control.
State-of-the-Art Performance: The model sets new benchmarks on the nuScenes dataset for multi-view consistency and controllability.

4. Experimental Results

Evaluated on the nuScenes validation set (6 cameras, 32 frames, 880×1280 resolution):

Quality & Consistency: OMNIDRIVE achieves an FVD of 45.7 and a multi-view consistency score (PC) of 65.6%, significantly outperforming baselines like MagicDrive-V2 (FVD 64.8, PC 62.8%) and UniMLVG. It reduces epipolar photometric error (EPC) by ~28% compared to the runner-up.
Controllability: The model achieves a BEV mAP of 21.55 and mIoU of 18.87 on generated data, surpassing all other methods.
Downstream Utility: A detector (BEVFormer-S) trained purely on 20k synthetic clips generated by OMNIDRIVE achieves 36.8 mAP and 44.0 NDS on real nuScenes validation data. This represents a +2.4 NDS improvement over a baseline trained only on real data, validating the high utility of the synthetic corpus.
Efficiency: OMNIDRIVE operates at the Pareto frontier of quality vs. latency, generating clips in 9.6 seconds (2.4× faster than UniMLVG) with lower memory usage.

5. Significance and Claims

The paper claims that OMNIDRIVE resolves the long-standing trade-off between fidelity and controllability in driving world models. By establishing a "shared symbolic interlingua" where symbolic language, rendered geometry, and pixel latents share a fully aligned coordinate frame, the system enables:

Genuine Controllability: Users can swap single fields in the WORLDSCRIPT (e.g., weather, trajectory) to alter only the targeted attribute without affecting others, demonstrating fine-grained, disentangled control.
Geometric Grounding: The co-compression approach ensures that geometric constraints are respected at the latent level, preventing the "cross-view drift" and "object teleportation" common in independent encoding schemes.
Practical Utility: The generated data is not merely visually plausible but functionally valuable, directly improving the performance of downstream perception models trained on real-world data.

The authors position OMNIDRIVE as a foundational step toward "genuine controllability" in generative world models, moving beyond disconnected control streams to a unified, agent-coordinated generation process.