X-World: Controllable Ego-Centric Multi-Camera World Models for Scalable End-to-End Driving

X-World is a controllable, action-conditioned multi-camera generative world model that simulates realistic, geometrically consistent future ego-centric video streams with optional scene and appearance controls, providing a scalable and reproducible foundation for evaluating end-to-end autonomous driving policies.

The Gist

Imagine you are teaching a robot to drive a car. In the old days, you had to take the robot out onto real roads, let it make mistakes, and hope it didn't crash. This is expensive, dangerous, and you can't easily recreate the exact same traffic jam or rainy day to test if the robot learned its lesson.

X-World is like a "Magic Movie Studio" built specifically for training these self-driving robots. Instead of just showing the robot a video, X-World lets you direct the movie in real-time.

Here is how it works, broken down into simple concepts:

1. The Core Idea: A "Choose Your Own Adventure" Movie

Think of X-World as a super-smart AI director.

• The Input: You give the director the last few seconds of a movie (what the car sees right now) and a script (what the car wants to do next, like "turn left" or "speed up").
• The Output: The director instantly generates the next few seconds of the movie from seven different camera angles (front, sides, rear) all at once.
• The Magic: If you tell the car to turn left, the movie instantly shows the car turning left, and the buildings, other cars, and trees move exactly as they should from every single camera angle. It's not just a video; it's a simulated reality.

2. The "Control Panel" (Controllability)

Usually, AI video generators are a bit chaotic. You ask for a "sunny day," and you get a sunny day, but you can't control where the other cars go. X-World is different because it has a remote control for everything:

• The Driver's Moves (Ego-Action): You can say, "Go straight," and the video shows the car going straight. If you say, "Swerve to avoid a pothole," the video shows the car swerving smoothly.
• The Traffic (Dynamic Agents): You can tell the AI, "Put a cyclist right in front of the car." The AI will generate a realistic cyclist appearing out of nowhere, and the car will react to it.
• The Road (Static Elements): You can change the road layout. "Make the lane lines disappear" or "Add a stop sign." The video updates the road geometry instantly.
• The Mood (Text Prompts): You can type "Make it rain" or "Change the time to sunset," and the entire lighting and weather of the 7-camera view will shift instantly, while the car keeps driving exactly the same way.

3. The "Seven Eyes" (Multi-Camera Consistency)

Imagine you are in a room with seven friends looking at a ball. If you throw the ball, all seven friends must see it move in the same direction and at the same speed.

• The Problem: Many AI video generators are like friends who aren't talking to each other. One might see the ball go left, while another sees it go right. This is called "geometric inconsistency."
• The X-World Solution: X-World is like a team of friends who are telepathically connected. It ensures that if the car turns, the front camera sees the turn, the side camera sees the side of the car, and the rear camera sees the road behind—all perfectly aligned. It never gets confused about where things are in 3D space.

4. The "Time Machine" (Long-Horizon Stability)

Most AI video generators are great at making 5 seconds of video, but if you ask for 5 minutes, the video starts to glitch. The car might turn into a tree, or the road might disappear.

• X-World's Trick: It uses a special "memory buffer" (called a Rolling KV Cache). Think of it like a scroll that keeps track of the last few seconds of the movie. As the movie plays, it forgets the very old stuff to make room for new stuff, but it never loses the "big picture." This allows it to generate 24+ seconds of smooth, stable driving without the world falling apart.

5. Why Do We Need This? (The Real-World Use)

Why build a movie studio for cars?

• Safety Testing: You can test a self-driving car against a "ghost" car that appears out of nowhere, or a pedestrian running into the street. You can do this a million times without hurting anyone.
• The "What If" Scenario: Imagine a real video where a car got stuck behind a parked truck. With X-World, you can ask: "What if the car had decided to go around the truck instead?" The AI generates that alternative reality instantly so engineers can see if that decision was safe.
• Training the Brain: Just like a pilot uses a flight simulator to practice emergencies, self-driving cars use X-World to practice dangerous situations in a safe, virtual loop.

Summary

X-World is a controllable, multi-camera movie generator that acts as a perfect simulator for self-driving cars. It lets engineers direct the future, change the weather, add obstacles, and test "what-if" scenarios instantly, all while keeping the physics and camera angles perfectly consistent. It turns the dangerous, expensive job of testing self-driving cars into a safe, repeatable, and creative video game.

Technical Summary

1. Problem Statement

The rapid advancement of End-to-End (E2E) autonomous driving and Vision-Language-Action (VLA) models has created a critical bottleneck: the lack of scalable, reliable, and reproducible evaluation and training infrastructure.

• Limitations of Current Methods: Existing E2E systems are primarily evaluated via real-world road testing, which is costly, geographically biased, difficult to reproduce, and unsafe for exploring rare or safety-critical scenarios.
• Simulation Challenges: Traditional simulators often fail to provide photorealistic sensor data or lack strict causal consistency (action-following). Existing video generation models often suffer from:
  • Geometric Inconsistency: Disagreement between multiple camera views.
  • Temporal Instability: Drift over long rollouts.
  • Lack of Control: Inability to precisely control dynamic agents, static road elements, or global appearance while maintaining action causality.
  • Latency: Most high-quality video diffusion models are bidirectional and offline, making them unsuitable for real-time, closed-loop interaction required by reinforcement learning (RL).

2. Methodology: X-World Architecture

X-World is an action-conditioned, multi-camera generative world model built upon video diffusion technology. It generates future multi-view video streams directly in pixel space based on historical observations and future actions.

Core Architecture

• Base Model: Built on WAN 2.2 5B, utilizing a high-compression 3D Causal Variational Autoencoder (VAE) (16× spatial, 4× temporal compression) and a Diffusion Transformer (DiT) backbone.
• View-Temporal Self-Attention: A custom attention mechanism that explicitly models interactions across both time and camera views. This ensures geometric consistency and object identity alignment across the 7 synchronized cameras (Front-Narrow, Front-Fisheye, Front-Left/Right, Rear-Left/Right, Rear).
• Decoupled Condition Injection: To handle heterogeneous control signals without interference, X-World uses separate cross-attention branches for different modalities:
  • Ego-Action: Input as a sequence of kinematic states (velocity, curvature, roll, pitch), normalized and encoded via Fourier features.
  • Dynamic Agents: Encoded via umT5 for semantics and Fourier features for position; injected via cross-attention.
  • Static Elements: Lane markings, boundaries, and signs encoded similarly to agents, with Classifier-Free Guidance (CFG) and dropout strategies to ensure strict adherence during inference.
  • Camera Parameters: Intrinsic/extrinsic parameters injected additively.
  • Text Prompts: Retained from WAN 2.2 for global appearance control (weather, time of day, location).

Training Pipeline (Two-Stage)

1. Stage I: Bidirectional I2V Training (Controllability)
   • Trained on 81-frame synchronized clips using Rectified Flow.
   • Goal: Learn high-fidelity multi-view generation with accurate action following and scene control.
   • Limitation: Relies on many-step sampling (offline), unsuitable for real-time streaming.
2. Stage II: Causal Few-Step Training (Streaming & Long-Horizon)
   • Converts the model into a causal, autoregressive generator for real-time interaction.
   • Chunk-wise Causality: Generates video in chunks; tokens within a chunk interact bidirectionally, but tokens cannot attend to future chunks.
   • Self-Forcing & DMD Loss: Trains the model on its own autoregressive rollouts (rather than ground truth) using Distribution Matching Distillation (DMD). This minimizes the train-test mismatch and prevents error compounding over long horizons.
   • Rolling KV Cache: Uses a fixed-size cache to store attention keys/values from previous chunks, enabling efficient, low-latency long-horizon generation.

3. Key Contributions

• Controllable Multi-View World Model: The first system to simultaneously support strict ego-action following, dynamic/static agent control, and global appearance editing across 7 synchronized camera views.
• Geometric & Temporal Consistency: Introduces a view-temporal self-attention mechanism that prevents the "drift" and view-mismatch common in multi-camera generation.
• Streaming Autoregressive Interface: Bridges the gap between high-quality offline video generation and real-time simulation, enabling closed-loop RL and interactive evaluation.
• C2V (Condition-to-Video) Capability: Supports generating videos purely from text and action prompts (without visual history), enabling data synthesis and style transfer (e.g., changing weather or location).

4. Results

Experiments demonstrate X-World's superiority in three key areas:

1. Action Controllability: The model faithfully executes commanded maneuvers (turns, lane changes, braking) with physically plausible outcomes.
2. Scene Editability: It can precisely control dynamic agents (e.g., inserting a cyclist) and static elements (e.g., lane topology) while maintaining consistency across views and time.
3. Long-Horizon Stability: The model successfully generates stable 24-second multi-camera rollouts without catastrophic drift, a significant improvement over standard diffusion models.
4. Cross-View Consistency: Objects and structures maintain consistent identity and geometry across all 7 camera angles.

5. Significance and Applications

X-World serves as a foundational "real-world simulator" for the next generation of autonomous driving:

• Closed-Loop Evaluation: Enables scalable, reproducible testing of VLA policies in counterfactual scenarios (e.g., "What if the car ahead was parked?") without real-world risk.
• Online Reinforcement Learning: Provides a safe, interactive environment for RL agents to explore rare and dangerous scenarios (e.g., ghost-outs, aggressive cut-ins) to learn recovery behaviors.
• Data Synthesis & Augmentation: Acts as a "data factory" to generate corner cases (extreme weather, rare objects) and perform Zero-shot style transfer (e.g., adapting domestic driving data to European road styles) to accelerate global deployment.

In summary, X-World moves generative world models from passive video synthesis tools to active, controllable engines for training and validating end-to-end autonomous driving systems.