DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving

This paper introduces DrivingGen, the first comprehensive benchmark for generative driving world models that addresses the lack of rigorous evaluation by combining a diverse dataset with a novel suite of metrics to assess visual realism, trajectory plausibility, temporal coherence, and controllability, thereby revealing critical trade-offs in current state-of-the-art models.

The Gist

Imagine you are teaching a robot how to drive a car. You can't just let it practice on real roads immediately; that's too dangerous and expensive. Instead, you want to build a super-realistic video game simulator where the robot can imagine the future: "If I turn left here, what will the other cars do? What will the rain look like? Will a pedestrian suddenly step out?"

This is the goal of Generative World Models for autonomous driving. They are AI systems that can "dream" up future driving scenarios.

However, there's a problem: How do you know if the simulator is any good?

The Problem: The "Fake News" of Driving

Currently, researchers have built many of these simulators, but they are testing them with the wrong ruler.

• The Old Way: They used general video metrics. It's like judging a driving simulator by asking, "Does this video look pretty?" A model could generate a beautiful, sunny beach drive, but if the physics are wrong (e.g., a car floats through a wall or a pedestrian vanishes into thin air), it's useless for teaching a real car.
• The Missing Pieces: Existing tests ignore:
  • Safety: Does the video handle fog, night, or snow?
  • Physics: Do the cars move like real cars, or do they jitter and teleport?
  • Consistency: Do the other cars stay the same car, or do they morph into different vehicles frame-by-frame?
  • Control: If you tell the simulator "turn left," does it actually turn left, or does it ignore you?

The Solution: Introducing "DrivingGen"

The authors of this paper created DrivingGen, the first comprehensive "report card" specifically designed to grade these driving simulators. Think of it as a driving school final exam for AI.

1. The Test Track (The Dataset)

Previous tests mostly used sunny, daytime city drives. DrivingGen is like a driving school that throws everything at you:

• Weather: Rain, snow, fog, sandstorms, and floods.
• Time: Dawn, day, night, and sunset.
• Locations: From busy streets in Tokyo to highways in the US and rural roads in Africa.
• Scenarios: Aggressive cut-ins, pedestrians waiting at crosswalks, and dense traffic jams.

They created a dataset of 400 diverse scenarios to ensure the AI isn't just memorizing one type of road.

2. The Grading Rubric (The Metrics)

DrivingGen doesn't just look at the video; it looks at the physics and the logic. They use four main categories to grade the AI:

• Distribution (The "Vibe Check"): Does the generated world look like a real driving dataset, or does it look like a cartoon? They measure how close the "feel" of the video is to reality.
• Quality (The "Visuals"):
  • Subjective: Does it look good to a human?
  • Objective: Are there weird flickering lights (like from streetlamps) that would confuse a real car's sensors?
  • Trajectory: Is the ride smooth, or does it jerk around like a drunk driver?
• Temporal Consistency (The "Memory Test"):
  • Scene: Does the background stay stable?
  • Agents: If a red car is there in frame 1, is it still a red car in frame 50?
  • The "Ghost" Test: Did a car suddenly disappear without a reason (like driving off a cliff)? Real cars don't just vanish; the AI shouldn't make them vanish either.
• Trajectory Alignment (The "Obedience Test"): If you tell the AI, "Drive this specific path," does it actually follow it? Many models look pretty but ignore the instructions.

The Results: The "Pretty but Dumb" vs. "Ugly but Smart" Dilemma

The authors tested 14 different AI models (from big tech companies and open-source projects) using this new exam. Here is what they found:

• The "Hollywood" Models: Some general video models (like the ones that make cool movie clips) make stunning, beautiful videos. They look amazing, but they break the laws of physics. Cars might slide sideways, or pedestrians might teleport. They are great for art, but terrible for driving.
• The "Engineer" Models: Some models built specifically for driving are very good at physics. They follow the rules of the road and stay on the path. However, their videos often look blurry, weird, or low-quality.
• The Gap: No single model is perfect yet. The best models are either "pretty but dangerous" or "safe but ugly."

Why This Matters

DrivingGen is a massive step forward because it stops researchers from just making "pretty pictures." It forces them to build simulators that are safe, reliable, and controllable.

Think of it this way: Before DrivingGen, we were judging driving simulators by how well they could paint a sunset. Now, we are judging them by how well they can navigate a snowstorm without crashing. This benchmark will help engineers build the "perfect" driving simulator, which is the key to making self-driving cars safe enough for us all to use.

Technical Summary

Here is a detailed technical summary of the paper "DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving."

1. Problem Statement

Generative video models have emerged as powerful "world models" capable of simulating future scenarios, offering significant potential for autonomous driving (AD) in areas like safe testing of corner cases, scalable simulation, and synthetic data generation. However, the field lacks a rigorous, comprehensive benchmark to evaluate these models effectively. Existing evaluations suffer from four critical limitations:

• Inadequate Metrics: Generic video metrics (e.g., FVD) overlook safety-critical imaging factors (glare, sensor artifacts) and fail to quantify trajectory plausibility.
• Neglected Consistency: Temporal and agent-level consistency (e.g., agents disappearing abruptly or changing identity) are often ignored.
• Lack of Controllability: Existing benchmarks rarely assess whether generated motion faithfully follows ego-vehicle conditioning (trajectory instructions).
• Data Bias: Current datasets (e.g., nuScenes, OpenDV) are heavily skewed toward clear weather and daytime urban scenes, failing to cover the diversity required for real-world deployment (night, snow, fog, global regions).

2. Methodology: The DrivingGen Benchmark

DrivingGen introduces a unified evaluation framework comprising a diverse dataset and a novel suite of metrics designed specifically for the driving domain.

A. Benchmark Dataset

The dataset consists of 400 samples (200 per track) curated to ensure diversity across weather, time of day, geography, and maneuvers. It is divided into two tracks:

1. Open-Domain Track: Evaluates generalization to unseen scenarios using internet-sourced data. It balances weather (rain, snow, fog, sandstorms) and time of day (dawn, day, night) to reduce the bias toward sunny daytime conditions found in existing datasets.
2. Ego-Conditioned Track: Evaluates trajectory controllability. It aggregates data from five open-source driving datasets (Zod, DrivingDojo, COVLA, nuPlan, WOMD) covering North America, Europe, and Asia. This track provides optional ego-trajectory inputs to test if the model can generate videos that align with specific driving instructions.

B. Evaluation Metrics

DrivingGen proposes a 4-dimensional metric suite covering Distribution, Quality, Temporal Consistency, and Trajectory Alignment.

1. Distribution Metrics:

   • Fréchet Video Distance (FVD): Measures visual distribution similarity.
   • Fréchet Trajectory Distance (FTD): A novel metric using a Motion Transformer (MTR) encoder to measure the distributional distance between generated and real driving trajectories.

2. Quality Metrics:

   • Subjective Quality: Uses CLIP-IQA+ for human-aligned perceptual scores.
   • Objective Quality: Includes Modulation Mitigation Probability (MMP) based on IEEE P2020 standards to detect PWM-induced flicker (critical for AD sensors).
   • Trajectory Quality: A composite score evaluating kinematic feasibility, including comfort (jerk/acceleration), motion (avoiding static outputs), and curvature (avoiding zig-zags).

3. Temporal Consistency:

   • Video Consistency: Uses optical flow to adaptively downsample videos before comparing DINOv3 features, preventing static videos from scoring artificially high.
   • Agent Consistency: Tracks individual agents (using YOLOv10 and SAM2) to detect appearance drift.
   • Agent Abnormal Disappearance: Uses a Vision-Language Model (Cosmos-Reason1) to distinguish between natural occlusions and unnatural, physics-breaking disappearances.
   • Trajectory Consistency: Measures the stability of speed and acceleration over time.

4. Trajectory Alignment (Ego-Conditioned only):

   • Average Displacement Error (ADE): Measures pointwise deviation from the input trajectory.
   • Dynamic Time Warping (DTW): Measures the overall shape and contour alignment between predicted and reference paths.

3. Key Contributions

• First Comprehensive Driving Benchmark: DrivingGen is the first benchmark to jointly evaluate visual realism, trajectory plausibility, temporal coherence, and controllability for generative driving world models.
• Diverse Data Distribution: It addresses the "long-tail" problem by explicitly including rare conditions (snow, fog, night, global regions) and complex interactions (pedestrians, cut-ins).
• Novel Metrics: Introduces driving-specific metrics like FTD, MMP, and Agent Abnormal Disappearance detection, moving beyond generic video quality scores.
• Open Source: The dataset, evaluation code, and results for 14 state-of-the-art models are publicly released.

4. Experimental Results & Insights

The authors benchmarked 14 models across three categories: General Video Models (e.g., Kling, Gen-3, Wan), Physical World Models (e.g., Cosmos), and Driving-Specific Models (e.g., Vista, DrivingDojo).

Key Findings:

• Trade-offs Exist: No single model excels in both visual fidelity and physical consistency.
  • General Models (Closed-source): (e.g., Kling, Gen-3) achieve top visual quality and stable agent behavior but often break physical laws (poor trajectory adherence).
  • Driving-Specific Models: (e.g., Vista, DrivingDojo) capture motion realism and follow trajectories better but lag in visual fidelity and exhibit more artifacts.
• Controllability Gap: Even top models show significant errors (high ADE/DTW) when conditioned on ego-trajectories, indicating a gap in following precise driving instructions.
• Hidden Failure Modes: Relying solely on FVD hides critical failures. Models with good FVD scores often exhibit "stop-go" jitter, identity drift, or non-physical disappearances, which DrivingGen successfully exposes.
• Human Alignment: The proposed metrics show strong correlation with human judgments, validating the benchmark's reliability.

5. Significance

DrivingGen establishes a unified, reproducible framework that shifts the focus from "looking good" to "being safe and controllable." By highlighting the current trade-offs between visual quality and physical plausibility, it guides future research toward developing driving world models that are not only photorealistic but also physically consistent and reliable for closed-loop simulation and planning. This is a critical step toward deploying scalable, safe, and data-efficient autonomous driving systems.