How Can Driving World Models Do Counterfactual Prediction?

Technical Summary: Counterfactual Prediction in Driving World Models

Problem Definition and Causal Gap
The paper addresses a fundamental mismatch between the intended capability of driving world models (counterfactual prediction) and their standard implementation (direct action-conditioned prediction). In autonomous driving, counterfactual prediction asks: "Given a factual driving log where an event occurred (e.g., a car cutting in), what would the camera have recorded if the ego vehicle had taken a different action (e.g., braking instead of accelerating)?"

The authors identify that current "direct predictions" condition the model only on the shared history ($H$) and the alternative action ($a'$). This approach fails to incorporate the factual continuation ($F^+$)—the observed evidence of what actually happened after the shared history. According to Pearl's "ladder of causation," a true counterfactual (Rung 3) requires abduction: inferring the specific state of the world ($w$) that generated the observed factual outcome, then intervening with the new action. Direct prediction, by ignoring $F^+$, effectively averages over all possible worlds consistent with the history ($p(w|H)$), rather than the specific world realized in the episode ($p(w|H, F^+)$). Consequently, direct models often generate fluent videos consistent with the new action but omit specific events (like a car emerging) that were actually present in the episode.

Methodology: A Training-Free Pipeline
To bridge this gap without retraining existing models, the authors propose a simple, inference-time pipeline grounded in the causal recipe of Abduction, Action, and Prediction. The method operates on a frozen world model and consists of four stages:

1. Abduce (Reconstruction): A frozen depth estimation model processes the factual continuation ($F^+$) to estimate metric depth. This lifts the 2D RGB frames into a 3D point cloud, reconstructing the observed surfaces of the realized world.
2. Transport (Intervention): The system applies the alternative ego trajectory ($a'$) to the camera pose while keeping the reconstructed 3D world fixed. Using forward splatting and a depth buffer, the observed 3D points are reprojected into the counterfactual camera view. This generates a warped evidence image ($E_t$) and a support mask ($M_t$) indicating which pixels are determined by the factual observation.
3. Complete (Prediction): The frozen world model is tasked with generating only the unsupported regions (areas not visible in the factual log or occluded).
   • For diffusion models, the input is a noisy encoding of the factual evidence masked by $M_t$. During the denoising process, the evidence regions are restored at each step to constrain the generation.
   • For autoregressive models, tokens corresponding to the evidence mask are fixed to the input values, while the model generates tokens for the remaining regions.
4. Combine (Refinement): The output from the world model is blended with the transported evidence. A weighting function ($\alpha_t$) ensures that reliable transported pixels are preserved verbatim, while boundaries are smoothed to harmonize colors and reduce seams.

Benchmark and Evaluation
Since real-world data cannot provide counterfactual ground truth (one cannot observe two futures from one past), the authors constructed a controlled benchmark using the CARLA simulator.

• Setup: They simulate a world with a predefined event (e.g., a car emerging from a side street). They run three scenarios: (1) Factual execution ($F, a_{obs}$), (2) Counterfactual execution ($P, a'$), and (3) Null execution where the event is suppressed ($U, a'$).
• Metrics:
  • Recovered Fraction ($Rec$): Measures how much the prediction resembles the counterfactual ground truth ($P$) versus the null reference ($U$) using embedding similarity (DINOv2 and CLIP). A score of 1 indicates perfect recovery of the event; 0 indicates complete omission.
  • Perceptual Fidelity: Measures the LPIPS distance between the prediction and the counterfactual ground truth to assess visual quality.

Key Results
Experiments on two representative world models (a diffusion-based model and an autoregressive model) yielded the following findings:

• Direct Prediction Failure: Standard direct action-conditioned predictions consistently failed to preserve the specific events present in the factual continuation. Their "recovered fraction" scores were low, indicating they generated plausible but incorrect futures where the event (e.g., the emerging car) did not occur.
• Method Efficacy: The proposed training-free pipeline significantly improved performance. It substantially increased the recovered fraction, successfully retaining the event signal that direct prediction lost.
• Visual Quality: The method reduced the perceptual distance (LPIPS) to the counterfactual ground truth, producing outputs that were both semantically correct (containing the event) and visually coherent.

Significance and Claims
The paper does not claim to have solved counterfactual prediction for all driving scenarios or to have introduced a new learning architecture. Instead, its significance lies in:

1. Diagnosis: Providing a precise causal explanation for why current world models fail at counterfactual tasks (the omission of abduction/evidence from the factual continuation).
2. Benchmarking: Establishing the first controlled benchmark with matched counterfactual ground truths to quantitatively measure this specific failure mode.
3. Constructive Baseline: Demonstrating that a simple, training-free geometric intervention can recover the "missing evidence" and significantly improve counterfactual fidelity. This serves as a reference point for future methods, suggesting that better counterfactual prediction may require explicitly integrating observed evidence into the generation process rather than relying solely on action conditioning.

The authors conclude that this work aims to draw attention to the gap between direct prediction and true counterfactual reasoning, motivating the development of methods that respect the causal structure of specific driving episodes.