Technical Summary: CRONOS – Benchmarking Counterfactual Physical Consistency in Video Models

Problem Statement

Recent advances in generative video modeling have enabled the synthesis of temporally coherent, high-fidelity futures, fueling the hypothesis that scaling video prediction may yield generalizable "world models" capable of capturing real-world dynamics [19, 22, 21]. However, visual realism does not necessarily imply the acquisition of causal representations that capture relationships between objects, scenes, and dynamics [42]. Current models may rely on superficial statistical regularities rather than underlying physical laws, failing to produce stable predictions when visual conditions (e.g., viewpoint, appearance) change while the physical event remains constant. Existing benchmarks for physical realism often evaluate predictions under fixed visual observations or rely on human/VLM judgments of plausibility, failing to systematically test whether a model's predictive representation is stable and structured under controlled interventions [30, 36, 52]. There is a critical gap in principled evaluations that move beyond perceptual quality to directly test counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in visual input.

Methodology

The authors introduce CRONOS, an intervention-based benchmark designed to evaluate counterfactual physical consistency. The methodology relies on a high-fidelity, photorealistic simulation environment built in Unreal Engine to generate controlled video sequences where the underlying physical event type is fixed, but specific visual factors are systematically varied.

1. Data Generation and Interventions

CRONOS generates videos across three canonical physical interaction scenarios:

• Fall (Roll-to-drop): An object rolls and falls from an edge.
• Collision: One object impacts another.
• Occlusion: An object rolls behind an obstacle and reappears.

For each event, the benchmark employs a full-factorial design to render counterfactual observations by intervening on four dimensions while holding others fixed:

• Scene: Changing background environment and layout (e.g., surface height).
• Camera Viewpoint: Changing the rendering angle while preserving 3D dynamics.
• Object Appearance: Changing visual attributes (e.g., color) without altering physical parameters.
• Object Category: Replacing the object with a different compatible one, altering both visual properties and physical parameters (mass, friction).

The dataset comprises 675 videos (3 events × 5 scenes × 5 object categories × 4 viewpoints × 3 appearances), with viewpoint variation omitted for occlusion events to preserve visibility structure.

2. Evaluation Metrics

CRONOS decomposes generation quality into object-centric metrics to disentangle 3D motion from appearance, validated through a human user study:

• Appearance Stability: Measures visual identity preservation using DINOv2 embeddings.
• Background Stability: Measures background coherence via pixel-wise error (MSE).
• 3D-Shape Stability: Assesses geometric stability by comparing reconstructed meshes (via SAM3D) across time using Chamfer distance.
• Motion Similarity: Evaluates motion agreement using an appearance-invariant motion encoder (DisMo).
• Physical Plausibility: Uses a VLM-as-judge (Qwen3-VL-32B) to detect high-level physical violations (e.g., object permanence, gravity).
• Success Rate: A binary pass/fail signal requiring all metrics to exceed calibrated thresholds.
• Sensitivity: Measures the deviation in performance across intervention types; lower sensitivity indicates higher counterfactual consistency.

Key Contributions

1. CRONOS Benchmark: A novel, intervention-based benchmark that systematically evaluates how video models respond to controlled changes in viewpoint, scene, object category, and appearance, isolating counterfactual physical consistency from general generation quality.
2. High-Fidelity Synthetic Environment: Unlike previous synthetic benchmarks relying on flat textures, CRONOS utilizes photorealistic Unreal Engine rendering with professional 3D assets to simulate realistic lighting, shadows, and dynamics.
3. Object-Centric Evaluation Framework: A suite of metrics that disentangles motion from appearance and shape, including a VLM-based judge for physical plausibility and a robust aggregation method (robust min/max over frames) to handle transient errors.
4. Comprehensive Empirical Analysis: An evaluation of state-of-the-art open-source video models (Cosmos2.5, CogVideoX1.5, MAGI-1, Wan2.2) under both Image-to-Video (I2V) and Video-to-Video (V2V) settings.

Results

The evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency:

• Low Success Rates: Even the strongest evaluated models achieve low success rates. Cosmos2.5-2B (V2V) and Wan2.2-14B (I2V) achieved only 22% and 20% success rates, respectively, while most other models scored below 15%.
• High Sensitivity to Interventions: Models show substantial variation in generation quality across intervention types.
  • Viewpoint Sensitivity: Viewpoint changes induced the largest variation, indicating models encode predictions in a strongly view-dependent manner rather than learning 3D-aware dynamics.
  • Object and Scene Sensitivity: Changes in object category and scene layout also caused significant performance drops, suggesting reliance on object-specific correlations rather than generalizable physical rules.
• Effect of Conditioning: Video conditioning (V2V) generally improved performance over image conditioning (I2V), particularly in motion similarity and object stability, likely due to the additional temporal signal providing initial direction and velocity.
• Effect of Model Size: Scaling Cosmos from 2B to 14B parameters yielded no improvement in physical event generation; in fact, the 14B variant performed worse in several metrics, suggesting that increased parameter count alone does not guarantee better counterfactual physical consistency.

Significance and Claims

The paper claims that CRONOS provides a controlled and reproducible testbed for diagnosing the limitations of current video models. The findings suggest that modern models primarily rely on superficial visual correlations rather than developing stable, causal representations of scene dynamics. The authors posit that structured, causally meaningful representations are essential for robust generalization and decision-making, and that current models fail to achieve this, particularly under nuisance variations like viewpoint changes.

The benchmark establishes a concrete target for developing models that perform consistently across changes in multiple conditions. The authors emphasize that while CRONOS uses synthetic data, its high visual fidelity allows for rigorous diagnostic testing of physical prediction capabilities that is difficult to achieve with real-world video due to the inability to independently control variables. The work does not claim that current models have achieved general world modeling but rather highlights the gap between visual realism and physical consistency, urging future research to focus on robustness under controlled interventions.