Technical Summary: YoCausal

Problem Statement

As Video Diffusion Models (VDMs) advance toward the goal of becoming "world models," a critical question remains: do these models genuinely understand causality, or do they merely overfit to statistical temporal patterns? Existing benchmarks for evaluating "world knowledge" in generative models largely rely on synthetic data or controlled laboratory recordings to isolate physical variables. This reliance creates a significant "sim-to-real" gap, limiting the assessment of real-world generalization. Furthermore, current evaluations often conflate the perception of the "arrow of time" (the inherent directionality of time) with genuine causal reasoning (understanding that event A causes event B). There is a lack of scalable, real-world benchmarks that can disentangle these two concepts to evaluate whether VDMs have internalized causal structures as part of their learned priors.

Methodology

The authors propose YoCausal, a two-level benchmark inspired by the Violation of Expectation (VoE) paradigm from cognitive science. The core hypothesis is that a causally aware model should exhibit "surprise" (lower probability/higher loss) when presented with counterfactual stimuli, such as temporally reversed videos, compared to forward videos.

1. Dataset Construction

Unlike prior works requiring synthetic rendering, YoCausal utilizes real-world videos from diverse domains (General, Physics, Human Action, Animal Action) sourced from existing datasets (e.g., Moments in Time, Physics IQ, Kinetics, Animal Kingdom).

• Zero-Cost Counterfactuals: The benchmark generates natural counterfactual samples by temporally reversing real-world videos. This allows for an arbitrarily extensible dataset without the sim-to-real gap.
• Scale: The initial evaluation covers 1,232 videos across four thematic subsets.

2. Evaluation Framework

The framework operates on two levels to separate temporal perception from causal cognition:

• Level 1: Reverse Surprise Index (RSI)

  • Mechanism: For each video, the model computes the denoising loss for both the forward sequence ($x_f$) and the reversed sequence ($x_r$) using identical sampled noise at the same timesteps.
  • Metric: RSI measures the proportion of videos where the model assigns a lower denoising loss to the forward sequence than the reversed one ($L(x_f) < L(x_r)$).
  • Interpretation: A high RSI indicates the model perceives the arrow of time. However, this alone does not confirm causal understanding, as it could stem from statistical temporal biases.

• Level 2: Causality Cognition Index (CCI)

  • Disentanglement: To isolate genuine causal reasoning, the dataset is partitioned into a Causal Subset ($D_c$) (videos with explicit cause-and-effect interactions) and a Non-Causal Subset ($D_{nc}$) (videos without such interactions, e.g., a car cruising).
  • Mechanism: A Vision-Language Model (VLM) automates this partitioning based on the presence of observable cause-and-effect mechanisms.
  • Metric: $CCI = RSI(D_c) - RSI(D_{nc})$.
  • Interpretation: A causally aware model should be significantly more "surprised" by reversed causal videos than by reversed non-causal videos. A high CCI indicates the model captures causal cues beyond statistical temporal patterns.

3. Validation

• VLM Reliability: The VLM-based partitioning was validated against human annotations (Kendall's $\tau = 0.7613$) and optical flow analysis, confirming the split is based on high-level semantics rather than low-level motion cues.
• Entropy Control: Experiments on entropy-symmetric subsets confirmed that RSI reflects event-level temporal structure rather than low-level entropy dynamics.

Key Results

The benchmark evaluated 13 state-of-the-art open-source VDMs (including Wan, CogVideoX, HunyuanVideo, and LTX-Video families) against a human baseline.

1. Time vs. Causality: There is a significant gap between perceiving the arrow of time and understanding causality. Several models achieved high RSI (perceiving time direction) but low or negative CCI, indicating they rely on temporal statistics rather than causal logic.
2. Human-Model Gap: Even the best-performing models exhibit a substantial gap relative to human-level causal cognition. Humans achieved the highest RSI and CCI across all subsets.
3. Scaling Laws: Causal cognition correlates significantly with model parameters and release dates ($\tau \approx 0.69$ and $0.60$ respectively). Larger models and newer architectures (e.g., DiT-based) show improved causal understanding, suggesting scaling laws extend to this higher-order reasoning.
4. Independence from Aesthetics: The benchmark shows zero correlation with aesthetic quality metrics, confirming that causal understanding is a distinct capability not confounded by visual fidelity.
5. Correlation with Physics: There is a moderate positive correlation with intuitive physics benchmarks (LikePhys), suggesting physical laws constrain object interactions, but causal cognition is not reducible to physics alone.

Significance and Contributions

The paper claims the following contributions:

• First Causality Benchmark for VDMs: YoCausal is the first benchmark specifically designed to evaluate causal cognition in video generation models, utilizing a scalable real-world dataset free from sim-to-real gaps.
• Cognitive-Science-Grounded Framework: It introduces a two-level framework that successfully disentangles "arrow-of-time" perception from genuine "causal cognition," addressing a limitation in prior physics-based benchmarks.
• Diagnostic Evidence: The evaluation reveals that current open-source VDMs lack robust causal understanding, highlighting a critical gap between current generative models and the aspirational "world model."
• Guidance for Future Research: By demonstrating that scaling and architectural evolution improve causal cognition, the work provides a quantitative basis for guiding the development of future world models, emphasizing causal reasoning as a distinct objective.

The authors note limitations, including the difficulty of evaluating temporally symmetric events (where forward and reversed videos are visually identical) and the reliance on model weights for denoising loss computation, which restricts evaluation of closed-source models. However, they posit that the benchmark effectively covers the dominant regimes of visually explicit causality relevant to robotic simulation, game engines, and autonomous driving.