Technical Summary: From Priors to Perception: Grounding Video-LLMs in Physical Reality

1. Problem Statement

While Video Large Language Models (Video-LLMs) demonstrate proficiency in general video understanding, they exhibit systematic and severe deficits in fine-grained physical reasoning. State-of-the-art (SOTA) models frequently fail to identify obvious physical violations (e.g., object levitation, rigid-body penetration) and, crucially, fail in counter-intuitive scenarios where visual facts contradict statistical expectations (e.g., a ball missing a set of dominoes).

Existing interventions, such as inference-time decoding, reinforcement learning, or invasive architectural modifications (e.g., integrating explicit trackers or Neural ODEs), suffer from limited generalization, high computational overhead, or a failure to address the root cause. Furthermore, current evaluation datasets often rely on capturing natural failures from generative video models. This paradigm conflates low-level visual artifacts (e.g., texture flickering, object melting) with genuine physical fallacies, causing models to regress into "shortcut-driven artifact detectors" rather than rigorous physical reasoners.

The authors identify that these failures stem not from a lack of perception or knowledge, but from Semantic Prior Dominance. The reasoning mechanism is hijacked by internal narrative scripts and statistical priors, forcing the model to overwrite objective visual evidence with subjective expectations. In anti-physics scenarios, models fabricate invisible forces to rationalize anomalies; in counter-intuitive scenarios, they hallucinate expected outcomes despite clear visual gaps.

2. Methodology

To address Semantic Prior Dominance without invasive architectural changes, the authors propose a unified framework comprising a high-fidelity dataset and a novel reasoning mechanism.

2.1 Programmatic Adversarial Curriculum (PACC)

PACC is the first high-fidelity adversarial video dataset synthesized based on physical laws, designed to strictly decouple visual artifacts from logical errors.

• Construction: Utilizing a Human-in-the-Loop (HITL) pipeline, PACC synthesizes paired adversarial videos (a factual positive and a manipulated negative) across eight fine-grained dimensions derived from developmental psychology and intuitive physics benchmarks.
• Taxonomy: The dataset covers two streams:
  1. Anti-Physics Stream: Targets rationalization fallacies by violating objective laws (e.g., Temporal: coherence violations, causal reversal; Permanence: existence/identity violations; Interaction: dynamic/constraint violations).
  2. Counter-Intuitive Stream: Targets stereotypical fallacies where expected events fail to occur (e.g., Near-Miss, Consequence Arrest).
• Quality Control: Unlike prior datasets, PACC ensures visual pristine-ness while enforcing physical invalidity, preventing models from exploiting generative artifacts as shortcuts.

2.2 Visual-Anchored Reasoning Chain (VARC)

VARC is a mechanism designed to force models to ground judgments in low-level visual facts before logical adjudication. It reformulates the reasoning process as a directed Markovian chain:

1. Forced Observation: The model must explicitly describe low-level spatial relationships and state changes (e.g., "spatial gap") without premature subjective judgment.
2. Causal Attribution: Conditioned on the objective observation, the model queries its physical knowledge base to detect conflicts (e.g., recognizing thermodynamic impossibilities).
3. Evidence-Based Verdict: The final judgment is strictly constrained by the preceding observation and attribution nodes.

2.3 Training Strategy: PhyAR

The authors introduce PhyAR (Physics-Anchored Reasoner), a model fine-tuned using standard LoRA on the PACC dataset.

• Paired Gradient Suppression: During training, positive and negative video pairs are jointly optimized. This induces gradient suppression on shared visual semantics (backgrounds, lighting), forcing the model to converge strictly on core physical discrepancies rather than static visual distributions.
• Architecture: The vision encoder is frozen; LoRA is applied exclusively to the visual projector and LLM backbone, ensuring an architecture-agnostic, lightweight solution.

3. Key Contributions

1. Unified Attribution Theory: The paper proposes that dual failures (anti-physics and counter-intuitive) are manifestations of Semantic Prior Dominance, where internal narrative scripts hijack the reasoning mechanism, overriding objective visual evidence.
2. Programmatic Adversarial Curriculum (PACC): A novel dataset of 758 high-fidelity adversarial video pairs that decouples generative artifacts from genuine physical fallacies, providing a rigorous diagnostic benchmark.
3. Visual-Anchored Reasoning Chain (VARC): A reasoning framework that mechanistically blocks prior hijacking by compelling explicit visual grounding.
4. Empirical Validation: Demonstration that standard LoRA fine-tuning with PACC and VARC effectively neutralizes prior interference in SOTA models, yielding substantial leaps in physical reasoning without architectural modifications.

4. Experimental Results

Experiments were conducted on the PACC test set (154 video pairs) against SOTA proprietary (GPT-4o, Gemini 2.5 Flash) and open-source Video-LLMs.

• Pair-wise Consistency Accuracy (PCA): PhyAR achieved an average PCA of 41.56%, significantly outperforming all baselines. Notably, proprietary models like GPT-4o scored 38.31%, while smaller open-source models collapsed to near-zero performance (e.g., Video-ChatGPT-7B at 0.65%) due to severe semantic inertia and "Yes-bias."
• Reasoning Alignment Score (RAS): PhyAR achieved an average RAS of 2.036 (on a 1-5 scale), surpassing open-source baselines (which stagnated near 1.0) and approaching proprietary model performance. Crucially, PhyAR outperformed all baselines in Counter-Intuitive categories, demonstrating the ability to resist statistical priors.
• Semantic Prior Bias (RPB): The Relative Prior Bias metric showed a reduction from 79.1% in the base model (VideoLLaMA3) to 24.9% in PhyAR, confirming the suppression of prior dependence.
• Ablation Studies: Removing VARC dropped PCA to 28.57%, and removing pair-wise optimization dropped it to 29.22%, validating that both visual grounding and paired gradient suppression are essential for overcoming semantic priors.

5. Significance and Claims

The paper claims to provide a fundamental shift in how Video-LLMs approach physical reasoning. By revealing that these models are not suffering from perception deficits but from Semantic Prior Dominance, the work challenges the prevailing notion that more data or larger models alone will solve physical reasoning gaps.

The significance lies in the demonstration that genuine physical reasoning can be achieved through:

1. Decoupling artifacts from logic: Using PACC to train models on pure physical fallacies rather than generative noise.
2. Enforcing visual grounding: Using VARC to force the model to "see" before it "thinks," thereby dismantling the shortcut-driven reliance on narrative scripts.

The authors position this work as an imperative step toward reliable World Models and Embodied AI, arguing that breaking the physical reasoning bottleneck is necessary for safe real-world interaction. They acknowledge current limitations, noting that PACC is currently limited in scale (758 pairs) due to the rigorous human-in-the-loop verification process, prioritizing fidelity over volume, but suggest future work could explore automated synthesis for broader scaling.