Which Way Does Time Flow? A Psychophysics-Grounded Evaluation for Vision-Language Models

1. Problem Statement

While modern Vision-Language Models (VLMs) excel at static image understanding and multimodal tasks, their ability to comprehend temporal dynamics and physical causality in videos remains unproven. Specifically, it is unclear whether VLMs possess the inductive biases (implicit assumptions about gravity, entropy, and causality) that humans use to determine the "Arrow of Time" (AoT).

Current temporal reasoning benchmarks often suffer from two critical flaws:

1. Lack of Temporal Dependency: Many tasks can be solved using scene context or object recognition without understanding event order.
2. Lack of Psychophysical Validation: Existing benchmarks often use arbitrarily reversed videos without human baselines, leading to ambiguous test cases where humans themselves cannot distinguish forward from backward playback.

The core question addressed is: Can VLMs distinguish between forward and backward video playback in natural scenarios where physical laws are violated in reverse?

2. Methodology

AoT-PsyPhyBENCH: A Psychophysically Validated Benchmark

The authors introduce AoT-PsyPhyBENCH, a benchmark derived from a psychophysical study by Hanyu et al. (2023).

• Source Data: 360 three-second clips from the Moments in Time dataset.
• Human Baseline: 10 human participants viewed clips in both forward and reversed directions.
• Filtering Criteria: The benchmark excludes the "Reciprocal" (cyclic motion) category, which humans find ambiguous (61.1% accuracy). It retains only five categories of irreversible processes where humans achieve high accuracy (>80%):
  1. Proceed: Forward locomotion.
  2. Fall: Free-fall/ballistic motion (gravity).
  3. Diffusion: Explosions/diffusion (entropy).
  4. Division: Manual division of materials.
  5. Put: Manual addition/construction.
• Final Dataset: 212 unique clips (424 total videos including reversed versions), ensuring low ambiguity and a reliable human-model comparison.

Experimental Setup

The authors evaluated a diverse range of VLMs across four experimental settings:

1. Zero-Shot: Standard prompting without examples.
2. Few-Shot: Providing 2–4 demonstration examples.
3. Reasoning Effort Ablation:
   • Proprietary Models: Adjusting "reasoning effort" parameters (Low/Medium/High) for models like GPT-5 and Gemini-2.5-Pro.
   • Open-Weight Models: Simulating reasoning via Chain-of-Thought (CoT) prompting (Simple vs. Multi-step).
4. Supervised Fine-Tuning (SFT): Training open-weight models (Qwen2-VL, Qwen2.5-VL) on 1k–5k pairs of forward/reversed videos.

Models Evaluated:

• Proprietary: GPT-4o, GPT-4.1, o3, o4-mini, GPT-5, Gemini-2.5-Pro.
• Open-Weight: Qwen2-VL, Qwen2.5-VL, QVQ-72B-Preview, Cosmos-Reason1-7B.

3. Key Results

Performance Gap

• Human Baseline: Humans achieve 89.2% overall accuracy.
• VLM Performance: Even the best-performing model (GPT-4.1) achieves only 60.1% accuracy.
• Random Baseline: Most open-weight models hover near 50% (random chance), indicating a complete lack of temporal understanding.

Failure of Reasoning Mechanisms

• Reasoning vs. Non-Reasoning: Explicit reasoning capabilities (e.g., Chain-of-Thought, "thinking" models like o3/o4-mini) do not improve performance. In some cases, they degrade it.
• Reasoning Effort: Increasing the "reasoning effort" in proprietary models (e.g., GPT-5, Gemini-2.5-Pro) often reduces accuracy and amplifies biases. For instance, Gemini-2.5-Pro performed best with low effort (61.1%) and worse with high effort (58.7%).
• CoT Prompting: Manually prompting models to "think step-by-step" failed to correct errors. Instead, models often generated plausible-sounding but incorrect reasoning that reinforced their initial wrong guess (e.g., assuming a reversed explosion was forward because the model "saw" an explosion).

Label Prediction Bias

• Forward Bias: Most models exhibit a strong bias toward predicting "Forward," even when the video is reversed.
  • Example: GPT-4o predicted 87% of clips as Forward.
  • Consequence: This leads to extremely low Backward F1 scores (often <30%), meaning models fail to detect reversed videos.
• Exception: Cosmos-Reason1-7B showed a "Backward Bias" (likely due to training on reversed clips), but still failed to achieve high overall accuracy.

Fine-Tuning and Frame Rates

• SFT Failure: Supervised Fine-Tuning on 5,000 examples did not push accuracy above chance (~50%). It only mitigated extreme label bias but did not teach the models the underlying physics.
• Frame Rate (FPS): Increasing FPS from 2 to 30 (full video) only marginally improved GPT-4.1's accuracy (60.1% → 60.6%), proving the deficit is not due to missing temporal information but a lack of physical reasoning.

4. Key Contributions

1. AoT-PsyPhyBENCH: The first psychophysically validated benchmark for temporal direction judgment, filtering out ambiguous cyclic motions to ensure a reliable human-model comparison.
2. Systematic Evaluation: The first comprehensive evaluation of modern VLMs (including reasoning and non-reasoning models) on physical irreversibility, revealing that current models lack the inductive biases for temporal continuity.
3. Open Resources: Release of the benchmark, evaluation scripts, and model outputs to facilitate research into physical and temporal reasoning.

5. Significance and Conclusion

The paper highlights a fundamental gap in current multimodal AI. While VLMs capture rich visual-semantic correlations, they lack the inductive biases required to understand physical laws (gravity, entropy, causality) and temporal continuity.

• Implication: Current "reasoning" capabilities in VLMs are superficial; they cannot compensate for a lack of genuine physical world understanding.
• Future Direction: The authors argue that future progress requires moving beyond statistical pattern recognition and instruction following toward systems that internalize the physics of the world. The release of AoT-PsyPhyBENCH provides a rigorous tool to measure this progress.

In summary, despite the apparent simplicity of the task (determining if a video is played forward or backward), current state-of-the-art VLMs perform near chance, failing to recognize violations of physical laws that humans detect almost instantly.