Technical Summary: What Does a Temporal Benchmark Score Measure?

Problem Statement

Current temporal video question answering (VQA) benchmarks aim to certify that Video Large Language Models (Video VLMs) understand time. However, the paper argues that aggregate benchmark scores conflate two distinct questions:

1. The Task Question: Does the specific question require watching motion across multiple frames in a specific order, or can it be answered from a single frame or language priors?
2. The Channel Question: When a task does require temporal order, does the model recover that order from the visual pixels (motion) or from the positional encoding (e.g., Rotary Positional Embeddings, RoPE)?

Standard diagnostics, such as frame-shuffle sensitivity and multi-frame gain, only address the Task Question. They fail to reveal the Channel Question. Consequently, two models with similar aggregate scores may resolve temporal order through entirely different mechanisms (e.g., one relying on visual sequence, the other on positional indices), making them non-interchangeable despite similar performance metrics.

Methodology

The authors propose a decomposition framework to audit temporal understanding by manipulating input channels independently and in conflict.

1. Input Channels and Manipulations

The study identifies three primary channels for temporal information in Video VLMs:

• Visual Frame Order: The sequence of pixels/motion.
• In-Context Timestamps: Explicit time labels provided in the prompt.
• Positional Encoding (RoPE): The internal indices assigned to tokens.

The authors construct specific intervention conditions:

• Information Ladder: Evaluates performance on blank pixels, single repeated frames, and full videos to isolate multi-frame dependence (Task Question).
• Conflict Conditions: The core innovation is the Reversal-Drop protocol. In the rev-corr condition, the visual frames are reversed (pixels indicate backward time), but the RoPE positions are held in their original forward order (positions indicate forward time).
  • If a model is Position-Dominant, it ignores the reversed pixels and answers based on the forward RoPE, maintaining accuracy.
  • If a model is Visual-Sequence-Dominant, it follows the reversed pixels, causing accuracy to collapse when scored against the forward ground truth.

2. Models and Benchmarks

• Models: Three families spanning different architectures:
  • Molmo2: Bidirectional attention over visual tokens with 1D sequential RoPE.
  • Qwen3-VL: Causal attention with 3D MRoPE (temporal, height, width axes) and DeepStack (intermediate ViT feature injection).
  • PerceptionLM: Causal attention with 1D RoPE and no DeepStack.
• Benchmarks: TempCompass (with paired reverse videos for ground truth verification) and TVBench (using the label-free Reversal-Drop metric).

3. Mechanistic Validation (Activation Patching)

To ensure the behavioral split is a genuine internal property rather than an artifact of the conflict protocol, the authors use activation patching. They capture residual stream activations from a clean, forward-order run and inject them into a shuffled/reversed run at specific layers. This localizes where the model can "repair" temporal understanding (e.g., at the first LM layer vs. deeper layers where DeepStack injects features).

Key Results

1. Dissociation of Channels

The study reveals a stark split in how models resolve temporal order:

• Molmo2 (Position-Dominant): When visual frames are reversed but RoPE is corrected to forward order, Molmo2's accuracy remains near baseline. It reads order off the positional indices, effectively ignoring the reversed visual content.
• Qwen3-VL (Visual-Sequence-Dominant): Under the same rev-corr condition, Qwen3-VL's accuracy collapses significantly. It reads order from the visual pixels, answering the reversed event even when the positional encoding suggests otherwise.
• PerceptionLM: Occupies an intermediate position, showing partial reliance on positions but less than Molmo2.

2. Limitations of Aggregate Scores

• Multi-Frame Gain: Both Molmo2 and Qwen3-VL show similar gains (+0.20) when moving from single frames to full videos, indicating both genuinely need multiple frames.
• Shuffle Sensitivity: Both models suffer when frames are shuffled, but for different reasons. Molmo2 is highly sensitive to RoPE shuffling (dropping ~14-18 points), while Qwen3-VL is less sensitive to RoPE shuffling alone but highly sensitive to pixel shuffling.
• Conclusion: Aggregate scores and standard multi-frame audits fail to distinguish these opposing mechanisms. Two models can have identical scores but fail on opposite types of inputs.

3. Mechanistic Findings

• Recovery Boundaries: Activation patching confirms the behavioral split is internal.
  • For Molmo2 (bidirectional), patching the first LM layer fully recovers performance from shuffled inputs.
  • For Qwen3-VL (causal), patching the first layer leaves a residual error. Full recovery requires patching up to the DeepStack injection depth (layers 0–2), where intermediate ViT features are re-injected. This confirms the split is architectural, not an artifact of the conflict construction.
• Timestamps: Timestamp reliance varies by model and task. Removing timestamps hurts Qwen3-VL more than Molmo2 on specific subsets, but neither model relies on them universally.

Significance and Contributions

The paper claims three primary contributions:

1. A Benchmark Audit: It demonstrates that temporal scores conflate task requirements (motion dependence) with channel usage (order resolution). Standard diagnostics (shuffle sensitivity, multi-frame gain) only answer the task question.
2. A Label-Free Screen (Reversal-Drop): The authors introduce the Reversal-Drop metric (accuracy loss when pixels are reversed but RoPE is held forward). This metric separates Position-Dominant models from Visual-Sequence-Dominant models without requiring new annotations or paired reverse labels (though paired labels allow for more granular analysis).
3. Evidence of Mechanistic Divergence: The study provides evidence that models with similar benchmark scores are not interchangeable. Molmo2 and Qwen3-VL resolve temporal order via opposite channels (RoPE vs. Pixels). This distinction is validated by activation patching, proving it is a real internal property of the network architecture.

Recommendations

The authors recommend that temporal evaluation should not rely on a single aggregate score. Instead, evaluations should report:

• Multi-frame gain: To determine if a task requires multiple frames.
• Shuffle sensitivity: To determine if a task requires ordered frames.
• Reversal-Drop (or conflict protocols): To determine which input pathway (visual vs. positional) the model uses to resolve order.

This decomposition exposes failure modes hidden by aggregate metrics, ensuring that models are evaluated on the specific mechanisms required for robust temporal understanding.