Technical Summary: Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning

1. Problem Statement

Video Large Language Models (Video LLMs) have achieved strong performance on standard benchmarks, yet recent studies indicate that high accuracy often stems from "shortcuts" rather than genuine spatiotemporal understanding. Models frequently rely on static cues (e.g., single-frame features) or language priors to answer questions, failing to track how events unfold over time. This issue is exacerbated during Reinforcement Learning (RL) post-training. Standard RL methods, such as GRPO, typically employ correctness-only rewards. If a model can answer a question correctly using a static shortcut, it receives a high reward without ever learning to process dynamic video content. Consequently, correctness-only RL may reinforce these shortcut policies, improving benchmark scores while potentially weakening the model's sensitivity to dynamic evidence.

Existing attempts to address this, such as using perturbed videos (e.g., shuffled frames or temporal reversals) to shape rewards, often treat the perturbed video only as a diagnostic signal for the original branch. They do not actively train the policy on the counterfactual input itself, limiting their ability to enforce consistent spatiotemporal reasoning.

2. Methodology: Counterfactual Relational Policy Optimization (CRPO)

The authors propose Counterfactual Relational Policy Optimization (CRPO), a dual-branch RL framework designed to improve spatiotemporal sensitivity. The core insight is that a model should respond predictably to controlled counterfactual visual worlds: if the visual evidence changes while the question remains fixed, the answer should change for dynamic questions (equivariance) and remain the same for static questions (invariance).

2.1 Dual-Branch Architecture

CRPO extends the standard GRPO framework by maintaining two parallel branches for each training prompt:

1. Original Branch: Processes the original video $X$ and question $Q$.
2. Counterfactual Branch: Processes a transformed video $X_T$ and the same question $Q$.

A Task Router (implemented as a text-only reasoning model) classifies the question into one of four types—Spatial, Temporal, Spatiotemporal, or Static—based on hypothetical transformation tests (e.g., "If we flip the world horizontally, would the answer change?"). Based on this classification, a specific transformation $T$ is applied:

• Spatial/Temporal/Spatiotemporal: Horizontal flip or temporal reversal.
• Static: A transformation is still applied, but the reward signal encourages answer invariance.

2.2 Counterfactual Relation Reward (CRR)

The key innovation is the Counterfactual Relation Reward (CRR), which links the two branches. Unlike prior methods that use perturbations only for reward shaping on the original branch, CRPO updates the policy using gradients from both branches.

The reward function combines:

• Correctness Reward ($R_{correct}$): Standard accuracy reward for the original branch.
• Behavioral Reward ($R_{behave}$): Applied to the counterfactual branch. It rewards the model if the answer changes for dynamic tasks ($o^T \neq o^*$) and remains correct for static tasks ($o^T = o^*$).
• Cross-Branch Relational Term ($R_{CRR}$):
  • For the original branch, $R_{CRR}$ rewards the policy if the counterfactual branch's answers exhibit the expected behavior (changing for dynamic, staying same for static), gated on the original answer being correct.
  • For the counterfactual branch, $R_{CRR}$ rewards the policy if the original branch is correct, gated on the counterfactual answer exhibiting the expected behavior.

This creates a symmetric mutual reward structure. The highest reward is achieved only when the model correctly answers the original video and demonstrates the correct relational behavior on the counterfactual video. This makes it difficult for shortcut policies (which would give the same answer regardless of the video transformation) to consistently maximize rewards across both branches.

2.3 Optimization and Null Option

CRPO uses a hybrid normalization strategy for advantages, computing the standard deviation jointly over both branches (2G values) to ensure stable training. To handle cases where the correct answer for a counterfactual video is not present in the original multiple-choice options (e.g., a reversed video implies a different direction), a "None of the above" (Null Option) is appended. Selecting this option for a dynamic task counts as a valid answer change, triggering the equivariance reward without requiring explicit counterfactual ground-truth labels.

3. Key Contributions

1. CRPO Framework: A simple dual-branch RL extension that uses Counterfactual Relation Rewards to enforce spatiotemporal sensitivity. It trains on both original and counterfactual branches, discouraging shortcut reliance without needing costly spatiotemporal annotations or counterfactual labels.
2. DyBench: A new paired counterfactual benchmark consisting of 3,014 videos (1,507 pairs) covering three sub-tasks: Reversible Dynamics (e.g., blooming vs. closing), Moving Direction, and Event Sequence.
3. Strict Evaluation Metric: The introduction of Pair Accuracy (P-Acc), which counts a pair as correct only if both the original and counterfactual videos are answered correctly. This metric prevents models from inflating scores by using fixed-answer shortcuts.

4. Experimental Results

Experiments were conducted on Qwen3-VL-4B and Qwen3-VL-8B backbones, comparing CRPO against standard GRPO, T-GRPO, and ArrowRL.

• Spatiotemporal Sensitivity: CRPO significantly outperforms all baselines on spatiotemporal-sensitive benchmarks. On Qwen3-VL-8B, CRPO improves DyBench P-Acc by +7.7 and TimeBlind I-Acc by +8.2 over the base model. The gains on paired metrics (P-Acc) are disproportionately larger than on standard accuracy, indicating a genuine reduction in shortcut reliance.
• General Performance: CRPO maintains competitive performance on general video understanding benchmarks (VideoMME, MVBench), demonstrating that enforcing spatiotemporal sensitivity does not degrade general capabilities.
• Ablation Studies:
  • Removing the cross-branch CRR reward or the counterfactual branch itself causes significant drops in P-Acc, confirming that the relational signal and dual-branch optimization are essential.
  • CRPO's gains cannot be attributed simply to increased data or compute (doubled rollouts), as controls with doubled data/rollouts failed to match CRPO's performance on paired metrics.
• Training Dynamics: CRPO shows slower initial convergence on correctness rewards compared to baselines, suggesting it suppresses early shortcut learning. However, it converges to similar final accuracy while maintaining a lower fraction of zero-advantage groups, indicating richer learning signals.

5. Significance and Claims

The paper claims that CRPO offers a direct training signal for spatiotemporal sensitivity that is difficult for static shortcuts to satisfy. By explicitly rewarding the relation between answers to factual and counterfactual videos, CRPO forces the model to ground its reasoning in the video's dynamic content rather than static priors.

The authors position DyBench and the Pair Accuracy metric as a necessary diagnostic tool to distinguish genuine spatiotemporal understanding from shortcut-based success. They conclude that while CRPO improves performance on dynamic tasks, it does so without sacrificing general video understanding, offering a step toward Video LLMs that are more sensitive to how visual states change over time. The work acknowledges limitations, such as the restriction to simple geometric/temporal transformations and the focus on short videos, leaving more complex semantic counterfactuals for future research.