Which Way Did It Move? Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Technical Summary: Diagnosing and Overcoming Directional Motion Blindness in Video-LLMs

Problem: Directional Motion Blindness
Despite rapid progress in temporal video understanding, Video Large Language Models (Video-LLMs) exhibit a fundamental failure in perceiving signed image-plane motion direction. On simple synthetic videos where a single object moves strictly left, right, up, or down, most Video-LLMs perform near random chance (25%), even when they correctly identify the object's appearance (e.g., color). The authors term this phenomenon directional motion blindness.

Through mechanistic analysis, the authors diagnose the root cause not as a lack of visual perception, but as a direction binding gap. They demonstrate that motion direction information remains linearly accessible throughout the Video-LLM pipeline—from the vision encoder, through the projector, to the LLM hidden states. However, the final readout token fails to reliably bind this internal signal to the correct verbal answer option (e.g., mapping the perceived "leftward" signal to the text option "Left"). This gap persists across diverse architectures and scales, indicating a structural limitation in how these models convert perceptual signals into language-level responses.

Methodology
The paper introduces a controlled diagnostic framework and a training-only solution to address this gap.

1. MODIRECT Dataset Family:
   To isolate and evaluate motion direction understanding, the authors introduce MODIRECT, a dataset family comprising three subsets:

   • MODIRECT-INST: An instruction-tuning dataset derived from the simplest synthetic domain (Primitive-on-Syn), used to train models on direction-related queries.
   • MODIRECT-SYNBENCH: A controlled synthetic benchmark with a $2 \times 2$ factorial design (Foreground: Primitive vs. Cutout; Background: Synthetic vs. Real). This allows for rigorous evaluation of in-domain performance and out-of-domain (OOD) generalization.
   • MODIRECT-REALBENCH: A real-world benchmark curated from existing datasets (Something-Something-V2, KTH, TOMATO) to test transferability to natural videos without real-world direction supervision during training.

2. Diagnostic Analysis:
   The authors employ linear probing and concept vector analysis (difference-in-means) to trace the motion direction signal.

   • Findings: While instruction tuning on the source domain (Primitive-on-Syn) successfully closes the binding gap for that specific domain, performance degrades significantly on complex OOD domains (e.g., Cutout-on-Real).
   • Root Cause of OOD Failure: Analysis of motion direction concept vectors reveals that while the orientation (direction in representation space) aligns across domains after tuning, the magnitude of the signal drops sharply in complex visual environments. The model learns the correct direction structure, but the signal becomes too weak to be reliably read out by the language head.

3. DeltaDirect:
   To address the magnitude deficit, the authors propose DeltaDirect, a training-only auxiliary objective applied at the projector level.

   • Mechanism: DeltaDirect predicts normalized 2-D motion vectors from adjacent-frame feature deltas ($\delta_t = F_{t+1} - F_t$) before the features enter the LLM.
   • Supervision: It uses analytically available motion vectors from synthetic data as supervision targets. The loss encourages the projector output to carry a stronger signed displacement signal.
   • Inference: The auxiliary branch is discarded after training, leaving the model architecture, input format, and decoding procedure unchanged.

Key Contributions

1. Identification of Directional Motion Blindness: The paper establishes that Video-LLMs fail to distinguish basic signed motion directions despite strong appearance recognition, diagnosing this as a direction binding gap rather than a perception failure.
2. Diagnosis of Generalization Failure: Through concept vector analysis, the authors show that instruction tuning aligns direction orientations across domains but fails to maintain signal magnitude in complex visual conditions, leading to OOD generalization gaps.
3. MODIRECT and DeltaDirect: The introduction of a controlled dataset family for motion direction and a diagnosis-driven, projector-level objective (DeltaDirect) that strengthens motion signals without altering inference.

Results

• Synthetic Performance: On MODIRECT-SYNBENCH, instruction tuning alone improves accuracy from ~26% to ~79%. Adding DeltaDirect further boosts the average to 85.4%, with a significant gain of 11.2 points on the hardest domain (Cutout-on-Real).
• Real-World Transfer: Without seeing real-world videos during training, DeltaDirect improves real-world motion direction accuracy on MODIRECT-REALBENCH by 21.9 points over the vanilla baseline (from 43.1% to 65.0%).
• General Video Understanding: Crucially, DeltaDirect preserves or improves performance on standard general video understanding benchmarks (e.g., MVBench, NExT-QA), demonstrating that the motion supervision does not trade off broader video capabilities.
• Scalability: The approach is effective across different backbones (LLaVA-Video, LLaVA-OneVision, Qwen3-VL) and training regimes (LoRA and Full Fine-Tuning).

Significance and Claims
The paper claims that the failure of Video-LLMs in motion direction is not due to a lack of visual data or model capacity, but a specific failure in binding internal perceptual signals to verbal outputs. By localizing this failure to a "magnitude deficit" in complex domains, the authors argue that diagnosis-driven supervision is a viable path to closing perception-language gaps. DeltaDirect demonstrates that explicitly supervising motion vectors at the visual-language interface can significantly improve motion understanding while maintaining general video capabilities, suggesting that future Video-LLMs may benefit from similar targeted interventions beyond simple scaling of models or datasets. The work highlights the importance of controlled synthetic benchmarks for diagnosing specific perceptual primitives that are often entangled in real-world evaluations.