Reasoning Resides in Layers: Restoring Temporal Reasoning in Video-Language Models with Layer-Selective Merging

1. Problem Statement

The Core Issue: While Multimodal Large Language Models (VLMs) are built by attaching visual encoders to pre-trained Language Models (LLMs) to gain perceptual capabilities, this adaptation often degrades the temporal reasoning (TR) abilities inherited from the original text-only backbone.

• The Trade-off: Existing VLMs excel at spatial perception (identifying objects) but fail at temporal logic (understanding event order, causality, and state changes).
• The Cause: The paper attributes this to a mismatch in training objectives. Multimodal pipelines prioritize spatial alignment and object detection, often eroding the internal representations required for higher-level temporal abstraction and causal inference.
• Limitations of Current Solutions:
  • Training-based methods (e.g., Direct Preference Optimization) require massive computational resources and additional supervised data.
  • Existing training-free merging approaches typically merge the VLM and its text backbone uniformly across all layers. This coarse-grained strategy is unreliable, often failing to recover reasoning without sacrificing visual perception.

2. Methodology: MERIT

The authors propose MERIT (Merging for Enhanced Reasoning In Temporal Tasks), a training-free, task-driven model merging framework. Instead of treating the model as a homogeneous block, MERIT operates at the layer level.

Key Components:

1. Layer-Selective Merging:

   • MERIT searches for an optimal "recipe" that determines which self-attention layers of the VLM should be replaced or interpolated with parameters from the paired text-only LLM backbone.
   • The goal is to selectively restore the reasoning capabilities of the text backbone while preserving the visual grounding of the VLM.

2. Objective Function:
   The framework optimizes a specific objective function $F(g)$ that balances reasoning recovery with perception preservation:
   $$F(g) = \text{Acc}_{TR}(g) - \lambda \cdot D_{TP}(g)$$

   • $\text{Acc}_{TR}(g)$: Accuracy on Temporal Reasoning tasks.
   • $D_{TP}(g)$: Degradation in Temporal Perception (relative to the original VLM).
   • $\lambda$: A hyperparameter controlling the penalty for perception loss.
   • This ensures the model improves reasoning without losing its ability to perceive temporal events.

3. Evolutionary Search (CMA-ES):

   • Since the search space of layer combinations is combinatorial and the objective is non-differentiable (requires evaluation), MERIT uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES).
   • Parameterization: A continuous gating vector $g \in [0, 1]^L$ is used, where each dimension corresponds to a self-attention layer.
   • Interpolation: For a selected layer, parameters are merged via directional interpolation: $\theta = \alpha \theta_{VLM} + (1-\alpha) \theta_{Text}$ (or vice versa), where $\alpha$ is a fixed interpolation weight.
   • Efficiency: The search is conducted on a small, targeted subset of the Video-MME benchmark (55 TP + 177 TR examples), making it computationally efficient compared to full-benchmark tuning.

3. Key Contributions

• Novel Framework: Introduction of MERIT, the first training-free framework to restore temporal reasoning in VLMs via layer-selective merging rather than uniform averaging.
• Discovery of Reasoning Layers: The paper provides empirical evidence that temporal reasoning is not uniformly distributed across the network. MERIT identifies specific layers that are disproportionately critical for reasoning.
• Task-Driven Optimization: The use of an explicit objective function that penalizes perception degradation ensures that reasoning recovery does not come at the cost of visual understanding.
• Mechanism Analysis: Through interventional masking and frame-level attribution, the authors demonstrate how MERIT works: it shifts the model's focus from misleading local visual cues to temporally distant, causally relevant evidence.

4. Experimental Results

The authors evaluated MERIT on three representative VLMs (LongVA-7B, InternVL3-8B, Qwen3-VL-4B) across multiple benchmarks.

• Performance Gains:

  • Temporal Reasoning (TR): MERIT consistently improved TR scores significantly (e.g., +23.9% on LongVA-7B, +10.8% on InternVL3-8B) compared to the base VLM.
  • Temporal Perception (TP): Unlike uniform merging, MERIT preserved or even slightly improved temporal perception scores.
  • Generalization: Recipes discovered on a small subset of Video-MME generalized effectively to four other distinct benchmarks (LongVideoBench, LVBench, MMBench-Video, Video-Holmes).

• Ablation Studies:

  • vs. Uniform Merging (ALL-LAYER): Uniform merging yielded inconsistent results and often degraded perception.
  • vs. Random Selection (RANDOM-K): Randomly selecting layers performed significantly worse than MERIT, proving that which layers are selected is more important than how many.

• Interventional Analysis:

  • Layer Masking: Masking the layers selected by MERIT caused a drastic drop in reasoning performance (e.g., ~50% drop) while having a much smaller impact on general performance, confirming these layers are the "locus" of reasoning.
  • Frame-Level Attribution: Case studies showed that the base VLM often anchored on salient but misleading local events (e.g., a phone call). MERIT shifted attention to temporally separated but causally linked events (e.g., a victim being restrained earlier in the video), enabling correct causal inference.

5. Significance and Conclusion

• Scalability: MERIT offers a training-free solution to a major bottleneck in multimodal AI. It avoids the high costs of retraining or collecting massive new datasets.
• Theoretical Insight: The work challenges the view that reasoning and perception are inextricably linked in a way that prevents recovery. Instead, it suggests that reasoning capabilities reside in specific, localized subnetworks that can be selectively restored.
• Future Impact: This approach provides a scalable path for enhancing specialized reasoning capabilities (beyond just temporal) in future multimodal systems by targeting specific architectural components rather than retraining the entire model.

In summary, the paper demonstrates that reasoning resides in specific layers and that by selectively merging these layers from a text-only backbone into a VLM, one can restore complex temporal reasoning capabilities without sacrificing the model's visual perception.