DRIFT: Transferring Reasoning Priors for Efficient MLLM Fine-Tuning

1. Problem Statement

Multimodal Large Language Models (MLLMs) have made significant strides in perception and alignment but consistently lag behind text-only Large Language Models (LLMs) in reasoning capabilities (e.g., mathematical and logical reasoning).

• Current Bottlenecks: Bridging this gap typically requires:
  1. Large-scale Multimodal Supervised Fine-Tuning (SFT): Collecting high-quality Chain-of-Thought (CoT) data for images is expensive and labor-intensive.
  2. Reinforcement Learning (RL): Adds computational overhead and training instability.
• The Limitation of Model Merging: A promising alternative is parameter-space model merging (interpolating weights between a text-only reasoning expert and an MLLM). However, the authors demonstrate that naive merging is fragile. It often fails to transfer reasoning effectively and can severely degrade multimodal perception and alignment, particularly when the underlying models (e.g., Qwen series) have divergent parameter distributions.

2. Methodology: DRIFT

The authors propose DRIFT (Directional Reasoning Injection for Fine-Tuning), a lightweight, gradient-based method that transfers reasoning knowledge without destabilizing the model.

Core Concept

Instead of directly interpolating weights in the parameter space, DRIFT operates in the gradient space. It treats the difference between a text-only reasoning expert and a multimodal model as a "reasoning direction" (prior) and uses this to bias the optimization trajectory during standard SFT.

Key Steps

1. Pre-computation of Reasoning Prior ($\Delta$):

   • Calculate the parameter difference between a text-only reasoning model ($\phi_{reason}$) and the target multimodal model ($\phi_{VL}$):
     $$\Delta = \phi_{reason} - \phi_{VL}$$
   • This difference is restricted to reasoning-relevant modules (MLP projections, attention projections, and normalization layers).
   • $\Delta$ is computed once and stored on the CPU.

2. Gradient Injection during SFT:

   • During the supervised fine-tuning of the MLLM on a small amount of multimodal data, the standard gradient ($g$) is modified.
   • The guided gradient ($\tilde{g}$) is computed as:
     $$\tilde{g} = g + \alpha \cdot \text{scale}(g, \Delta)$$
   • Scaling Strategies: The paper explores three variants:
     • Absolute: Directly adds $\Delta$.
     • Grad-Norm: Aligns the update direction with $\Delta$ while preserving the magnitude of $g$.
     • Grad-Norm w/ Adaptive $\alpha$: Dynamically adjusts the injection strength based on the cosine similarity between the current gradient and the prior ($\Delta$).

3. Efficiency:

   • DRIFT introduces no additional trainable parameters.
   • It only modifies the backward pass (gradient calculation), leaving the forward pass unchanged.
   • It requires only a small curated dataset (4K examples) compared to the massive datasets needed for standard SFT or RL.

3. Key Contributions

1. Empirical Analysis of Merging: The authors rigorously show that parameter-space merging is not a "free lunch." It is highly sensitive to model families; while it works for some (e.g., LLaMA/Mistral), it often degrades performance in others (e.g., Qwen) due to parameter misalignment.
2. DRIFT Framework: Introduction of a novel, lightweight method that injects reasoning priors via gradient guidance rather than weight interpolation, preserving multimodal alignment while enhancing reasoning.
3. Efficiency and Performance: Demonstrated that DRIFT achieves reasoning capabilities comparable to training-intensive methods (SFT + RL) but with orders of magnitude less data and compute time (approx. 2 hours vs. >1 day).

4. Experimental Results

The method was evaluated on benchmarks including MathVista, MathVision, MathVerse, WeMath, and LogicVista.

• vs. Parameter Merging: DRIFT consistently outperforms naive merging and advanced merging techniques (Task Arithmetic, TIES, DARE). While merging often reduced scores (e.g., -2.1 on MathVista for Qwen2.5-VL), DRIFT improved them (+2.0).
• vs. Standard SFT: DRIFT outperforms standard SFT trained on the same small dataset, proving the reasoning prior provides valuable complementary signals.
• vs. Training-Intensive Methods: DRIFT matches or surpasses heavy methods like R1-OneVision and X-Reasoner (which require days of training and massive CoT datasets) while completing in ~2 hours with only 4K high-quality examples.
• Generalization:
  • Perception: Unlike standard SFT which sometimes degraded general perception (e.g., on RealWorldQA), DRIFT preserved or slightly improved multimodal perception capabilities.
  • Robustness: DRIFT showed better robustness to noisy data compared to standard SFT.
  • Generality: The method worked across different backbone/expert pairings (e.g., LLaVA + DART, Qwen + DeepSeek-R1).

5. Significance

• Democratization of Reasoning: DRIFT lowers the barrier to entry for creating reasoning-capable MLLMs. It eliminates the need for expensive, large-scale multimodal CoT data collection and multi-day RL training.
• Stability: It solves the instability issues associated with naive model merging, offering a robust way to combine the "vision" of MLLMs with the "reasoning" of text-only experts.
• Practicality: The method integrates seamlessly into existing SFT pipelines (e.g., via LLaMA-Factory) with negligible overhead, making it highly scalable for both research and industrial deployment.

In conclusion, DRIFT demonstrates that gradient-space priors are a superior mechanism for transferring reasoning capabilities compared to parameter-space merging, offering a highly efficient, stable, and effective path to bridging the reasoning gap in multimodal models.