Technical Summary: FRISM

Problem Statement

Vision-Language Models (VLMs) have achieved remarkable success in perception tasks but often lack the complex reasoning capabilities found in Large Reasoning Models (LRMs). While model merging offers a parameter-efficient way to inject reasoning capabilities into VLMs without costly post-training, existing methods operate at a coarse-grained layer level. These approaches typically treat entire layers as atomic units, leading to a significant trade-off: enhancing reasoning often degrades visual perception capabilities, and vice versa. Current layer-wise merging strategies (e.g., FRANK, IP-Merging) struggle to disentangle reasoning and visual representations within the same layer, resulting in suboptimal performance on both fronts.

Methodology

The authors propose FRISM (Fine-grained Reasoning Injection via Subspace-level model Merging), a framework that shifts the merging paradigm from the layer level to the subspace level.

Core Hypothesis

FRISM is motivated by the observation that reasoning capabilities and visual representations reside in distinct subspaces within the parameter space, even within a single layer. Consequently, a uniform layer-wise scaling coefficient is insufficient; it inevitably couples beneficial reasoning (which may require high magnitude) with visual noise.

Technical Framework

FRISM operates in two stages:

1. Decomposition & Initialization (Stage 1):

   • Instead of treating the LRM task vector ($\tau_{lrm}$) as a monolithic update, FRISM decomposes it into orthogonal subspaces using Singular Value Decomposition (SVD): $U, S, V^\top = \text{SVD}(\tau_{lrm})$.
   • The singular vectors ($U, V$) and singular values ($S$) are frozen to preserve the semantic direction of the reasoning task.
   • A lightweight, learnable gating parameter $g$ is introduced for each subspace to adaptively modulate the injection intensity. The effective singular values are computed as $S_{eff} = \text{Sigmoid}(g) \odot S$.
   • The merged layer is formulated as: $\theta_{merged} = \theta_{vlm} + \lambda_{lrm} \cdot U (S_{eff}) V^\top$.

2. Injection & Training (Stage 2):

   • FRISM employs a label-free self-distillation strategy with a dual-objective optimization function, eliminating the need for scarce multimodal reasoning datasets.
   • Visual Preservation ($L_{distill}$): Minimizes the Kullback-Leibler (KL) divergence between the output distributions of the original VLM (teacher) and the merged model (student) on unlabeled visual perception data (e.g., VizWiz). This ensures the merged model retains robust visual understanding.
   • Reasoning Maximization ($L_{inject}$): Maximizes the spectral magnitude of the injected subspaces ($-\sum \|S_{eff}\|^2$). This term encourages the absorption of reasoning priors defined by the LRM task vectors.
   • Total Objective: $L = L_{distill} + \alpha \cdot L_{inject}$. This acts as a spectral filter, automatically discovering subspaces that offer the optimal trade-off between reasoning injection and visual preservation.

Key Contributions

1. Subspace-Level Merging Paradigm: The paper rethinks model merging by moving from coarse-grained layer-level operations to fine-grained subspace-level modulation. This allows for the precise identification and integration of reasoning-dominant components without disrupting visual capabilities.
2. Adaptive SVD Modulation: FRISM is the first to propose decomposing LRM task vectors via SVD for reasoning injection, enabling adaptive reweighting of subspaces under a visual-preservation objective.
3. Label-Free Self-Distillation: The authors design a training strategy that relies on dual-objective optimization (KL divergence minimization and spectral magnitude maximization) to achieve stable reasoning transfer without requiring labeled VL reasoning datasets.
4. Plug-and-Play Efficiency: The method is highly parameter-efficient, requiring only the training of gating parameters while keeping the base model and decomposed singular vectors frozen.

Experimental Results

Extensive experiments were conducted across various model scales (3B, 7B, 32B) and architectures (Qwen2.5-VL, ThinkLite-VL, InternVL) on diverse benchmarks.

• Reasoning Performance: FRISM consistently outperforms existing merging baselines (Task Arithmetic, Ties-Merging, DARE, IP-Merging) and often surpasses the base VLM significantly. For instance, on the 7B scale, FRISM achieved a reasoning score of 49.4 on average, surpassing the base VLM by 2.0 points and approaching the performance of high-cost post-trained models like ThinkLite-VL-7B.
• Visual Preservation: Unlike other methods that suffer from performance degradation in visual perception tasks, FRISM largely preserves or even slightly improves visual capabilities. On VL perception benchmarks, FRISM showed the least performance drop among all methods.
• Efficiency: FRISM is highly efficient, requiring only 0.53M trainable parameters and 0.49 hours of training time on a single GPU. This is significantly more efficient than post-training methods like DRIFT, which require ~7000M parameters and ~2 hours, while achieving comparable or superior reasoning performance.
• Ablation Studies: Experiments confirmed that both the SVD decomposition process and the reasoning maximization loss term are critical. Removing SVD (reverting to layer-wise) or removing the injection loss resulted in performance close to the base VLM, validating the necessity of fine-grained subspace control and active reasoning injection.

Significance and Claims

The paper claims that FRISM effectively bridges the gap between powerful LRMs and VLMs by resolving the inherent conflict between injecting reasoning and preserving visual perception. By treating reasoning injection as a subspace-level filtering problem rather than a layer-level balancing act, FRISM offers a versatile, plug-and-play solution that is compatible with various VLM architectures.

The authors emphasize that their approach provides an efficient alternative to resource-heavy post-training, achieving state-of-the-art reasoning performance while maintaining visual robustness without the need for expensive labeled reasoning datasets. The work suggests that the capabilities of models are not uniformly distributed but are distinctively encoded within specific rank subspaces, a finding that enables more precise and effective model merging.

Note: The paper acknowledges limitations, including increased inference time due to longer output sequences (overthinking) and the current inability to generalize across models of different sizes or structures. It also notes that the reasoning objective relies on a proxy (subspace magnitude) rather than direct supervision due to data scarcity.