Technical Summary: When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Merging

1. Problem Statement

Model merging combines multiple fine-tuned models into a single unified model by operating directly in weight space, offering a lightweight alternative to retraining. While existing methods primarily focus on resolving conflicts between task updates (e.g., via masking or pruning), they often overlook a specific failure mode: spectral over-accumulation.

The authors identify that when multiple tasks share aligned spectral directions (i.e., overlapping singular vectors), a simple linear combination of task matrices repeatedly aggregates these shared components. This phenomenon leads to:

• Singular-value inflation: The singular values corresponding to shared subspaces become disproportionately large.
• Spectral imbalance: The merged model becomes dominated by a few top spectral subspaces containing shared knowledge, while task-specific components in remaining subspaces are suppressed.
• Performance degradation: Contrary to the intuition that shared knowledge is beneficial, its over-counting biases the model and reduces downstream performance across vision and language benchmarks.

2. Methodology: Singular Value Calibration (SVC)

To address spectral over-accumulation without requiring additional data, training, or optimization, the authors propose Singular Value Calibration (SVC), a post-processing method.

Theoretical Foundation

The method relies on an output-space projection analysis. By decomposing the merged task matrix $\Delta W_{merge}$ via Singular Value Decomposition (SVD), the authors define a shared output-space basis $\{u_r\}$. They project individual task matrices onto this basis to measure how much each task contributes to specific spectral subspaces.

• Projection Coefficient ($s_i^r$): This metric quantifies the scaling of task $i$'s response in subspace $r$ after merging. If $s_i^r > 1$, it indicates that the merged update has amplified task $i$'s direction due to constructive accumulation from other tasks (spectral over-counting).
• Inflation Mechanism: The paper demonstrates that positive cross-task inner products in the output space lead to $s_i^r > 1$, causing the merged singular value $\sigma_r$ to exceed the projection-optimal magnitude required to preserve individual task capabilities.

Algorithm Steps

1. Basis Construction: Compute the SVD of the merged task matrix $\Delta W_{merge} = U\Sigma V^\top$ to obtain the left singular vectors $\{u_r\}$ as the shared coordinate system.
2. Overlap Quantification: For each subspace $r$ and task $i$, calculate the projection coefficient $s_i^r = \frac{\langle a_{merge}^r, a_i^r \rangle}{\|a_i^r\|_2^2}$, where $a^r$ represents the response along $u_r$.
3. Calibration Factor: Aggregate these coefficients across tasks to derive a calibration factor $\gamma_r$ for each subspace. This factor is designed to down-weight subspaces exhibiting strong over-counting (where many $s_i^r$ are large).
   • $\gamma_r = K / \sum_{i=1}^K \max(\alpha, s_i^r)$, where $\alpha$ is a hyperparameter (default $1/K$) preventing instability.
4. Rescaling: Adjust the singular values while keeping the singular vectors (directions) fixed: $\tilde{\sigma}_r = \gamma_r \sigma_r$.
5. Reconstruction: Reconstruct the calibrated merged matrix $\tilde{\Delta W}_{merge} = \sum \tilde{\sigma}_r u_r v_r^\top$ and update the pre-trained weights.

3. Key Contributions

• Identification of Spectral Over-Counting: The paper isolates a specific failure mode in model merging where redundant aggregation of shared knowledge inflates singular values in dominant subspaces, distinct from traditional weight conflicts.
• SVC Method: A training-free and data-free post-processing technique that quantifies output-space overlap and rescales singular values to restore spectral balance. It acts as a plug-in module compatible with existing merging baselines.
• Theoretical and Empirical Validation: The authors provide a theoretical analysis linking projection mismatches to singular-value inflation and demonstrate that this phenomenon persists across various merging strategies (e.g., Task Arithmetic, TIES, DARE).

4. Experimental Results

The authors evaluated SVC on diverse benchmarks using Vision (ViT-B/32, ViT-B/16, ViT-L/14) and Language (BERT, T5, LLaMA2-7B, T0) models.

• Computer Vision (CV): On an 8-task classification benchmark, SVC improved the baseline Task Arithmetic (TA) by 13.0% (from 68.9% to 81.9% average accuracy). It also yielded consistent gains over other strong baselines like TIES, DARE, and TSV-M.
• Natural Language Processing (NLP): SVC improved performance across classification and generative tasks. For instance, on the T0 model with IA3 fine-tuning, SVC improved TA by 12.3%. On LLaMA2-7B, it showed significant gains in AlpacaEval and GSM8K benchmarks.
• Generality: The method proved effective across different model sizes and architectures. It also functioned as a "preference optimization" tool, allowing the merged model to prioritize specific tasks by adjusting calibration based on a target task.
• Efficiency: SVC introduces minimal overhead, requiring only a single offline SVD and reconstruction step without gradient computation or auxiliary data.

5. Significance and Claims

The paper claims that model merging failures are not solely due to conflicting updates but also stem from the redundant accumulation of aligned components. By shifting the focus from conflict resolution to spectral balance, SVC offers a simple yet effective mechanism to correct the "over-counting" of shared knowledge.

The authors position SVC as a complementary tool that can be applied on top of existing merging methods (including spectral-space methods like Iso-CTS) to further refine performance. They emphasize that while other methods may address conflicts or rank collapse, SVC specifically targets the inflation of dominant subspaces caused by constructive interference, thereby restoring a more balanced spectrum and improving generalization across multiple tasks.