Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging

Technical Summary: Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging

Problem Statement
While fine-tuning large language models (LMs) for specific tasks yields strong performance, maintaining separate models for each task is costly regarding storage and deployment. Model merging offers a solution by combining multiple task-specific models into a single multi-task model without additional training. However, existing merging methods often fail when applied to models fine-tuned with Low-Rank Adaptation (LoRA), resulting in significant performance degradation.

The authors argue that this degradation stems from parameter interference driven by the interaction between model parameters and data distributions. Specifically, when merging LoRA modules (e.g., $W_m = W_0 + B_1A_1 + B_2A_2$), the update term from one task ($B_2A_2$) acts as an unintended "perturbation" on the latent features of another task ($h_1$), shifting outputs away from the intended response. Prior data-free approaches focusing solely on task vector orthogonality or alignment often overlook this crucial interplay between latent features and parameter updates.

Methodology: OSRM
The paper proposes Orthogonal Subspaces for Robust model Merging (OSRM), a method that constrains the LoRA subspace prior to fine-tuning to minimize interference with out-of-task data distributions.

1. Core Objective: The method aims to reduce the interference term $\|B_2A_2 h_1\|$ by minimizing the Frobenius norm $\|A_2 H_1^\top\|_F$, where $H_1$ represents the latent feature matrix of samples from a different task ($T_1$).
2. Analytical Solution: To avoid trivial solutions (e.g., $A_2=0$) and maintain representation power, the authors impose a constraint that $A_2$ must have orthonormal rows ($A_2 A_2^\top = I$). Under this constraint, the optimal initialization for $A_2$ is derived analytically using the eigendecomposition of the sample covariance matrix $S = \frac{1}{k-1}H^\top H$. Specifically, $A_2$ is initialized using the eigenvectors corresponding to the smallest eigenvalues of $S$. This places the LoRA subspace in directions where the out-of-task data has minimal variance.
3. Practical Extensions:
   • Relaxation during Fine-Tuning: While the theoretical solution suggests freezing $A_2$, the authors find this degrades single-task accuracy. Instead, they use the analytical solution solely as the initialization for $A_2$, allowing it to be updated during fine-tuning. Empirical results show the deviation from the initial orthogonal state is marginal (approx. 14%), preserving the method's validity.
   • Scalability and Privacy: For merging $N$ tasks, the method aggregates latent features from all tasks except the target task $T_t$. To reduce memory usage and address privacy concerns, the authors propose averaging sample-wise latent features before concatenation.
   • Post-Hoc Application: The method can be extended to existing LoRA modules by decomposing learned weights using the OSRM-derived basis, though this may incur some performance loss.

Key Contributions

• Identification of Data-Parameter Interplay: The paper highlights that parameter interference in LoRA merging is not just a function of weight conflicts but is deeply tied to how parameters interact with specific data distributions.
• Pre-Fine-Tuning Constraint: Unlike prior works that operate during or after merging, OSRM intervenes before fine-tuning by initializing the LoRA subspace to be orthogonal to irrelevant data distributions.
• Plug-and-Play Integration: The approach is designed to integrate seamlessly with existing merging algorithms (e.g., Task Arithmetic, Fisher Merging, TIES, RegMean, EMR) without requiring changes to the merging logic itself.
• Robustness: The method demonstrates reduced sensitivity to hyperparameters such as scaling coefficients, sample sizes, and the number of tasks.

Experimental Results
The authors evaluated OSRM on eight datasets from the GLUE benchmark using three widely used LMs (RoBERTa-large, T5-large, Llama3.2-1B) and two larger LMs (Llama3.2-3B, Llama3-8B).

• Performance Gains: OSRM consistently outperformed existing merging baselines across all tested models and merging techniques.
  • On RoBERTa-large, OSRM improved average performance significantly over baselines (e.g., +13% on CoLA for Task Arithmetic, +21% on MNLI for Fisher Merging).
  • On T5-large, it substantially outperformed baselines on RegMean (+3.76% average) and Fisher (+7.9% average).
  • On Llama3.2-1B and 3B, OSRM surpassed baselines on TA, RegMean, TIES, and EMR, with per-task improvements reaching up to 12.78%.
• Single-Task Preservation: The method minimally impacts single-task accuracy (average gap <1%) and often surpasses baseline single-task performance.
• Robustness:
  • Scaling Coefficient: OSRM maintained superior performance across a wide range of scaling coefficients ($\lambda \in [0.1, 1.3]$), whereas baselines were more sensitive.
  • Sample Size: The method performed well with as few as 100 samples for initialization.
  • Task Count: Performance advantages became more pronounced as the number of merged tasks increased ($N > 5$).

Significance and Claims
The paper claims that OSRM offers a generalizable strategy for combining LoRA models more effectively by explicitly addressing the interaction between data and parameters. The authors emphasize that their approach provides a plug-and-play solution that enhances merging performance while preserving single-task accuracy. They argue that considering data-parameter interactions is critical for robust model merging, moving beyond purely data-free strategies that focus only on weight alignment. The results suggest that constraining the LoRA subspace before fine-tuning is a vital step to mitigate the unintended output shifts that currently plague LoRA model merging.