Bridging Domains through Subspace-Aware Model Merging

This paper introduces SCORE, a novel model merging method that resolves singular subspace conflicts between domain-specific models by projecting them into a shared orthogonal basis, thereby significantly improving generalization to unseen domains compared to existing approaches.

The Gist

Imagine you have a team of expert chefs. One chef is a master at cooking perfect meals for a rocky, mountainous region (where the ingredients are tough and the weather is cold). Another chef is a master at cooking for a snowy, forested region (where the ingredients are fresh and the air is crisp).

Both chefs are amazing at their specific jobs. But what happens if you want to cook a meal for a new, unseen location: a snowy mountain peak where both the tough mountain rocks and the fresh forest snow exist together?

If you just ask them to swap recipes or average their techniques, the result might be a disaster. The mountain chef might try to use heavy, rocky spices on the delicate snow ingredients, and the snow chef might try to use delicate herbs on the tough rocks. They would fight over the kitchen, and the final dish would be confusing and inedible.

This is exactly the problem computer scientists face when they try to merge AI models.

The Problem: The "Kitchen Fight"

In the world of Artificial Intelligence, researchers often train separate models on different types of data (different "domains").

• Model A learns to recognize cats in photos taken in sunny parks.
• Model B learns to recognize cats in photos taken in rainy alleys.

Both models are great at their specific job. But when researchers try to combine them into one "super model" that can recognize cats in any weather (including a new, unseen stormy day), the models often clash.

The paper explains that when models are trained on different types of environments (domains), their internal "thought processes" (mathematical patterns called subspaces) overlap in a messy way. It's like two people trying to drive the same car but fighting over the steering wheel. The dominant directions of one model push the other out of the way, causing the merged model to forget how to handle the new, unseen situations.

The Solution: SCORE (The Diplomatic Chef)

The authors propose a new method called SCORE (Subspace COnflict-Resolving mErging). Think of SCORE as a brilliant Diplomatic Chef who knows how to merge these two experts without them fighting.

Here is how SCORE works, using our kitchen analogy:

1. The Audit (SVD): First, SCORE looks at the "recipe books" of both chefs. It breaks down their cooking styles into their most important ingredients (mathematically, this is called Singular Value Decomposition). It identifies the "top notes" of their cooking.
2. The Shared Table (Orthogonal Basis): Instead of letting the chefs argue over the same counter space, SCORE builds a new, shared table. It creates a neutral ground where both chefs can lay out their best ingredients side-by-side without stepping on each other's toes.
3. The Conflict Filter (Trimming): This is the magic step. When the chefs put their ingredients on the shared table, some ingredients clash (e.g., the mountain spices vs. the snow herbs).
   • SCORE looks at the "off-diagonal" ingredients—the ones that represent the conflict between the two styles.
   • It keeps the strong, clear ingredients (the diagonal) that both chefs agree on.
   • It prunes (cuts away) the noisy, conflicting ingredients that would ruin the dish. It's like saying, "Okay, Chef A, your heavy rock-spice is too strong for Chef B's snow-herbs. Let's remove that specific clash, but keep the rest of your unique flavor."
4. The New Masterpiece: Finally, SCORE rebuilds the recipe using this cleaned-up, conflict-free list. The result is a single model that understands both the mountain and the snow, and can now handle a snowy mountain it has never seen before.

Why This Matters

Before this paper, merging AI models was often like throwing ingredients into a blender and hoping for the best. It worked okay for simple tasks, but failed when trying to generalize to new, weird situations.

SCORE is like a smart filter that ensures the merged model doesn't just "average" the two experts, but actually integrates their best qualities while removing the parts that cause them to fight.

The Results:
The researchers tested this on many different scenarios, from recognizing animals in different landscapes to diagnosing diseases in medical images.

• Old methods: Often performed worse than just using a single model.
• SCORE: Consistently beat all other methods, creating a model that was better at handling "unseen" situations than any single expert or previous merging technique.

The Bottom Line

SCORE is a new way to combine AI experts. Instead of forcing them to compromise or fight, it finds a common language, removes the parts of their knowledge that clash, and builds a unified expert that is smarter and more adaptable than the sum of its parts. It allows us to take specialized AI models and combine them into a robust tool that can handle the messy, unpredictable real world.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in Model Merging: while existing methods effectively combine models for multi-task scenarios (where tasks have different label spaces or semantics), they struggle with Domain Generalization (DG).

• The Challenge: In DG, multiple models are fine-tuned on the same task (e.g., image classification) but on different data distributions (domains, e.g., "rocky terrain" vs. "snowy terrain"). The goal is to merge these "domain experts" into a single model that generalizes to unseen domains.
• The Core Issue: The authors identify that domain-specific fine-tuning induces stronger subspace conflicts than traditional multi-task learning. Because the models solve the same classification problem, their parameter updates (delta weights) align along similar singular directions. When merged naively (e.g., via averaging), these competing dominant singular directions interfere with each other, leading to catastrophic forgetting or poor generalization on unseen distributions.
• Evaluation Gap: Most existing merging works rely on in-distribution evaluation or require access to validation data. The authors propose a Leave-One-Domain-Out (LODO) protocol where the merged model is evaluated on an unseen target domain without any further training or data access.

2. Methodology: SCORE

The authors propose SCORE (Subspace Conflict-Resolving mErging), a method designed to identify and mitigate singular subspace conflicts without requiring source data or optimization.

Key Steps of SCORE:

1. SVD Decomposition: For each domain-specific delta weight matrix $\Delta_d$, the method computes the Singular Value Decomposition (SVD) to extract the leading left ($U_d$) and right ($V_d$) singular vectors.
2. Concatenation: The leading singular vectors from all $D$ domains are concatenated to form matrices $U_*$ and $V_*$.
3. Orthogonalization (Shared Basis): Since the concatenated vectors are not guaranteed to be orthogonal, SCORE computes the SVD of $U_*$ and $V_*$ to derive a shared orthogonal basis ($U_\perp, V_\perp$). This basis represents the principal components common to all domains.
4. Basis Transformation: Each domain's delta matrix is projected into this shared basis:
   $$\Delta'_d = U_\perp^\top \Delta_d V_\perp$$
   In this transformed space:
   • Diagonal elements represent agreement (energy aligned with shared principal directions).
   • Off-diagonal elements represent conflicts (coupling between different shared directions).
5. Conflict Resolution (Trimming): To resolve conflicts, the method applies a trimming function to the transformed matrices:
   • It keeps all diagonal elements (preserving domain-specific knowledge aligned with the shared basis).
   • It keeps off-diagonal elements that are within a statistical threshold (e.g., $< 1.96\sigma$ of the mean off-diagonal magnitude), preserving significant shared variability.
   • It prunes (sets to zero) off-diagonal outliers that represent destructive noise or strong interference between domains.
6. Reconstruction: The trimmed matrices are summed to form a merged singular value matrix $\Sigma_{score}$, which is then reconstructed using the shared basis:
   $$\hat{M} = U_\perp \Sigma_{score} V_\perp^\top$$
   The final model is $\theta_{final} = \theta_{pre} + \hat{M}$.

3. Key Contributions

1. Empirical Analysis of Subspace Overlap: The authors demonstrate via the Subspace Alignment Ratio (SAR) metric that domain generalization scenarios exhibit significantly higher subspace overlap (conflict) than multi-task scenarios. This high overlap is the primary cause of failure in standard merging methods for DG.
2. SCORE Algorithm: A novel, data-free, optimization-free merging method that explicitly resolves singular subspace conflicts by orthogonalizing the basis and pruning conflicting off-diagonal components.
3. Comprehensive Benchmarking: The paper evaluates merging on 8 diverse domain generalization datasets (including natural images like PACS, DomainNet, and medical datasets like FedISIC) across 3 model scales (ViT-B/32, ViT-B/16, ViT-L/14).
4. Superiority over Ensembles: The authors show that model merging can outperform traditional logit ensembles while maintaining the inference cost of a single model, proving that the compositional knowledge is effectively integrated rather than just averaged.

4. Experimental Results

• Performance: SCORE consistently outperforms state-of-the-art merging methods (Task Arithmetic, TIES, TSV, ISO-C, etc.) across all benchmarks.
  • On ViT-B-32, it achieved an average accuracy of 65.69%, surpassing the next best method (TSV at 64.95%) by 0.74 percentage points.
  • On ViT-L-14, it achieved 73.04%, beating the runner-up by 0.58 percentage points.
• Medical Imaging: The method showed particular robustness on medical datasets (FedISIC, RetinaDomains), where class imbalance and distribution shifts are severe.
• Ablation Studies:
  • Trimming is crucial: Simply keeping the diagonal (ignoring off-diagonals) or keeping the full matrix (ignoring conflicts) resulted in lower performance. The "Trimmed" strategy (keeping significant off-diagonals while pruning outliers) yielded the best results.
  • Outperforming Ensembles: SCORE outperformed logit ensembles by 1.12% to 1.90% across model sizes, demonstrating that merging creates a more coherent representation than simply averaging predictions.

5. Significance and Impact

• Paradigm Shift in Generalization: The paper establishes model merging as a viable, efficient strategy for domain generalization, moving beyond the traditional "train from scratch" or "fine-tune on all data" approaches.
• Efficiency: It offers a data-free solution. Practitioners can combine pre-existing fine-tuned models from public repositories (e.g., Hugging Face) to create a robust model for unseen domains without needing access to the original training data or performing expensive retraining.
• Scalability: The method scales well with model size (from B to L variants) and is computationally efficient, requiring only SVD operations on weight matrices.
• Theoretical Insight: By linking the failure of merging in DG to singular subspace conflicts, the paper provides a theoretical framework for understanding why standard averaging fails when distributions shift but tasks remain the same.

In summary, SCORE provides a mathematically grounded solution to the "conflict" problem in model merging, enabling the creation of robust, generalizable models by intelligently resolving competing singular directions in the parameter space.