🤖 machine learning

Efficient Multi-Source Knowledge Transfer by Model Merging

This paper proposes a scalable and precise multi-source transfer learning framework that decomposes source models into rank-one components via Singular Value Decomposition (SVD), aggregates the most salient components, and fine-tunes only the principal singular values to efficiently adapt to target tasks while preserving robustness.

Original authors: Marcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marcin Osial, Bartosz Wójcik, Bartosz Zieliński, Sebastian Cygert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn how to be a master chef.

The Old Way (Traditional Transfer Learning):
Usually, you'd start by hiring one famous chef (a pre-trained model) who knows how to cook everything. You then take them to a specific kitchen and teach them your family's secret recipe for lasagna. This works well, but it's slow and expensive.

The "Multi-Source" Problem:
Now, imagine you don't just have one chef. You have access to a massive library of 20 different chefs. One is a master of Italian pasta, another is a sushi expert, a third is a BBQ pitmaster, and so on.

The Problem: If you try to hire all 20 chefs at once to teach you lasagna, you run into a logistical nightmare. You can't fit them all in the kitchen (memory issues), and if you try to listen to all of them talking at once, you get confused (noise and interference).
The Current Solution (aTLAS): The current best method is like trying to have all 20 chefs stand in a circle and shout their advice simultaneously. You try to figure out who to listen to by giving each chef a volume knob. But as you add more chefs, the room gets too crowded, the noise gets too loud, and the system crashes. It's inefficient and messy.

The New Solution: AXIS (The "SVD" Method)

The paper introduces a new method called AXIS. Think of AXIS not as a room full of shouting chefs, but as a super-smart editor who reads the cookbooks of all 20 chefs and creates a single, perfect "Greatest Hits" recipe book for you.

Here is how AXIS works, step-by-step:

1. Deconstructing the Knowledge (The SVD)

Instead of listening to the chefs talk, AXIS takes their "cookbooks" (the mathematical weights of their models) and breaks them down into their smallest, most basic ingredients.

The Analogy: Imagine every chef's knowledge is a giant, complex soup. AXIS uses a special sieve (called Singular Value Decomposition or SVD) to separate the soup into individual ingredients: salt, pepper, garlic, carrots, etc.
Some ingredients are huge and essential (like the main flavor of the soup). Others are tiny specks of dust or random noise.

2. The "Greatest Hits" Selection (Aggregation)

AXIS looks at the ingredients from all 20 chefs. It ignores the noise and the tiny, useless specks. It only grabs the Top 10% of the most important ingredients from the entire group.

The Magic: If the Italian chef has the best "tomato sauce" ingredient and the BBQ chef has the best "smoke flavor" ingredient, AXIS picks those specific high-quality ingredients. It doesn't care that the chefs are different; it just cares about the quality of the ingredient.
The Result: It mixes these top ingredients into a single, compact "Master Broth." This broth is small enough to fit in your pocket, but it contains the essence of all 20 chefs.

3. The Final Tune-Up (Adaptation)

Now, you take this "Master Broth" and add it to your base kitchen. But you don't just dump it in and hope for the best. You do a final, quick adjustment.

The Analogy: You taste the broth and realize, "Hmm, for my specific lasagna, I need a little more salt and a little less pepper." You only tweak the main flavors (the principal singular values) to match your specific dish. You don't need to retrain the whole kitchen; you just adjust the seasoning.

Why is AXIS a Game-Changer?

1. It's Super Efficient (The "Backpack" vs. The "Truck")

Old Way (aTLAS): To learn from 20 chefs, you need a truck big enough to carry all 20 of them. If you add a 21st chef, you need a bigger truck. It gets expensive and slow.
AXIS: You only need a small backpack. No matter if you have 5 chefs or 500 chefs, you just extract the "top ingredients" and put them in the same small backpack. The size of the backpack never changes, making it incredibly fast and cheap to run.

2. It's Resilient (The "Noise-Canceling Headphones")

Imagine one of the 20 chefs is actually a prankster who puts salt in the sugar bowl (a "corrupted" model).
Old Way: The prankster's noise drowns out the good advice because the system tries to listen to everyone equally.
AXIS: Because AXIS only looks for the strongest, most important ingredients, the prankster's tiny, noisy contribution gets filtered out immediately. It's like wearing noise-canceling headphones; you only hear the clear, strong signal.

3. It Works Everywhere
The paper tested this on both Vision (teaching computers to see images like cats, cars, and flowers) and Language (teaching computers to write and answer questions). In both cases, AXIS learned faster, used less memory, and made fewer mistakes than the previous best methods.

The Bottom Line

AXIS is a smart way to combine knowledge from many different AI models without getting overwhelmed. Instead of trying to memorize every detail from every source, it identifies the "golden nuggets" of knowledge, combines them into a compact package, and then fine-tunes that package for your specific needs. It's the difference between trying to drink from a firehose and sipping from a perfectly brewed cup of coffee.

1. Problem Statement

The paper addresses the challenge of Multi-Source Knowledge Transfer, where the goal is to combine knowledge from numerous pre-fine-tuned models (source tasks) to adapt a model to a new, unseen target task without retraining from scratch.

Current Limitations: Existing methods, such as aTLAS (the state-of-the-art baseline), suffer from two main issues:
1. Coarse-Grained Precision: They lack the ability to perform fine-grained knowledge extraction, often treating task vectors as monolithic blocks rather than analyzing their internal structure.
2. Scalability Bottlenecks: Methods like aTLAS require storing and learning coefficients for every source task vector during the adaptation phase. This leads to memory and runtime costs that scale linearly with the number of source models, making it infeasible to aggregate large numbers of models or large parameter counts.
The Gap: There is a need for a method that can efficiently aggregate knowledge from many sources, select only the most relevant information, and remain robust to noisy or corrupted source data.

2. Methodology: The AXIS Framework

The authors propose AXIS (Aggregation by eXtraction of Important Singular components), a two-stage framework that leverages Singular Value Decomposition (SVD) to decompose, aggregate, and adapt model weights.

Stage 1: Extraction and Aggregation (Offline)

Instead of keeping all source task vectors in memory, AXIS decomposes them into their fundamental components:

SVD Decomposition: For each source task matrix $\Delta_i$ (the difference between a fine-tuned model and the pre-trained base), the method computes the SVD: $\Delta_i = U_i \Sigma_i V_i^T$ . This breaks the matrix into rank-one components (triplets of left singular vectors, singular values, and right singular vectors).
Normalization: Singular values are normalized by the Frobenius norm of the task matrix to ensure scale comparability across different tasks.
Global Ranking & Selection: All normalized components from all source tasks are pooled together. The method performs a global ranking based on the magnitude of the singular values and selects only the Top-K most significant components.
Synthesis: These Top-K components are summed to create a single, compact merged task matrix $\Delta_m$ $Δ_{m}$ .
- Key Advantage: This step reduces the memory footprint to a constant size, regardless of the number of source models ( $T$ ).

Stage 2: Target Task Adaptation (Online)

The synthesized matrix $\Delta_m$ is then adapted to the specific target task:

Re-orthogonalization: $\Delta_m$ undergoes a final SVD ( $\Delta_m = U_t \Sigma_t V_t^T$ ). This step is crucial for stabilizing the basis and decorrelating the vectors, ensuring the merged knowledge forms a coherent update direction.
Parameter-Efficient Fine-Tuning (PEFT): The model is adapted to the target task by fine-tuning only the Top-N singular values in $\Sigma_t$ $Σ_{t}$ . The singular vectors ( $U_t, V_t$ $U_{t}, V_{t}$ ) and the remaining singular values are frozen.
- This allows the model to "recalibrate" the importance of the most dominant directions in the merged knowledge base with minimal trainable parameters.

3. Key Contributions

Novel Multi-Source Framework: AXIS introduces a spectral approach to model merging that outperforms state-of-the-art methods (aTLAS) across 21 distinct vision tasks and 7 NLP benchmarks.
Scalability and Efficiency: By decoupling the aggregation (offline) from the adaptation (online) and using SVD to compress knowledge, AXIS maintains a constant memory and runtime footprint during fine-tuning, regardless of the number of source models. This enables scaling to hundreds of source tasks.
Robustness: The method demonstrates superior resilience to:
- Parameter Corruption: It is less sensitive to noisy or corrupted source vectors because the SVD selection process naturally filters out low-magnitude (likely noisy) components.
- Input Degradation: It performs better under partial input information (e.g., masked patches) and common image corruptions.
- Sparsity: It effectively leverages knowledge from heavily pruned source models.
Generalizability: The framework is modality-agnostic, showing consistent performance improvements in both Vision (ViT architectures) and Language (T5 architectures) domains.

4. Results

Performance: AXIS consistently outperforms aTLAS and other merging baselines (like TIES-Merging, DARE) across various parameter budgets (10%, 20%, 40% of trainable parameters).
- Vision: Achieved an average accuracy of 78.42% on ViT-B-32 compared to 75.13% for aTLAS.
- NLP: Consistently higher accuracy across seven diverse NLP benchmarks.
Efficiency:
- Memory/Time: While aTLAS's runtime and memory scale near-linearly with the number of sources, AXIS remains constant.
- Training: AXIS achieves near-parity with full fine-tuning (N=100%) while using significantly fewer epochs (10 vs. up to 76) and fewer parameters.
Robustness Experiments:
- When a single source vector is corrupted with high-magnitude Gaussian noise, AXIS maintains high accuracy, whereas aTLAS degrades significantly.
- Under input patch dropout (up to 80% masked), AXIS degrades much slower than baselines.

5. Significance

The paper presents a paradigm shift in multi-source transfer learning. By moving from coarse-grained vector averaging to fine-grained spectral component selection, AXIS solves the scalability bottleneck that has prevented the aggregation of large "model zoos."

Practical Impact: It allows practitioners to leverage the collective knowledge of hundreds of specialized models without requiring massive computational resources or access to original training data.
Theoretical Insight: The work provides empirical evidence that the most transferable knowledge in fine-tuned models is concentrated in the principal singular components (high-magnitude singular values), while lower-ranked components often represent task-specific noise or interference.
Future Direction: The method's ability to function even with minor architectural differences or different pre-training histories (as shown in ablation studies) suggests a path toward more universal and robust foundation model adaptation.