🤖 machine learning

Model Merging by Output-Space Projection

This paper proposes a novel model merging framework that formulates the combination of fine-tuned checkpoints as a convex quadratic program over residual updates, providing a theoretically grounded, closed-form diagnostic for predicting merge quality while empirically outperforming existing heuristic methods across language and vision benchmarks.

Original authors: Bethan Evans, Benjamin Etheridge, Stephen Roberts, Jared Tanner

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bethan Evans, Benjamin Etheridge, Stephen Roberts, Jared Tanner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of five expert chefs. Each one has spent months perfecting a specific dish: one is a master of pizza, another of sushi, a third of pasta, and so on. Now, you want to create a single "Super Chef" who can cook all five dishes perfectly, but you don't have time to train a new person from scratch.

The Problem with Current Methods
Right now, the standard way to combine these chefs is like a committee vote.

Task Arithmetic: You just take the average of their recipes.
Model Soups: You mix their ingredients together in a big pot and hope it tastes good.
TIES/DARE: You try to remove the ingredients they disagree on or randomly throw some away.

These methods work okay, but they are a bit like guessing. They rely on simple rules (like "everyone gets an equal vote") rather than actually calculating the perfect recipe for the Super Chef. They don't know exactly how much of the pizza chef's knowledge to keep versus the sushi chef's.

The Paper's New Idea: The "Output-Space Projection"
The authors of this paper propose a smarter way to mix these models. Instead of just averaging weights (ingredients), they look at the results (the finished dishes).

Here is the analogy:
Imagine you have a "Calibration Kitchen" where you test the chefs. You give them a specific ingredient (input) and ask for a specific dish (output).

The Base Model: This is your "blank slate" chef who knows nothing.
The Residuals: This is the difference between what the blank chef makes and what the expert chefs make. It's the "extra flavor" each expert adds.
The Goal: You want to mix these "extra flavors" together to get the perfect dish for every input.

The paper says: "Let's treat this like a math puzzle."

They frame the problem as a Quadratic Program (QP). In plain English, this is a fancy way of saying: "We can write down a perfect mathematical equation that tells us exactly how much of each expert's 'extra flavor' to add to minimize mistakes."

How It Works (The "Projection" Metaphor)
Think of the "perfect dish" as a target on a dartboard.

Each expert chef throws a dart (their update) that lands somewhere near the target.
The "residual" is the distance between where they landed and the bullseye.
Current methods just guess how to average those darts.
This paper's method asks: "If we project all these darts onto a specific line or plane (a subspace), which combination gets us closest to the bullseye?"

They found that if you look at the "energy" of the mistakes (how far off the darts are), you can mathematically calculate the optimal direction to mix them.

The Diagonal Method (The Cheap Version): This assumes the experts' skills are independent. It's like saying, "The pizza chef only affects pizza, and the sushi chef only affects sushi." This is fast and easy to compute.
The Optimal Basis Method (The Expensive Version): This realizes that skills might overlap. Maybe the pizza chef's sauce technique actually helps with the pasta too. This method finds the best possible mix of directions to capture all the useful "energy" from the experts.

Key Findings

It's a Unifying Theory: The paper shows that all the popular methods people use right now (Task Arithmetic, Model Soups, TIES) are actually just special, simplified versions of this new math puzzle. They are "heuristic" (rule-of-thumb) guesses, whereas this new method finds the actual mathematical solution.
It Predicts Success: Before even merging the models, the authors created a "diagnostic tool." By looking at the calibration data, they can calculate a score (the "fraction of residual energy captured") that predicts how well the merged model will perform. It's like checking the ingredients before cooking to know if the dish will be good.
It Works Better: When they tested this on image recognition (like identifying cars or animals) and language models (like LLMs), their math-based method either matched or beat the existing "guessing" methods.
- The "cheap" diagonal version was often good enough and very fast.
- The "expensive" optimal version was even better when the experts' skills were complex and overlapping.

The Bottom Line
The paper doesn't just say "mix these models." It provides a mathematical blueprint for how to mix them perfectly. It turns the art of model merging into a solvable geometry problem, showing that by projecting the errors of the models onto the right mathematical space, you can create a single, super-performing model without needing to retrain it from scratch.

They also note that while the perfect solution requires heavy math (solving for eigenvectors), the simpler version (diagonal masking) is often a great, fast shortcut that still outperforms the old ways of doing things.

Technical Summary: Model Merging by Output-Space Projection

Problem Statement

The proliferation of task-specific fine-tuned language and vision models has created a demand for model merging—the process of combining multiple fine-tuned checkpoints into a single multi-task model without retraining. While existing practical approaches such as Task Arithmetic, Model Soups, TIES, and DARE have demonstrated empirical success, they rely on heuristic design choices (e.g., fixed scalar weights, stochastic masking, or sign-based pruning). Crucially, these methods lack a formal mathematical characterization of optimal combination weights and do not possess a precise loss function for which they are provably optimal.

Methodology

The authors propose a framework that formulates model merging as a convex quadratic programme (QP) over residual updates, grounded in an output-space projection perspective.

1. Formulation as a Convex QP

The paper begins by observing that merging via diagonal masks on fine-tuned residuals can be cast as a convex QP.

Setup: Given a base model with weights $\theta$ and $K$ fine-tuned models with residual updates $\{\delta^{(k)}_N\}$ at a specific layer $N$ , the goal is to find a merged residual $\delta_{merge}$ .
Diagonal Masking: The method applies a diagonal mask $D_k = \text{diag}(d^{(k)})$ to each residual before summing: $\delta_{merge}(d) = \sum_{k=1}^K D_k \delta^{(k)}_N$ .
Objective: The method minimizes a squared-output calibration objective $J(d) = \sum_{j \in \mathcal{K}} \|A_j d + b_j\|^2$ over a calibration set $\mathcal{K}$ . Here, $b_j$ represents the base model's error on input $x_j$ , and $A_j$ encodes the linearized effect of the residuals on the output.
Optimality: Solving this QP yields weights that minimize the squared-output loss. Existing methods (Task Arithmetic, Model Soups, TIES, DARE) are shown to correspond to specific feasible points or special cases of this QP (e.g., fixed scalars or specific stochastic masks), implying the QP solution is theoretically superior or equal under the same objective.

2. General Basis and Irreducible Error

The authors extend the diagonal mask approach to a general basis formulation to address the limitations of axis-aligned (diagonal) merging.

Projection Perspective: Merging is viewed as approximating the residual errors of a base model using a subspace of candidate corrections. The optimal solution corresponds to projecting base-model residuals onto the subspace spanned by candidate updates in the output space.
Irreducible Error: The paper derives a closed-form diagnostic for the irreducible error of any chosen basis. This error is determined by the fraction of residual energy captured by the chosen subspace.
Optimal Basis: The analysis shows that the optimal basis for the relaxed projection problem is given by the leading eigenvectors of the residual energy matrix $S = \sum_{j} b_j b_j^\top$ .
Trade-off Characterization: The framework quantifies the trade-off between the computational efficiency of the diagonal QP and the performance gains of the optimal basis. It establishes that diagonal merging is optimal when model updates decompose into independent modes in a shared basis; otherwise, the "suboptimality gap" is measurable via the captured energy fraction.

3. Multi-Layer Extension

For multi-layer merging, the authors propose a sequential layer-wise algorithm. Since the joint optimization across layers is non-convex due to nonlinear interactions, the method applies the single-layer QP sequentially. At each step, the residual is updated using the current merged model, and the QP is solved for the next layer, recomputing the necessary Jacobians.

Key Contributions

Unified Framework: The paper subsumes existing heuristic merging methods (Task Arithmetic, Model Soups, TIES, DARE) as special cases of a diagonal-mask QP, providing a unified mathematical perspective.
Convex Formulation: It establishes that model merging can be formulated as a convex quadratic program over residual updates, guaranteeing a global minimum for the chosen basis.
Closed-Form Diagnostic: The authors introduce a computable diagnostic—the fraction of residual energy captured by a chosen basis—which predicts downstream merge quality using only the calibration set, without requiring full downstream evaluation.
Optimal Basis Characterization: The work identifies the output-weighted principal components of residual activations as the optimal basis for minimizing squared-output loss, clarifying when diagonal merging is sufficient and when a more complex basis is required.
Empirical Validation: The framework is validated across language (LLaMA 3.1) and vision (ViT-B/32, MNIST) benchmarks, demonstrating consistent gains over existing heuristics.

Experimental Results

Single-Layer Performance: In single-layer merging experiments on ViT-B/32 (fine-tuned on CIFAR-10, STL-10, Imagenette, EuroSAT) and LLaMA 3.1, the QP solution achieves the lowest Mean Squared Error (MSE) across all datasets and the best average accuracy compared to Task Arithmetic, Model Soups, TIES, DARE, and Fisher-weighted merging.
Basis Analysis: Experiments on MNIST digit classification tasks demonstrate that the optimal basis (top eigenvectors of the residual matrix) consistently achieves the lowest MSE. The standard diagonal basis, while less effective than the optimal basis, remains competitive and outperforms benchmark heuristics, validating its use as an efficient proxy.
Multi-Layer Gains: The sequential multi-layer extension on LLaMA 3.1 shows consistent improvements in both MSE and accuracy across code, instruction-following, and math tasks compared to baseline methods.
Diagnostic Utility: The "captured energy" metric successfully predicts the performance ranking of different basis choices, confirming the theoretical link between residual energy capture and downstream loss.

Significance and Claims

The paper claims to provide a unifying theoretical framework that explains the success of existing heuristics and offers a principled method for improving them.

Theoretical Rigor: Unlike previous methods that rely on ad-hoc rules, this approach offers a precise mathematical notion (minimizing squared-output loss via projection) for which the method is optimal.
Practical Utility: The framework provides a computable diagnostic to determine when the expensive optimal basis is necessary versus when the cheaper diagonal QP suffices.
Modesty: The authors acknowledge limitations, noting that the framework is derived for a linearized squared-output objective. They explicitly state that extending the framework to non-quadratic losses (like cross-entropy) and developing scalable approximations for the optimal basis in high-dimensional settings remain important directions for future work. They also note that lower MSE does not always perfectly correlate with higher task accuracy, a limitation observed in their LLM experiments.

In summary, the paper reframes model merging from a heuristic combination of weights to a convex optimization problem in output space, offering both a theoretical justification for existing methods and a path toward more optimal merging strategies.