🤖 machine learning

How does the optimizer implicitly bias the model merging loss landscape?

This paper reveals that the success of model merging is governed by a non-monotonic relationship with a unified "effective noise scale" derived from optimizer dynamics, demonstrating that factors like learning rate, weight decay, batch size, and data augmentation shape the global loss landscape geometry to determine whether independently trained solutions can be effectively combined.

Original authors: Chenxiang Zhang, Alexander Theus, Damien Teney, Antonio Orvieto, Jun Pang, Sjouke Mauw

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenxiang Zhang, Alexander Theus, Damien Teney, Antonio Orvieto, Jun Pang, Sjouke Mauw

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two expert chefs. Chef A is a master of Italian pasta, and Chef B is a wizard at making French pastries. You want to combine their skills into one "Super Chef" who can do both, but you don't want to hire a third person or buy new equipment. You just want to mix their existing recipes.

In the world of Artificial Intelligence, this is called Model Merging. Researchers take two AI models that have been trained separately (like our two chefs) and try to blend their "weights" (their internal knowledge) into a single model.

Sometimes, this works like magic: the new model is great at both tasks. Other times, it's a disaster, and the new model can't do anything well.

The Big Question: Why does it work sometimes and fail others?

This paper, presented at ICLR 2026, answers that question by looking at how the models were trained. The authors discovered a hidden "secret sauce" called the Effective Noise Scale.

Here is the breakdown using simple analogies:

1. The "Noise" in the Kitchen

When a chef learns a recipe, they don't just read a book perfectly. They taste, adjust, make mistakes, and try again. In AI training, this "mistake-making" is called noise. It comes from things like:

Learning Rate: How big of a step the model takes when learning. (A big step = more noise; a tiny step = less noise).
Batch Size: How many examples the model looks at at once. (Looking at fewer examples = more noise; looking at many = less noise).
Weight Decay: A rule that keeps the model from getting too "stubborn" or complex.
Data Augmentation: Showing the model slightly different versions of the same picture (like flipping an image) to make it learn better.

The paper argues that all these different settings actually control the same thing: how much "jitter" or "noise" is in the model's learning path.

2. The Goldilocks Zone (Not Too Hot, Not Too Cold)

The researchers found that the amount of noise is the key to whether two models can be merged.

Too Little Noise (The Silent Library): If the training is too quiet and precise (small learning rate, huge batch size), the models get stuck in very specific, narrow "valleys" in their knowledge landscape. They are so specialized that when you try to mix them, they clash. It's like trying to mix two very rigid, precise recipes; they don't blend well.
Too Much Noise (The Chaotic Storm): If the training is too chaotic (huge learning rate, tiny batch size), the models never settle down. They are too unstable to be useful.
Just Right (The Goldilocks Zone): The paper found a "sweet spot" of moderate noise. When the training has just the right amount of jitter, the models end up in "wide, flat valleys."

The Analogy: Imagine the AI's knowledge as a landscape of hills and valleys.

Low noise trains the model to sit in a tiny, deep, narrow hole. If you try to walk from Chef A's hole to Chef B's hole, you have to climb a steep mountain in between. They can't merge.
Moderate noise trains the model to sit in a wide, flat meadow. If you try to walk from Chef A's spot to Chef B's spot, the ground is flat the whole way. You can mix them easily!

3. The Surprising Findings

The paper tested this with many different settings and found some counter-intuitive things:

Bigger Steps are Better: Usually, people think taking smaller, careful steps (low learning rate) is safer. But for merging, taking bigger steps (higher learning rate) actually helps the model find those "wide meadows" where merging is easy.
More "Jitter" Helps: Using smaller groups of data (smaller batch sizes) or adding random data tricks (augmentation) creates just enough chaos to keep the model flexible and mergeable.
It's a Universal Rule: This rule applies whether you are training a model to recognize cats and dogs, write stories, or translate languages.

4. Why This Matters

Before this paper, if you wanted to merge two AI models, you had to guess and check. You'd train 100 models, try to merge them, and hope one worked. It was like throwing darts in the dark.

Now, we know that training dynamics shape the landscape. By simply adjusting the "noise" (learning rate, batch size, etc.) to hit that "Goldilocks" zone, we can intentionally train models that are designed to be merged.

In a nutshell:
To get two AI models to work together later, don't train them to be too perfect and rigid. Train them with a little bit of "chaos" and "big steps." This keeps their internal world flexible and flat, making it easy to combine them into a super-model later without breaking anything.

1. Problem Statement

Model merging aims to combine independently trained models into a single model that retains the capabilities of its components without additional inference costs. While methods like Linear Interpolation (averaging weights) and Task Arithmetic (combining task vectors) are popular, their success is highly unpredictable.

The Core Question: Why do some independently trained models merge successfully while others fail, even if they achieve similar individual performance?
The Gap: Existing literature connects optimizer noise to the flatness or generalization of a single minimum. However, it remains unclear how optimization dynamics (learning rate, batch size, etc.) shape the global loss landscape geometry between independent solutions, which determines their mergeability.

2. Methodology & Framework

The authors propose that the Effective Noise Scale ( $\tilde{S}$ ) is the unifying factor controlling model merging success.

Theoretical Foundation

The paper models Stochastic Gradient Descent (SGD) as a discretized Stochastic Differential Equation (SDE). The diffusion strength (noise) of the optimization trajectory is governed by:
$\tilde{S} \propto \frac{\eta}{B(1 - \mu)}$
Where:

$\eta$ : Learning rate.
$B$ : Batch size.
$\mu$ : Momentum.
The term also implicitly includes data augmentation ( $\Sigma_A$ ) and weight decay (via its effect on effective learning rate in scale-invariant networks).

Experimental Setup

The authors conducted extensive experiments across:

Architectures: MLP, ResNet-18, DenseNet-121, Vision Transformers (ViT), and GPT-2 style Transformers.
Datasets: SVHN, CIFAR-10/100, TinyImageNet, WILDS-FMoW, RESISC45, and TinyStories (for language modeling).
Merging Methods:
1. Linear Interpolation: $\theta_{merge} = (1-\alpha)\theta_A + \alpha\theta_B$ .
2. Task Arithmetic: $\theta_{merge} = \theta_{base} + \sum \alpha_i (\theta_{task_i} - \theta_{base})$ .
3. TIES-Merging: A method to resolve interference when merging different tasks.
Metric: Accuracy Gain (or Loss Gain for LLMs), defined as the performance improvement of the merged model over the average of the single models.

3. Key Contributions & Findings

A. The Non-Monotonic "Critical Point" of Effective Noise

The most significant finding is that merging success is a non-monotonic function of the effective noise scale ( $\tilde{S}$ ):

Low Noise: Solutions are too "sharp" or confined; merging yields little to no benefit.
High Noise: Solutions become unstable or diverge; merging fails.
Intermediate (Critical) Noise: There exists an optimal regime where merging yields the maximum performance gain.

Implication: Simply increasing noise (e.g., via larger learning rates) is not always better; it must be tuned to a specific "sweet spot."

B. Unifying Optimizer Components

The paper demonstrates that distinct hyperparameters all modulate merging success by controlling $\tilde{S}$ :

Learning Rate ( $\eta$ ): Larger learning rates (up to the critical point) identify solutions that are more compatible for merging. This holds even when single-model performance is identical to smaller learning rates.
Weight Decay ( $\lambda$ ): In scale-invariant networks (e.g., those with BatchNorm/LayerNorm), larger weight decay prevents the effective learning rate from vanishing, effectively increasing noise and improving mergeability.
Batch Size ( $B$ ): Smaller batch sizes inject more gradient noise, leading to better merging compatibility.
Momentum ( $\mu$ ): Higher momentum reduces oscillations but shapes the landscape toward broader regions, improving mergeability.
Data Augmentation: Acts as an additional noise source, complementing batch and learning rate noise.

C. Impact on Loss Landscape Geometry

Linear Interpolation: Higher effective noise creates a "flatter" and wider basin between solutions, reducing the loss barrier along the interpolation path.
Task Arithmetic: The geometry depends heavily on initialization.
- Transfer Learning (Pretrained Base): Larger learning rates lead to smoother, flatter landscapes and better merging.
- Non-Transfer (Scratch): Larger learning rates can lead to sharper minima, making merging harder.
Permutation Symmetry: Higher noise simplifies the "re-basing" process (aligning neurons via permutation), making independent solutions easier to align linearly.
Feature Alignment: Interestingly, higher noise reduces feature alignment (Linear CKA) between branches. This suggests that feature diversity (induced by noise) is beneficial for merging, whereas highly aligned representations offer little gain.

D. Language Modeling & Transfer Learning

LLMs: The findings extend to Transformer language models (TinyStories). Larger learning rates and weight decay improve merging (measured by loss reduction).
Transfer Learning: In fine-tuning scenarios (e.g., CLIP on FMoW), a moderate-to-large learning rate identifies solutions that are easier to merge. However, the absolute best merged model often comes from a moderate learning rate, balancing mergeability with single-model performance.

4. Significance & Implications

Theoretical Unification: The paper bridges the gap between optimization dynamics and model merging, showing that the same noise mechanisms affecting single-model generalization also dictate the compatibility of independent solutions.
Practical Guidance: It suggests that practitioners can improve model merging not just by trying different architectures, but by tuning the effective noise scale.
- Strategy: Use a learning rate and batch size that induce a "moderately large" noise regime to find mergeable solutions.
- Simplification: Hyperparameter search for merging can be simplified by focusing on $\tilde{S}$ rather than individual parameters.
Beyond Averaging: The results imply that training dynamics should be manipulated specifically to produce "mergeable" models, potentially leading to more robust ensembles and specialized model combinations (e.g., in Mixture of Experts).

5. Limitations

Scale: Experiments were conducted on relatively small-scale models (e.g., ResNet-18, small GPT-2) due to computational constraints, though the trends are consistent across architectures.
Theoretical Proof: The paper provides strong empirical evidence and theoretical intuition (via SDEs) but lacks rigorous mathematical proofs for the non-monotonic relationship in general non-convex settings.

Conclusion

The paper establishes that optimization noise is the key determinant of model merging success. By controlling the effective noise scale through learning rate, weight decay, batch size, and augmentation, one can steer the loss landscape geometry to create regions where independent solutions are naturally compatible, unlocking the full potential of model merging techniques.