🤖 machine learning

RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging

RegMean++ improves upon the RegMean model merging approach by incorporating both intra-layer and cross-layer dependencies into the regression objective, leading to superior effectiveness and generalization across diverse tasks and distribution shifts.

Original authors: The-Hai Nguyen, Dang Huu-Tien, Takeshi Suzuki, Le-Minh Nguyen

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: The-Hai Nguyen, Dang Huu-Tien, Takeshi Suzuki, Le-Minh Nguyen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef trying to create the "Ultimate Fusion Dish" by combining the recipes of five different world-class chefs.

Each chef is an expert in a specific cuisine (one is a master of Italian, one of Thai, one of Mexican, etc.). You want to combine their skills into one single "Super-Recipe" that can cook anything delicious.

The Problem: The "RegMean" Method (The Independent Ingredient Approach)

The previous way of doing this, called RegMean, was a bit like a chef who looks at each ingredient in isolation.

The chef thinks: "Okay, the Italian chef uses a lot of basil, and the Thai chef uses a lot of basil. I'll just find a middle ground for the basil." They do this for the salt, the pepper, the oil, and the garlic, one by one.

The flaw? They forget that ingredients interact! If you add too much lime (from the Thai recipe) to a heavy cream sauce (from the Italian recipe) without considering how the acid affects the fat, the whole dish might curdle. RegMean merged the "layers" of the model independently, ignoring how information flows from one layer to the next. It was like mixing ingredients without realizing that the order and the way they react to each other matters.

The Solution: "RegMean++" (The Symphony Approach)

The authors of this paper introduced RegMean++. Instead of looking at ingredients in a vacuum, RegMean++ looks at the entire cooking process.

Think of it like a Symphony Orchestra.
In a standard orchestra (RegMean), the violin player just plays their notes, and the drummer just plays theirs. They are both playing the right notes, but they aren't necessarily "listening" to how the violin's melody changes the way the drums should hit.

RegMean++ is like a conductor. It realizes that the "sound" produced by the violins in the first movement changes the "vibe" for the cellos in the second movement. It accounts for dependencies:

Intra-layer dependency: How the different parts of a single instrument work together.
Cross-layer dependency: How the music from the first section of the song sets the stage for the next section.

By "listening" to how the merged model's layers interact with each other during the merging process, RegMean++ ensures that the final "song" (the merged AI model) is harmonious, smooth, and doesn't have any jarring, "curdled" moments.

Why does this matter? (The Results)

Because RegMean++ understands the "flow" of information, it produces a much smarter "Super-Chef." The researchers tested this on various AI tasks (like recognizing images or writing text) and found:

Better Generalization: The model isn't just good at what it was trained on; it's much better at handling "weird" or unexpected situations (like seeing a blurry photo or a different type of image).
Sustainability: You can keep adding new "chefs" (new tasks) to the mix over and over again, and the model doesn't "forget" the old recipes as easily.
Robustness: It’s harder to "break" the model. Even if the input is messy or noisy, the model stays steady.

In Short:

RegMean was like mixing colors by looking at each tube of paint separately.
RegMean++ is like painting a masterpiece by understanding how the colors bleed into one another on the canvas to create a beautiful, cohesive image.

Technical Summary: RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging

1. Problem Statement

Model merging aims to combine multiple task-specific "candidate models" into a single "merged model" with multi-task capabilities without the high computational cost of traditional multi-task learning or the need for original training data.

The existing state-of-the-art, Regression Mean (RegMean), formulates merging as a closed-form linear regression problem. It minimizes the discrepancy between the predictions of the merged linear layers and those of the candidate models. However, RegMean has a fundamental flaw: it merges each linear layer independently. By treating layers in isolation, it ignores the intra-layer (sub-component interactions) and cross-layer (how information propagates through the network) dependencies. This lack of structural awareness prevents the merged model from maintaining the complex feature representations necessary for high-level generalization and robustness.

2. Methodology: RegMean++

The authors propose RegMean++, an extension designed to explicitly incorporate the architectural dependencies of the transformer model into the merging objective.

Core Innovation (Cross-Layer Dependency): In standard RegMean, the input features ( $X^{(l)}_i$ ) used to calculate the regression for layer $l$ are taken directly from the candidate models. In RegMean++, the input features for the current layer $l$ are computed by passing the data through the previously merged layers of the merged model ( $f^{(l-1)}_M$ ). This ensures that the merging process accounts for how the evolving merged model transforms information as it flows through the depth of the network.
Mathematical Objective: RegMean++ retains the efficient closed-form solution of RegMean but updates the input statistics. It uses the activations produced by the merged model's preceding layers to compute the inner-product matrices ( $G^{(l,j)}_i$ ) used in the regression.
Algorithmic Flow:
1. Perform a forward pass through the merged layers to get current activations.
2. Run a forward pass through the candidate layers to get their specific activations.
3. Solve the regularized regression problem for each linear layer (MLP and Attention) using these aligned statistics.
4. Merge non-linear weights (LayerNorm, GELU, etc.) via simple averaging.

3. Key Contributions

Architectural Awareness: Introduced a method that captures the dynamic information flow within deep neural networks.
Layer-wise Insights: Through empirical analysis, the authors discovered that:
- Middle and deep layers are the most critical for merging; using only these preserves $>98\%$ accuracy.
- MLP modules are more effective sources for merging than attention heads, likely because they act as "knowledge dictionaries."
- Mid-depth layers often provide more reliable features for merging than the very last layer.
Versatility: Demonstrated effectiveness across both Vision (ViT architectures) and Language (Llama and Gemma models) tasks.

4. Experimental Results

The authors benchmarked RegMean++ against 11 advanced methods (including Data-Free, Training-Free, and Test-Time Adaptation methods) across diverse settings:

In-Domain (ID) & Out-of-Domain (OOD) Performance: RegMean++ consistently outperforms RegMean (e.g., an average improvement of 2.0% on ViT-B/32) and achieves competitive results with advanced test-time adaptation methods like DOGE AM.
Sustainability & Sequential Merging: In scenarios where tasks are added incrementally (sequential merging), RegMean++ shows superior stability and significantly less "catastrophic forgetting" of earlier tasks compared to other methods.
Robustness: RegMean++ demonstrated higher accuracy on corrupted datasets (e.g., Gaussian noise, JPEG compression), proving it creates more robust representations.
Scalability: It maintains high performance even as the number of merged tasks scales up to 20, whereas many other methods (like Task Arithmetic or Model Soups) show significant degradation.

5. Significance

RegMean++ represents a significant step forward in training-free model merging. By moving from "layer-wise independent merging" to "dependency-aware merging," it bridges the gap between simple weight averaging and complex re-training. Its ability to provide high-performance multi-task models with minimal computational overhead (only requiring a few extra forward passes) makes it highly practical for real-world applications where privacy constraints prevent access to training data and computational resources are limited.