🤖 machine learning

Multi-Objective Bayesian Optimization for Model Merging

This paper introduces MOBO-Merge, a framework that employs multi-objective Bayesian optimization to efficiently select merge parameters for combining multiple models in weight space, demonstrating superior performance over random search across various merge operators and model configurations.

Original authors: Utkarsh Agarwal, Vamshi Bonagiri, Raul Astudillo, Monojit Choudhury

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Utkarsh Agarwal, Vamshi Bonagiri, Raul Astudillo, Monojit Choudhury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef who has spent years perfecting two different recipes: one is the world's best chocolate cake, and the other is the most savory, complex lasagna. You have a magical kitchen where you can't just mix the ingredients together in a bowl and hope for the best; instead, you have to blend the entire finished cakes and lasagnas layer by layer to create a new dish. This is the world of Artificial Intelligence (AI) model merging. In this corner of computer science, researchers take two or more AI models that have already been trained to do specific tasks—like one that is great at solving math problems and another that is excellent at writing code—and try to combine them into a single, super-smart model without having to retrain them from scratch.

The tricky part is figuring out how to mix them. If you mix them 50-50, you might get a mediocre cake-lasagna that tastes like neither. If you mix them 90-10, you might lose the lasagna flavor entirely. This is called the merge parameter problem. Usually, finding the perfect mix is like trying to find a needle in a haystack by randomly grabbing handfuls of hay. It's expensive, time-consuming, and often frustrating because you can't see the "recipe" (the math behind the AI) to guide you. This paper asks a simple but powerful question: Can we use a smart, guessing-based strategy to find the best mixes much faster than just guessing randomly?

The Smart Chef's Guide to Mixing AI Models

Meet MOBO-Merge, the new "smart chef" introduced in this paper. The authors, a team from Mohamed bin Zayed University of Artificial Intelligence, realized that mixing AI models is a bit like trying to find the perfect balance between two competing flavors. You want your new AI to be good at both instruction-following (listening to you) and math reasoning (solving problems), but usually, making it better at one makes it slightly worse at the other.

Instead of trying to find one single "perfect" mix, the authors decided to look for the Pareto front. Think of this as a map of all the "best possible compromises." On this map, every point represents a mix where you can't get more math skill without losing some instruction skill, and vice versa. The goal isn't to pick just one winner, but to discover the whole landscape of great options so you can choose the one that fits your needs best.

The Problem: The "Black Box" Kitchen

The challenge is that testing a new mix is incredibly expensive. To see if a mix works, you have to build the new AI model and run it through hundreds of test questions. It's like baking a whole new cake just to taste a spoonful. You can't taste it while it's baking (no "gradients" to guide you), and you only have a limited number of ingredients (a limited budget for tests).

If you just tried random mixes (like throwing darts at a board), you might get lucky, but you'd waste a lot of time and ingredients. If the mixing recipe is simple (like just blending two models with one knob), random guessing isn't too bad. But if the recipe is complex—like having different knobs for different layers of the cake—random guessing becomes a disaster.

The Solution: The "Smart Guessing" Machine

The authors created MOBO-Merge, a framework that uses Multi-Objective Bayesian Optimization. In plain English, this is a "smart guessing" machine.

Here is how it works:

The Surrogate Model: Instead of baking a cake every time, the machine builds a "prediction map" based on the few mixes it has already tested. It learns, "Oh, when we mix 30% of Model A and 70% of Model B, the math score goes up, but the instruction score drops."
The Acquisition Function: This is the machine's intuition. It looks at the map and asks, "Where should I test next to learn the most?" It doesn't just look for the highest score; it looks for the spots that will help it draw the best "compromise map" (the Pareto front).
The Loop: It picks a new mix, tests it, updates its map, and repeats.

The authors tested this on two famous AI families: Qwen3-4B and Llama-3.1-8B. They tried mixing them in different ways:

Linear: A simple blend (one knob).
TIES: A more complex method that tries to fix "clashing" instructions between models (four knobs).
Block-Linear: A method that treats different layers of the AI like separate blocks, allowing for very fine-tuned mixing (four or eight knobs).

What They Found

The results were like a treasure hunt with a very smart map.

Simple Mixes: When the mixing was simple (just one knob), the "smart guessing" (MOBO-Merge) was only slightly better than random guessing. In fact, for one specific test with the Qwen model, random guessing actually did a tiny bit better. This suggests that for simple recipes, you don't need a super-computer to find the mix.
Complex Mixes: But when the mixing got complicated (using TIES or Block-Linear with multiple knobs), MOBO-Merge shined. It found much better mixes than random guessing. For example, with the Llama model using the TIES method, MOBO-Merge found a mix that was significantly better than what random search could find.
The "Three-Model" Challenge: The authors also tried mixing three models at once (Instruction + Math + Code). Here, the advantage of the smart guessing was huge. In one case with the Llama model, MOBO-Merge found a mix that was more than double as good as the best random mix.

One of the most interesting discoveries was that there is no single "best" mixing method. Sometimes the simple Linear method works best; other times, the complex TIES or Block-Linear methods win. It depends entirely on which AI models you are mixing and what you are trying to achieve.

Why This Matters

The paper suggests that MOBO-Merge is a powerful tool for anyone trying to combine AI models. It doesn't invent a new way to mix the models; instead, it provides a smarter way to search for the right mix.

By using this method, researchers can find high-quality combinations of AI capabilities without needing to run thousands of expensive tests. It turns a process that used to be a game of luck into a systematic exploration. The authors found that in 11 out of 12 different test scenarios, MOBO-Merge found better "compromise maps" than random guessing.

However, the authors are careful to note that this isn't a magic wand that solves everything. The method still requires running tests, which costs money and time. It works best when the mixing recipe is complex, and it doesn't guarantee that the mix will work perfectly on every single task. But for those looking to build flexible, multi-skilled AI systems without retraining them from scratch, this "smart guessing" approach offers a much more efficient path forward.

In the end, the paper shows that while we can't always predict the perfect recipe for a super-AI, we can certainly stop guessing blindly and start exploring the possibilities with a little bit of mathematical intuition.

Technical Summary: Multi-Objective Bayesian Optimization for Model Merging

Problem Formulation

Model merging offers a compute-efficient alternative to additional fine-tuning by combining trained models directly in weight space. However, selecting optimal merge parameters is challenging because downstream evaluations are expensive, gradients with respect to merge parameters are unavailable, and source models often possess conflicting capabilities (e.g., instruction following vs. mathematical reasoning).

The authors formulate merge-parameter selection as an expensive black-box multi-objective optimization problem. In this setting:

Input: A low-dimensional merge parameter space $X$ (e.g., global weights, block-wise coefficients) and a set of source models $\theta_1, \dots, \theta_K$ .
Process: A merge operator $M$ combines source models into a merged model $\theta(x) = M(\theta_1, \dots, \theta_K; x)$ .
Objective: Maximize a vector of performance metrics $f(x) = (f_1(\theta(x)), \dots, f_M(\theta(x)))$ , where each $f_i$ corresponds to a distinct capability (e.g., instruction, math, code).
Constraint: Evaluations are costly and noisy, limiting the total number of configurations that can be tested.
Goal: Instead of finding a single optimal solution, the goal is to efficiently approximate the Pareto front—the set of non-dominated solutions that expose the optimal trade-offs between competing capabilities.

Methodology: MOBO-Merge

The paper introduces MOBO-Merge, a framework that applies multi-objective Bayesian Optimization (MOBO) to the model merging task. The framework is merge-operator agnostic, meaning it can be applied to various merging strategies (Linear, SLERP, TIES, block-wise) without modification to the optimization core.

The algorithm operates as follows:

Probabilistic Surrogate Modeling: MOBO-Merge maintains a probabilistic model of the objective functions based on previously evaluated merge parameters. In the experiments, this is instantiated using independent Gaussian Process (GP) surrogates for each objective.
Acquisition Function: To select the next merge configuration, the framework uses a Noisy Expected Hypervolume Improvement (NEHVI) acquisition function. This function scores candidate parameters based on their expected contribution to the quality of the current Pareto front approximation, specifically targeting the expansion of the dominated hypervolume.
Sequential Optimization: The process iterates:
- Fit the surrogate model to the current dataset.
- Optimize the acquisition function to select the next merge parameter $x_{t+1}$ .
- Instantiate the merged model, evaluate it on downstream benchmarks, and update the dataset.
Output: The algorithm returns the set of non-dominated solutions found during the search, approximating the Pareto front.

Experimental Setup

The authors evaluated MOBO-Merge on two model families: Qwen3-4B and Llama-3.1-8B.

Settings:
- Two-model (AB): Merging an instruction-following checkpoint and a mathematical-reasoning checkpoint.
- Three-model (ABC): Adding a code-generation checkpoint to the mix.
Merge Operators:
- Linear: Global weighted average.
- SLERP: Spherical linear interpolation (restricted to two-model settings in the main comparison).
- TIES: Interference-aware merging that trims and resolves sign conflicts.
- Block-Linear 4x: A block-wise linear interpolation where layers are partitioned into four disjoint blocks, each with its own mixing coefficient.
Baselines: MOBO-Merge (using NEHVI) was compared against uniform random search over the same parameter space and evaluation budget (100 sequential steps).
Metrics: Performance was measured on held-out partitions of IFEval (instruction), GSM8K (math), and HumanEval (code). The primary metric for comparing search strategies was the hypervolume of the recovered Pareto front.

Key Results

The experiments demonstrate that guided multi-objective search significantly outperforms random search, particularly in higher-dimensional parameter spaces:

Superiority over Random Search: Across 12 distinct experimental configurations (combinations of model family, objective set, and merge operator), MOBO-Merge achieved a higher mean held-out hypervolume than random search in 11 of 12 cases.
Dimensionality Dependence:
- One-dimensional (Linear): The gain was marginal. For Qwen AB, random search slightly outperformed MOBO-Merge; for Llama AB, the gain was small. This suggests that for simple global interpolation, random search is already competitive.
- Higher-dimensional (TIES, Block-Linear): The gains were substantial. For example, in the Llama ABC setting, MOBO-Merge with Block-Linear 4x achieved a hypervolume of 0.0634 compared to 0.0289 for random search. Similarly, for Llama AB with TIES, MOBO-Merge reached 0.1145 versus 0.0764 for random search.
Operator Variability: No single merge operator was uniformly best.
- TIES performed best in three of the four family-setting combinations.
- Block-Linear 4x was the strongest performer for the Llama three-model (ABC) merge.
Pareto Front Quality: Visualizations of the Pareto fronts showed that MOBO-Merge consistently discovered configurations that offered better trade-offs than those found by random search. In some cases (e.g., Qwen), the method found merges that simultaneously outperformed both source models on their respective primary tasks.

Significance and Claims

The paper claims that MOBO-Merge provides a principled, general, and sample-efficient framework for model composition. Its primary contributions and significance are:

Reframing Merging as Optimization: It establishes model merging not as a fixed procedure but as a black-box multi-objective optimization problem, allowing for the systematic discovery of capability trade-offs.
Efficiency in Expressive Spaces: The framework is particularly valuable for "expressive" merge parameterizations (like TIES and block-wise mixing) where the search space is too large for exhaustive or random search to be effective. It enables practitioners to find high-quality trade-offs with significantly fewer evaluations.
Operator Agnosticism: By decoupling the search strategy from the merge operator, MOBO-Merge allows for a fair comparison of different merging strategies and facilitates the discovery of which operator works best for specific model families and objective sets.
Practical Utility: The method replaces ad-hoc merge-weight tuning with an automated process that exposes controllable trade-offs, enabling practitioners to select models that best match their specific needs under fixed compute budgets.

The authors conclude that while simple linear interpolation may not require complex optimization, the increasing complexity of modern merging strategies (block-wise, interference-aware) makes multi-objective Bayesian optimization a necessary tool for unlocking strong performance trade-offs.