💻 computer science

Surrogate Benchmarks for Model Merging Optimization

This paper introduces low-cost surrogate benchmarks that utilize constructed models to predict merged model performance from hyperparameters, thereby enabling efficient development and comparison of hyperparameter optimization algorithms for model merging without the computational expense of full-scale experiments.

Original authors: Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa

Published 2026-06-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rio Akizuki, Yuya Kudo, Nozomu Yoshinari, Yoichi Hirose, Toshiyuki Nishimoto, Kento Uchida, Shinichi Shirakawa

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two expert chefs, Chef A and Chef B. Chef A is amazing at making spicy curry, and Chef B is a master of delicate sushi. You want to create a single "Super Chef" who can do both perfectly.

In the world of Artificial Intelligence, this is called Model Merging. Instead of training a new chef from scratch (which takes years and huge amounts of ingredients/data), you try to blend the "recipes" (the internal settings) of Chef A and Chef B together.

The Problem: The "Secret Sauce" is Hard to Find

The tricky part is that blending these recipes isn't just a simple mix. You have to decide exactly how much of Chef A's style to keep and how much of Chef B's to add for every single step of the cooking process. These decisions are called hyperparameters.

If you get the mix right, you get a Super Chef. If you get it wrong, the result is a disaster.

To find the perfect mix, researchers usually use a trial-and-error method (like an evolutionary algorithm). They try thousands of different combinations, test the result, and keep the best ones. But here's the catch: Testing a single combination is incredibly slow and expensive. It's like hiring a team of food critics to taste-test a new dish, which takes 5 minutes per dish. If you need to test 1,000 combinations, that's over 58 hours of non-stop tasting! This makes it very hard for scientists to invent new, better ways to find the perfect mix because the cost is too high.

The Solution: A "Crystal Ball" (Surrogate Benchmarks)

The authors of this paper, from Yokohama National University, asked: "What if we could predict how good a mix will be without actually tasting it every time?"

They built a Surrogate Benchmark, which they call SMM-Bench. Think of this as a Crystal Ball or a Super-accurate Food Critic Simulator.

Here is how they built it:

The Training Phase: They actually ran the expensive, slow tasting process thousands of times (over 130,000 times for one type of mix and 40,000 for another). They recorded every "recipe setting" they tried and the resulting "taste score."
The Learning Phase: They fed all this data into a smart computer program (a machine learning model). This program learned the patterns: "Oh, when we use 0.8 of Chef A's spice and 0.2 of Chef B's, the score is usually high."
The Result: Now, instead of spending 5 minutes to taste a new recipe, the Crystal Ball can predict the score in a fraction of a second.

What They Tested

They built two versions of this Crystal Ball:

SMM-Bench-PS: For blending the "ingredients" directly (like mixing flour and sugar).
SMM-Bench-DFS: For blending the "cooking steps" (like deciding whether to use Chef A's knife technique or Chef B's chopping style).

They tested their Crystal Ball by letting it predict the results of new experiments. The Crystal Ball was surprisingly accurate, predicting the "taste scores" with about 95% accuracy compared to the real thing.

Why This Matters

The paper shows that researchers can now use this Crystal Ball to:

Test new ideas instantly: Instead of waiting days to see if a new mixing algorithm works, they can run it on the Crystal Ball in minutes on a regular laptop.
Compare fairly: Everyone can use the same simulator to see which algorithm is best, without needing expensive supercomputers.

In the paper, they used their Crystal Ball to compare two different "mixing strategies" (Sep-CMA and DE). They found that one strategy was slightly better than the other, and they did this entire comparison in just a few minutes. If they had used the real "tasting" method, it would have taken days and required powerful graphics cards.

The Bottom Line

This paper doesn't invent a new way to merge models itself. Instead, it invents a low-cost simulator that lets scientists practice, test, and improve their model-merging strategies without breaking the bank or waiting weeks for results. It's like giving chefs a simulation game to practice their fusion recipes before they ever step into a real kitchen.

Technical Summary: Surrogate Benchmarks for Model Merging Optimization

Problem Statement
Model merging techniques integrate multiple models into a single model, often enhancing capabilities without requiring large datasets or expensive computations. However, most merging techniques rely on hyperparameters that significantly impact the final model's performance. While recent works, such as Akiba et al. (2025), have demonstrated that optimizing these hyperparameters (e.g., using evolutionary algorithms like CMA-ES) yields superior results, the process is computationally prohibitive. Evaluating a single merging configuration can take several minutes on high-end GPUs; consequently, running optimization algorithms that require thousands of evaluations (e.g., 1,000 evaluations taking over 58 hours) creates a substantial barrier to developing new optimization algorithms or conducting fair, reproducible comparisons. Existing benchmarks for model merging evaluate final performance but do not reduce the computational cost of the optimization process itself.

Methodology: SMM-Bench
The authors propose SMM-Bench, the first surrogate benchmark designed specifically for model merging optimization. The core approach involves collecting data on merging hyperparameters and their corresponding evaluation scores to train surrogate models that predict performance without executing the actual merging and inference steps.

The benchmark is defined across two distinct search spaces, mirroring previous optimization settings:

SMM-Bench-PS (Parameter Space): Focuses on layer-wise merging using task arithmetic with two source LLMs (Shisa-gamma-7b-v12 and WizardMath-7B-V1.1). The search space consists of 64 continuous design variables (weights for each layer of the two models), restricted to the range $[0, 1]$ .
SMM-Bench-DFS (Data Flow Space): Focuses on layer stacking and input scaling. It utilizes two source models (EvoLLM-JP-v1-7B and Shisa-gamma-7b-v1) and defines a mixed search space comprising 32 categorical variables (selecting layers from Model A, Model B, or no insertion) and 63 continuous variables (input scaling factors for layers).

Data Collection and Model Construction:

Data Generation: The authors collected over 133,000 data points for PS and over 40,000 for DFS. Data was gathered using random sampling, CMA-ES, and Tree-structured Parzen Estimator (TPE) strategies to ensure diverse coverage of the search space.
Surrogate Models: The collected data was split 9:1 for training and testing. LightGBM models were trained to predict evaluation scores (accuracy on Japanese math datasets: gsm8k-ja and MGSM) based on input hyperparameters. The models were optimized using Optuna with five-fold cross-validation.

Key Results

Predictive Accuracy: The surrogate models demonstrated high predictive performance. For SMM-Bench-PS, the models achieved $R^2$ scores of 0.950 (gsm8k-ja) and 0.921 (MGSM). For SMM-Bench-DFS, scores were 0.962 and 0.957, respectively. Kendall's Tau coefficients also indicated strong rank correlation.
Optimization Simulation: The authors simulated optimization trajectories using Random Search, CMA-ES, and TPE on the surrogate benchmarks. The results showed that the surrogate benchmarks successfully captured the behavior of these algorithms compared to "true" benchmarks (actual model merging evaluations). While the DFS surrogate tended to slightly overestimate performance, it still effectively simulated optimization trends.
Algorithm Comparison: The authors demonstrated the utility of the benchmark by comparing Separable CMA-ES (Sep-CMA) and Differential Evolution (DE) on SMM-Bench-PS. Sep-CMA outperformed DE on both datasets. Crucially, this comparison was conducted in minutes on a laptop, whereas performing the same evaluation with actual LLM merging would require many GPU days.

Significance and Claims
The paper claims that SMM-Bench provides a low-cost mechanism for the development and evaluation of model merging optimization algorithms. By replacing expensive actual evaluations with fast surrogate predictions, the benchmark enables:

Rapid Algorithm Development: Researchers can iterate on new optimization strategies without incurring the massive computational costs associated with LLM merging.
Reproducible Comparisons: It facilitates fair and reproducible comparisons between different optimization methods (e.g., evolutionary strategies, Bayesian optimization) under standardized conditions.
Accessibility: The benchmark lowers the barrier to entry for AutoML research in the domain of model merging, allowing experiments that would otherwise be infeasible due to resource constraints.

The authors position this work as a foundational step toward sophisticated model merging optimization, analogous to the role surrogate benchmarks play in Neural Architecture Search (NAS) and general hyperparameter optimization.

The Problem: The "Secret Sauce" is Hard to Find

The Solution: A "Crystal Ball" (Surrogate Benchmarks)

What They Tested

Why This Matters

The Bottom Line

More like this