Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging

Technical Summary: Black-Box Optimization of Mixed Binary-Continuous Variables in Evolutionary Model Merging

Problem Definition
The paper addresses the optimization challenges inherent in Evolutionary Model Merging, specifically within the Data Flow Space (DFS). While model merging offers a cost-effective alternative to training Large Language Models (LLMs) from scratch, the automated search for optimal merging configurations using evolutionary algorithms remains poorly understood. The authors identify that DFS merging involves a complex search space comprising:

1. Mixed Binary-Continuous Variables: The problem requires simultaneously optimizing discrete decisions (which layers to include, represented by a binary vector $z$) and continuous parameters (scaling weights for selected layers, represented by a vector $x$).
2. High Dimensionality: Modern LLMs involve hundreds of layers across multiple models, creating search spaces with thousands of dimensions.
3. Conditional Dependencies (Type-I Interaction): A critical structural challenge where the effective dimensionality of the continuous variables depends entirely on the binary configuration. If a layer is not selected ($z_{i,j}=0$), its corresponding scaling weight ($x_{i,j}$) becomes irrelevant to the objective function.

The paper argues that standard black-box optimization methods, such as CMA-ES, are ill-suited for this problem because they are designed primarily for continuous variables and fail to account for these mixed-variable interactions and conditional dependencies.

Methodology and Formulation
The authors formally characterize the DFS merging problem as a black-box optimization problem with mixed binary-continuous variables. The objective is to minimize a function $f(x, z)$ representing the merged model's performance on a target task, subject to:

• $z \in \{0, 1\}^m$ (Binary layer selection)
• $x \in \mathbb{R}^n$ (Continuous scaling weights)

The formulation explicitly highlights the Type-I interaction (as defined by Akimoto et al. [11]), where the active continuous dimensions are conditional on the binary vector. The paper contrasts two approaches:

• Unstructured DFS: Treats the problem as a generic black-box search, optimizing all parameter groups regardless of selection status, thereby wasting evaluations on irrelevant dimensions.
• Structured DFS: Respects the conditional dependency by first selecting active layers (optimizing $z$) and then optimizing scaling weights only for those selected layers (optimizing $x|z$).

Key Contributions

1. Structured Survey: The paper categorizes existing evolutionary model merging techniques into three distinct groups: Parameter-Space (PS) merging, Data Flow Space (DFS) merging, and Hybrid approaches.
2. Formal Characterization: It provides the first formal definition of the DFS merging problem as a mixed binary-continuous black-box optimization problem, explicitly identifying the conditional dependencies that hinder standard solvers.
3. Bridging Communities: The work connects the model merging community with the broader evolutionary computation and black-box optimization literature, specifically highlighting the relevance of algorithms like CatCMA and ICatCMA (which address similar interaction challenges) to the model merging context.

Experimental Results
The authors conducted preliminary empirical validation using two pre-trained models (google/flan-t5-small and declare-lab/flan-alpaca-large) on a 15-question benchmark. They compared an unstructured DFS approach against a structured DFS approach over 10 iterations:

• Accuracy: The Structured DFS approach achieved 26.7% accuracy, matching the baseline (Model A alone), whereas the Unstructured DFS approach degraded to 20.0%. This represents a 6.7% improvement over the unstructured method.
• Search Efficiency: The structured approach reduced the effective search space by 51.4%. While the unstructured method modified all 192 parameter groups every iteration, the structured method dynamically focused on an average of 93.3 layers per iteration.
• Dynamic Dimensionality: The experiments confirmed that the number of active layers varied dynamically (from 53 to 125) across iterations, validating the hypothesis that the effective continuous dimensionality changes based on the binary configuration.

Significance and Claims
The paper claims that the optimization challenges identified—specifically the mixed binary-continuous structure with conditional dependencies—are not unique to model merging but arise in any domain requiring simultaneous optimization of combinatorial and continuous variables in a black-box setting (e.g., Neural Architecture Search, drug discovery).

The authors position their work as a foundational step to:

• Explain why current evolutionary model merging approaches (which often ignore these structural properties) may be inefficient or suboptimal.
• Motivate the application of advanced mixed-variable optimizers (like ICatCMA with warm-starting and hyper-representation techniques) to model merging.
• Propose future research directions, including adapting CatCMA for layer selection, developing theoretical convergence guarantees for mixed-variable DFS search, and scaling these methods to larger models (e.g., 7B parameters) using surrogate-assisted search.

The authors remain modest regarding their current findings, acknowledging that the study is preliminary, uses small models (76M–247M parameters), and relies on random sampling rather than full evolutionary algorithms. They frame the results as empirical validation of the theoretical formulation, motivating larger-scale future work.