🤖 machine learning

Exploiting Separability in Multi-Scale Grey-Box Bayesian Optimization

This paper proposes a bilevel Bayesian optimization framework for grey-box problems that exploits variable separability by using an outer loop to optimize black-box variables and an inner loop to exactly solve white-box subproblems, thereby satisfying constraints without penalties and achieving superior performance in terms of regret, iterations, and wall-clock time compared to existing methods.

Original authors: Joshua E. Hammond, Tyler A. Soderstrom, Brian A. Korgel, Michael Baldea

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joshua E. Hammond, Tyler A. Soderstrom, Brian A. Korgel, Michael Baldea

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to invent the world's most delicious cake. You have two types of ingredients to juggle. First, you have the "secret sauce"—a mysterious, expensive liquid that only a master alchemist can brew. You don't know exactly how it works, and brewing a single batch takes hours of expensive equipment and rare materials. Second, you have the "standard recipe"—flour, sugar, and eggs. You know exactly how these interact; you have a perfect, written cookbook that tells you precisely how much flour to mix with how much sugar to get the right texture.

In the world of engineering and science, this is a common puzzle called optimization. Scientists and engineers are constantly trying to find the "best" settings for complex systems, like designing a new drug, a more efficient engine, or a better chemical plant. Usually, they use a method called Bayesian Optimization. Think of this as a super-smart, curious robot that tries to guess the best settings by testing a few, learning from the results, and guessing again. The robot builds a "map" of the possibilities to find the peak of the mountain (the best solution) without having to climb every single step.

But here's the problem: if the robot tries to learn the map for both the secret sauce and the standard recipe all at once, it gets overwhelmed. It wastes time and money trying to "learn" the flour and sugar, even though we already have the perfect cookbook for those! It's like hiring a genius to teach you how to tie your shoelaces when you've been doing it since you were three. This paper tackles that exact inefficiency.

The Paper's Big Idea: Splitting the Job

The authors, Joshua Hammond and his team, realized that many real-world problems have this exact "split personality." They have a black-box part (the expensive, unknown secret sauce) and a white-box part (the cheap, known recipe). Their solution is to stop trying to learn everything at once and instead split the job into two levels, like a boss and a specialist.

They call this a bilevel approach. Imagine a smart manager (the outer loop) who only worries about ordering the secret sauce. The manager doesn't need to know how to bake the cake; they just need to know which secret sauce works best. Once the manager picks a sauce, they hand it to a specialist baker (the inner loop). The baker knows the recipe perfectly. They take that specific sauce and immediately calculate the perfect amount of flour and sugar to go with it, solving the "white-box" part exactly and instantly.

By doing this, the manager's "map" only has to cover the secret sauce, not the whole kitchen. This makes the map much smaller and much easier to draw. The robot doesn't waste energy guessing how to mix sugar; it just focuses on finding the right sauce.

What They Found: A Massive Speedup

To prove this works, the team created a playground of 13 different problems. These ranged from simple math puzzles to realistic engineering challenges, like designing a heat exchanger for a factory or optimizing a chemical reactor. In these tests, they compared their new "split-job" method against the old "do-it-all" method.

The results were staggering. In these simulations, the new method found better solutions 11 to 108 times faster (in terms of "regret," which is a fancy way of saying how far off the best answer they were) than the old method. In some cases, like designing a distillation column for separating chemicals, the old method was so confused by the complexity that the new method was 106 times better.

Crucially, the new method didn't just find better answers; it found them without wasting time. The old method often tried to test settings that were physically impossible (like trying to bake a cake with negative sugar), wasting expensive "secret sauce" on dead ends. The new method, because it used the "specialist baker" to check the rules instantly, almost never wasted a single drop of the expensive sauce on impossible ideas.

The Catch and the Confidence

The paper is very clear about what this method doesn't do. It doesn't work if the "recipe" part is also a mystery or if the recipe changes depending on the secret sauce in a way that can't be solved exactly. The method relies on the "white-box" part being solvable by a computer quickly.

The authors tested this rigorously. They ran 8,450 independent experiments (a huge number for this kind of study) to make sure their results weren't just luck. They found that the method is robust, meaning it works well even if you change the settings or the type of "specialist baker" you use. However, they also noted a trade-off: if the "recipe" part is very tricky with many local traps (like a maze with many dead ends), the "specialist" needs to be very smart to find the right path. If the specialist isn't smart enough, the whole system might get stuck in a local trap.

In short, the paper suggests that when you have a mix of expensive unknowns and cheap knowns, you shouldn't treat them as one big mess. Instead, separate them. Let the smart robot handle the unknowns, and let the math handle the knowns. It's a simple idea, but in the world of expensive engineering experiments, it turns a slow, stumbling walk into a sprint.

Technical Summary: Exploiting Separability in Multi-Scale Grey-Box Bayesian Optimization

Problem Definition

The paper addresses grey-box optimization problems where the decision variables naturally partition into two distinct groups:

Black-box variables ( $x_{BB}$ ): Inputs to an expensive, non-differentiable function $f_{BB}$ (e.g., outputs from Density Functional Theory simulations or complex experiments).
White-box variables ( $x_{WB}$ ): Variables governed by known, differentiable, closed-form equations (e.g., mass/energy balances, thermodynamic relations) that depend on the output of the black-box function.

The objective is to minimize a function $J(x_{WB}, y, x_{BB})$ subject to white-box equality constraints $f_{WB}(x_{WB}, y) = 0$ and inequality constraints $g(x_{WB}, y, x_{BB}) \leq 0$ , where $y = f_{BB}(x_{BB})$ .

The core challenge identified is that standard monolithic surrogate-based optimization (treating the entire system as a single black box) forces the surrogate model to learn both the unknown black-box structure and the known white-box physics. This redundancy leads to the "curse of dimensionality," requiring significantly more expensive evaluations to build an accurate model over the combined space $[x_{WB}, x_{BB}]$ . Furthermore, standard methods often handle constraints via penalty functions or chance constraints, which can lead to infeasible solutions or approximation errors.

Methodology: Bilevel Reformulation

The authors propose a bilevel Bayesian optimization (BO) framework that exploits the separability of the problem structure. The method relies on Assumption 1, which posits that for any fixed black-box input $x_{BB}$ , the resulting optimization over the white-box variables $x_{WB}$ is a well-posed nonlinear program (NLP) solvable to global optimality.

The optimization is decomposed as follows:

Outer Loop (Bayesian Optimization): A Gaussian Process (GP) surrogate is built only over the low-dimensional black-box space $x_{BB}$ . The surrogate models the scalar value function $x_{BB} \mapsto J(x_{WB}^*, y, x_{BB})$ , where $x_{WB}^*$ is the optimal solution to the inner problem. The acquisition function (Expected Improvement) selects the next candidate $x_{BB}$ .
Inner Loop (Global NLP Solver): For every candidate $x_{BB}$ proposed by the outer loop, the black-box function is evaluated to obtain $y$ . Then, a global optimizer (specifically Basin-Hopping with SLSQP local minimization, or multi-start SLSQP) solves the white-box subproblem exactly:
$\min_{x_{WB}} J(x_{WB}, y, x_{BB}) \quad \text{s.t.} \quad f_{WB}(x_{WB}, y) = 0, \quad g(x_{WB}, y, x_{BB}) \leq 0$
If the inner problem is infeasible for a given $y$ , a large penalty is returned to the outer loop.

Key Technical Properties:

Dimensionality Reduction: The GP surrogate operates in $\mathbb{R}^{n_{BB}}$ rather than $\mathbb{R}^{n_{WB} + n_{BB}}$ .
Exact Constraint Satisfaction: White-box constraints are satisfied exactly by the inner solver whenever it converges to a feasible point, eliminating the need for penalty functions, moment approximations, or chance constraints within the surrogate model.
Scalar Surrogate: Unlike methods that surrogate intermediate outputs $y$ (requiring multi-output GPs), this method surrogates the final optimal objective directly.

Key Contributions

Bilevel Reformulation: A novel decomposition strategy that reduces surrogate dimensionality from $n_{WB} + n_{BB}$ to $n_{BB}$ while ensuring exact constraint satisfaction for the white-box subproblem without penalty functions.
Benchmark Suite: The creation of a comprehensive suite of 13 separable grey-box problems (ranging from synthetic functions to engineering applications like CSTR design, heat exchangers, and membrane separation). This suite includes 2–5 total variables and 0–3 constraints, with verified global optima.
Empirical Evidence: Extensive testing across 8,450 independent optimization runs (8,190 BO runs, 130 NLP runs, 130 Basin-Hopping runs). The study systematically evaluates hyperparameters (initial sample size, exploration parameter) and inner-solver choices.

Results

The proposed bilevel BO strategy demonstrates significant performance improvements over standard black-box BO and global search baselines:

Regret Reduction: Bilevel BO achieves 11× to 108× lower regret than black-box BO across all 13 problems. The geometric mean improvement is 3,192× for the SLSQP inner solver variant.
Efficiency: The method reaches high-quality solutions with fewer black-box evaluations. On 10 of 13 problems, bilevel BO matches the final performance of black-box BO (after 250 evaluations) using less than 10% of the evaluation budget.
Feasibility: Bilevel BO achieves 100% feasibility on 8 of 12 constrained problems, whereas black-box BO wastes 6–72% of its budget on infeasible queries.
Robustness: The advantage holds across different initialization sizes and exploration parameters.
Solver Comparison: While the multi-start SLSQP inner solver is generally faster and more robust for narrow feasible regions, the Basin-Hopping (BH) inner solver provides superior performance on problems with highly multimodal inner landscapes (e.g., Rastrigin), though it can struggle with narrow feasible regions (e.g., Small-Feasible-Region 1).

Significance and Claims

The paper claims that the primary source of the observed advantage is structural, not algorithmic. The gains are derived entirely from exploiting the separable problem structure—reducing the surrogate dimension and satisfying constraints exactly—rather than from novel acquisition functions or kernel choices.

The authors argue that monolithic surrogates "waste samples learning what is already known." By solving the known subproblem (white-box) exactly and approximating only the unknown component (black-box), the method overcomes the curse of dimensionality inherent in high-dimensional grey-box problems. The paper positions this bilevel decomposition as a principled approach for multi-scale engineering design where expensive simulations (black-box) feed into well-understood macroscopic models (white-box).

The work does not claim to solve non-separable composite functions or to replace methods that rely on uncertainty propagation for risk-averse decision-making (such as COBALT or BOCF), noting that those methods address settings where the separability assumption does not hold. Instead, it offers a specialized, high-performance solution for the specific class of problems where variable separability exists.