🤖 machine learning

Constrained Bayesian Optimisation with Multiple Information Sources

This paper proposes a general multi-source framework for constrained Bayesian optimization that extends Max-value Entropy Search to efficiently identify feasible and optimal solutions by leveraging auxiliary data sources, even when they are only weakly correlated with the true objective.

Original authors: Hauke Maathuis, Roeland De Breuker, Saullo Castro, Maike Osborne

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hauke Maathuis, Roeland De Breuker, Saullo Castro, Maike Osborne

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the single best spot to set up a lemonade stand in a massive, foggy city. You have two goals:

Maximize Sales: You want the spot with the most foot traffic.
Stay Safe: You must avoid spots that are illegal (like on a highway) or dangerous (like a construction site).

The problem is that checking a spot is expensive.

The "Real" Check: Sending a person to stand there for an hour to count people and check for police is very costly and slow.
The "Fake" Check: Looking at a blurry, low-resolution map or asking a local who might be slightly wrong is cheap and fast, but not 100% accurate.

This is the core problem the paper tackles: How do you find the best, safe spot using as few expensive checks as possible, while using the cheap, imperfect maps to help you guess where to look?

The Old Way vs. The New Way

The Old Way (Traditional Bayesian Optimization):
Most previous methods acted like a cautious explorer who only trusted the expensive, real checks. If the "safe zone" (where you can legally sell lemonade) was tiny or hidden, the explorer would waste a lot of money checking random spots, often getting stuck in the fog. They ignored the cheap maps because they weren't perfect.

The New Way (MS-CMES):
The authors propose a new strategy called MS-CMES. Think of this as a smart team of explorers.

The Team: They have a "High-Fidelity" expert (expensive, accurate) and several "Low-Fidelity" assistants (cheap, fast, but sometimes wrong).
The Strategy: Instead of ignoring the assistants, the team uses them to scan the whole city quickly. The assistants point out "promising neighborhoods."
The Safety Net: The team knows the assistants might be wrong about the rules (e.g., they might think a highway is safe). So, the team uses a special math trick (called Variance Correction) to say, "Okay, the map says this spot is safe, but since the map is blurry, let's be a little more skeptical and check it carefully before we commit."

How It Works (The Metaphors)

1. The "Trust Region" (The Flashlight)
Imagine you are in a dark room. Instead of trying to map the whole room at once, you shine a flashlight on a small area. You explore that small area thoroughly. If you find something good, you move the flashlight closer to it. If you find nothing, you shrink the flashlight beam to look closer, or move it to a new promising spot.

In the paper: This is called a Trust Region. It stops the algorithm from wasting money checking random, useless parts of the city. It focuses the expensive checks on the most likely "winning" areas.

2. The "Blind Spot" Correction
Sometimes the cheap maps are so bad they are almost useless. If you blindly followed them, you might walk off a cliff.

In the paper: The method calculates a "correlation score." If the cheap map is only weakly related to reality, the system automatically adds a "safety buffer" (inflating the uncertainty). It essentially says, "This cheap data is shaky, so I won't trust it completely, but I'll still use it to get a rough idea of where to look."

3. The "Information Gain" (The Treasure Hunt)
The algorithm doesn't just look for the best spot; it looks for the spot that teaches it the most.

In the paper: It asks, "If I check this spot with the cheap map, will it tell me more about where the best spot is than if I checked that other spot?" It balances the cost of the check against the value of the new information.

What Did They Find?

The authors tested this on several "cities" (math problems) ranging from small towns to massive metropolises with thousands of streets (high dimensions).

The Result: Their new method (MS-CMES) was much better at finding the "safe" zones than the old methods.
The Surprise: Even when the cheap maps were very blurry and only weakly related to reality, the new method still used them to find the solution faster. The old methods often gave up or got stuck because they couldn't find a single safe spot to start with.
The Efficiency: By using the cheap maps to guide the search and only using the expensive checks to confirm the best spots, they found the optimal solution with far fewer total checks.

Summary

The paper introduces a smarter way to solve difficult optimization problems. Instead of relying solely on expensive, perfect data, it cleverly mixes in cheap, imperfect data to explore the space faster. It uses a "flashlight" approach to focus on promising areas and a "safety buffer" to ensure the cheap data doesn't lead you astray. The result is a system that finds the best, safe solutions much faster, even when the rules are complex and the data is scarce.

Technical Summary: Constrained Bayesian Optimisation with Multiple Information Sources

Problem Statement

Bayesian Optimisation (BO) is a standard framework for optimizing expensive black-box functions, yet it faces significant challenges when dealing with unknown constraints and small feasible regions. In high-dimensional spaces, where data is sparse (e.g., crashworthiness design), discovering even a single feasible point is difficult. Existing methods typically rely on a single information source (the "ground truth"), which is often computationally prohibitive to evaluate frequently.

While many engineering and scientific applications possess auxiliary information sources (e.g., coarse simulations, simplified physics models, or analytical approximations) that are cheaper to evaluate but potentially biased or noisy, their integration into constrained BO remains under-explored. Most existing multi-source (or multi-fidelity) methods address either constraints or multiple sources in isolation. There is a lack of scalable frameworks that can leverage cheap, weakly correlated auxiliary sources to guide early exploration in constrained settings without compromising the accuracy of the final solution.

Methodology: MS-CMES

The authors propose Multi-Source Constrained Max-value Entropy Search (MS-CMES), a unified framework that extends Max-value Entropy Search (MES) to settings with multiple information sources and black-box constraints.

1. Multi-Source Gaussian Process Modelling

The framework models the objective $f$ and constraints $c_i$ across a set of sources $S = \{0, \dots, L\}$ , where $L$ is the target (expensive, accurate) source and $0$ is the cheapest auxiliary source.

MISO Framework: Instead of assuming a strict hierarchical fidelity (as in Kennedy-O'Hagan models) or a shared latent structure (as in Multi-Task GPs), the authors adopt the Multi-Information Source Optimisation (MISO) approach.
Additive Structure: Each source-specific function is modeled as $u^{(\ell)}(x) = u^{(L)}(x) + \Delta^{(\ell)}(x)$ , where $u^{(L)}$ is a latent target function and $\Delta^{(\ell)}$ is a source-specific discrepancy modeled as an independent Gaussian Process (GP).
Joint Inference: This additive structure allows for joint inference over the latent function and discrepancies, capturing inter-source correlations while maintaining flexibility for sources with weak or unknown correlations.

2. Acquisition Function: Entropy Search with Variance Correction

The core of the method is an acquisition function $\alpha_n(x, \ell)$ that balances information gain about the constrained optimum $f^*$ against the evaluation cost $\lambda(x, \ell)$ .

Objective: Maximize the expected information gain about the constrained optimum $f^* = \max_{x \in \mathcal{X}_f^{(L)}} f^{(L)}(x)$ .
Variational Lower Bound: Since the mutual information $I(u^{(\ell)}; f^*)$ is intractable, the method uses a variational lower bound adapted from Takeno et al. (2022). This involves sampling the constrained optimum $f^*$ via discrete Thompson sampling and estimating the probability that a new observation falls within the feasible set defined by $f^*$ .
Variance Correction: A critical novelty is the variance correction mechanism. Because the feasible set is defined at the target source $L$ but observations may come from source $\ell$ , there is a risk of spurious feasibility. The authors introduce a correction factor based on the correlation coefficient $\rho(x, \ell)$ between the auxiliary and target sources. This inflates the variance of predictions from weakly correlated sources, preventing the algorithm from overconfidently selecting points that appear feasible on cheap sources but are infeasible on the target source.

3. Scalability via Trust Regions

To handle high-dimensional input spaces ( $d \ge 40$ ), the acquisition optimization is restricted to a dynamically updated Trust Region (TR) centered at the best observed feasible point (or the point with minimal constraint violation if none exists). This prevents the search from drifting into unpromising regions and improves computational efficiency.

Key Contributions

Unified Framework: The first principled, information-theoretic framework for constrained BO that integrates multiple information sources. It combines cheap auxiliary samples with a trust-region heuristic to scale to high-dimensional spaces.
Variance Correction: A systematic approach to handling discrepancies between sources by correcting predictive variances based on inter-source correlation, ensuring robustness even when auxiliary data is weakly correlated.
Systematic Comparison: A rigorous evaluation of scaling multi-source GP models (MISO vs. KOH vs. MTGP), demonstrating that the additive MISO structure is more robust to varying correlation strengths than hierarchical or strict multi-task approaches.
Empirical Validation: Extensive experiments on synthetic and physics-based benchmarks (up to $d=100$ ) showing consistent outperformance over state-of-the-art constrained BO methods.

Experimental Results

The authors evaluated MS-CMES on five benchmark problems (Pressure Vessel, Rosenbrock, Rastrigin, Rotated Rastrigin, Different Powers) with dimensions ranging from 4 to 100.

Feasibility Discovery: In high-dimensional, sparse feasible regions (e.g., Rastrigin variants), most baseline methods (including CMFBO, VBO, and random search) failed to find a single feasible point. MS-CMES successfully identified feasible regions immediately after the initial design of experiments.
Optimization Performance: MS-CMES consistently achieved lower final objective values compared to baselines like SCBO, FuRBO, and CMES-IBO+.
Robustness to Correlation: The MISO model maintained high accuracy even when auxiliary sources were uncorrelated or weakly correlated, whereas hierarchical models (KOH) degraded significantly in these scenarios.
Ablation Studies: The trust-region heuristic was shown to be crucial for efficiency in high dimensions, and the method proved robust to the number of Monte Carlo samples ( $K$ ) used for entropy estimation.

Significance and Claims

The paper claims that MS-CMES addresses a critical gap in the literature by providing a scalable solution for constrained optimization in the presence of multiple information sources.

Early Exploration: The method is particularly effective in the early stages of optimization where feasible regions are unknown. By leveraging cheap, potentially biased data, it can populate the design space and guide the search toward feasibility more efficiently than single-source methods.
General Applicability: The framework does not assume a strict fidelity hierarchy, making it suitable for diverse real-world scenarios where auxiliary data (e.g., simplified simulations, archival data) may be structurally divergent from the target.
Practical Impact: The authors highlight the potential for this approach in domains ranging from engineering design (e.g., crashworthiness, aerodynamics) to drug discovery, where balancing the cost of high-fidelity experiments with low-fidelity simulations is paramount.

The authors modestly note limitations, specifically the assumption that objectives and constraints are evaluated jointly at a source and the reliance on user-specified cost functions. However, they argue that the proposed framework offers a significant step forward in making constrained Bayesian optimization practical for complex, high-dimensional, and data-sparse engineering problems.