🤖 machine learning

Regret Bounds for Expected Improvement Algorithms in Gaussian Process Bandit Optimization

This article resolves the open question of the convergence of Expected Improvement in noisy Gaussian Process bandit optimization by proposing a variant with a standard incident that achieves a regret bound of $\mathcal{O}(\gamma_T\sqrt{T})$ without requiring prior knowledge of the RKHS norm or noise parameters, and furthermore introduces an improved algorithm that converges faster than existing counterparts.

Original authors: Hung Tran-The, Sunil Gupta, Santu Rana, Svetha Venkatesh

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hung Tran-The, Sunil Gupta, Santu Rana, Svetha Venkatesh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find the highest peak in a vast, foggy mountain range. You cannot see the entire map, and every time you take a step to check your altitude, your altimeter provides a slightly shaky, noisy measurement. This is the problem of Gaussian Process Bandit Optimization: searching for the best solution to a complex problem when only noisy, incomplete information is available.

To solve this, you need a strategy. The most popular strategy is called Expected Improvement (EI). Imagine EI as a hiker who asks: "If I go to this new location, how much better will my view be compared to the best location I have seen so far?"

The Problem: The "Noisy" Hiker

For a long time, scientists knew that this "Expected Improvement" strategy worked well in practice but could not mathematically prove why it worked, especially when the altimeter measurements were noisy.

The main obstacle was the "incumbent" – the currently best location the hiker remembers.

In a perfect world (without noise), the hiker simply remembers the highest peak found so far. This number only increases, making tracking easy.
In the noisy world, the "best" location might just be a lucky error in the measurement. If the hiker uses this erroneous number as a reference, the mathematics becomes messy and collapses. Previous attempts to fix this required the hiker to know secret, hidden numbers about the mountain range (such as exactly how smooth the terrain is or how shaky the altimeter is). In the real world, however, one usually does not know these secrets.

The Solution: A New Way to Walk

The authors of this paper, Hung Tran-The and his team, proposed a new method to solve this problem of the "noisy hiker."

1. The Standard Solution (GP-EI):
They proved that one can use a standard, simple reference value (the best predicted average height from the map, rather than the noisy raw measurement) and still guarantee that the hiker will eventually find the peak.

The Result: They showed mathematically that this method converges (finds the peak) and provided a "regret bound." In hiker terminology, "regret" is the total height one missed by not standing on the true peak at every step. They proved that the regret of their hiker grows slowly enough that the hiker is efficient.
The Bonus: Unlike previous methods, their hiker does not need to know the secret "smoothness" of the mountain range or the "shakiness" of the altimeter. They simply start hiking.

2. The Super-Fast Solution (Improved-GP-EI):
They realized that for very complex mountain ranges (high dimensions), the first method could still take a long time because the hiker might check the same areas too often.
Therefore, they created Improved-GP-EI.

The Analogy: Imagine the hiker divides the mountain range into a grid of ever-smaller boxes. Instead of checking the entire mountain range at once, they focus on one box, map it, and if it looks promising, they divide that box into smaller boxes to look more closely. If a box looks boring, they ignore it.
The Result: This "divide and conquer" strategy makes the hiker much faster. They proved that this new method finds the peak even faster than the first one, and it still requires no secret mountain parameters.

The Proof: Why Trust the Hiker?

The paper is mathematically heavy, but the core logic is as follows:

They decomposed the hiker's errors (regret) into two parts: the error in the map's prediction and the error in the noisy measurement.
They used a clever trick that incorporates the "variance" (how uncertain the map is). They showed that the uncertainty in the map, as the hiker explores, naturally decreases in a predictable way.
By proving that the sum of these decreasing uncertainties remains under control, they proved that the hiker will not wander aimlessly forever.

The Test Drive

To ensure their theory was not just a beautiful mathematical trick, they tested it on computer simulations:

Synthetic Mountains: They created fake, complex mathematical landscapes (such as the Hartmann and Ackley functions) and let their algorithm search for the peak.
The Competition: They compared their "Improved-GP-EI" hiker with other famous hikers (such as GP-UCB and Standard-GP-EI).
The Result: Their Improved-GP-EI hiker found the peaks faster and more reliably than the others, especially when the "secret parameters" (such as the exact noise level) were unknown.

Summary

In short, this paper takes a popular but mathematically shaky strategy (Expected Improvement), repairs its theoretical cracks, and builds a faster, more robust version that does not force the user to know hidden details about the problem. It proves that even with noisy data, an intelligent, greedy strategy can efficiently find the best solution without needing a crystal ball.

1. Problem Statement

The paper addresses the problem of Gaussian-Process-(GP-)Bandit Optimization, specifically in a non-Bayesian setting (where the function is fixed but unknown and lies in a Reproducing Kernel Hilbert Space, or RKHS). The goal is to sequentially select points $x_t$ to maximize an expensive black-box function $f(x)$ based on noisy observations $y_t = f(x_t) + \epsilon_t$ .

Performance is measured by cumulative regret $R_T = \sum_{t=1}^T (f(x^*) - f(x_t^+))$ , where $x^*$ is the global optimum and $x_t^+$ is the best point reported by the algorithm up to iteration $t$ .

The Core Challenge:
While the Expected Improvement (EI) algorithm is a popular heuristic for Bayesian optimization due to its simplicity, its theoretical convergence properties in the noisy setting remained an open question.

In noise-free settings, the "incumbent" (the current best value) is simply the maximum observed value, which is monotonic.
In noisy settings, the incumbent is typically defined as the maximum of the GP predictive mean ( $\mu_t^+$ ) or the maximum of the noisy observation ( $y_t^+$ ).
Unlike the noise-free case, the GP predictive mean $\mu_t^+$ is not monotonic over time. This lack of monotonicity makes standard theoretical analysis (which relies on bounding regret via the monotonicity of the incumbent) extremely difficult. Previous works required either knowledge of unknown parameters (such as the RKHS norm bound $B$ and the sub-Gaussian property of the noise $R$ ) or used non-standard incumbents that required expensive global optimization steps.

2. Methodology

The authors propose two main algorithms to close these theoretical and practical gaps.

A. GP-EI with Standard Incumbent

The first contribution is a rigorous analysis of a standard GP-EI algorithm, where the incumbent is defined as the maximum of the GP predictive mean over the observed points:
$\xi_t = \mu_t^+ = \max_{x_i \in D_{t-1}} \mu_{t-1}(x_i)$

Acquisition Function: The algorithm selects the next point $x_t$ by maximizing the Expected Improvement:
$\alpha_{EI}(x) = \mathbb{E}[\max\{0, f(x) - \mu_t^+\} | D_t]$
This is computed in closed form using the function $\rho(u, v) = u\Phi(u/v) + v\phi(u/v)$ .
Time-Varying Scaling Factor: To handle noise and ensure convergence without knowledge of the RKHS norm ( $B$ ) or the noise parameter ( $R$ ), the algorithm uses a time-varying scaling parameter $\omega_t = \sqrt{\gamma_{t-1} + 1 + \ln(1/\delta)}$ , where $\gamma_t$ is the maximum information gain.
Key Theoretical Innovation: The authors overcome the non-monotonicity of $\mu_t^+$ by exploiting the monotonicity of the variance function ( $\sigma_t(x)$ decreases as $t$ increases). They decompose the instantaneous regret and bound the sum of "improvement terms" using this variance property, a technique not previously applied in this manner to GP bandit optimization.

B. Improved-GP-EI

To address limitations of the standard GP-EI (specifically, that the regret bound for Matérn kernels is not always sublinear and the scaling parameter $\omega_t$ grows too quickly, causing unnecessary exploration), the authors propose Improved-GP-EI.

Inspiration: Adapted from the $\pi$ -GP-UCB algorithm (Janz et al., 2020).
Partitioning Strategy: The search space is divided into hypercubes (a covering). On each hypercube, an independent GP is fitted using only local data.
Global Scaling: Instead of a time-varying $\omega_t$ , a fixed global scaling $\omega_T = \sqrt{\ln(T)\ln\ln(T)}$ is used, which grows only polylogarithmically.
Mechanism: In each step, the algorithm selects a point by maximizing EI within the specific hypercube containing the current best candidate, thereby effectively localizing the optimization.

3. Key Contributions

First Theoretical Guarantees for Standard GP-EI:
The paper provides the first proof that GP-EI with a standard incumbent ( $\xi = \mu_t^+$ ) converges in the noisy setting.
- Regret Bound: The cumulative regret is bounded by $O(\gamma_T \sqrt{T})$ .
- Parameter Independence: Crucially, the algorithm requires no knowledge of the RKHS norm bound ( $B$ ) or the sub-Gaussian parameter of the noise ( $R$ ), which are typically unknown in real-world applications.
Improved-GP-EI Algorithm:
A new variant that achieves faster convergence rates for Matérn kernels.
- Regret Bound: For Matérn kernels with smoothness $\nu > 1$ , the regret is $\tilde{O}(T^{\frac{d(2d+3)+2\nu}{d(2d+4)+4\nu}})$ . This is sublinear for all $\nu > 1$ and $d \geq 1$ , improving upon the limitations of standard GP-EI.
- Efficiency: By using a partitioning strategy and a polylogarithmic scaling parameter, it avoids the rapid growth of the variance scale observed in previous methods and reduces unnecessary exploration.
Novel Proof Techniques:
The authors introduce a proof strategy that handles the non-monotonicity of the GP predictive mean incumbent by leveraging the decreasing monotonicity of the GP variance function to bound the sum of improvement terms.

4. Results

Theoretical Results

Standard GP-EI: Achieves a regret of $O(\gamma_T \sqrt{T})$ . For Squared Exponential (SE) kernels, this is sublinear. For Matérn kernels, it is sublinear if $2\nu > d$ .
Improved-GP-EI: Achieves a tighter regret bound that is sublinear for all $\nu > 1$ . It matches the regret rate of $\pi$ -GP-UCB but removes the need to know $B$ and $R$ .

Empirical Results

The authors tested their algorithms on synthetic functions (Hartmann3, Shekel, Hartmann6, Ackley) and functions generated from RKHS spaces.

Performance: Improved-GP-EI consistently outperformed standard GP-EI, Modified-GP-EI, and the state-of-the-art $\pi$ -GP-UCB.
Robustness: Unlike $\pi$ -GP-UCB, which requires heuristic tuning of unknown parameters ( $B$ and $R$ ) and performs poorly when these are guessed incorrectly, Improved-GP-EI is parameter-free regarding these bounds and demonstrated superior stability.
Efficiency: The partitioning strategy in Improved-GP-EI reduced computational complexity. For example, the average runtime per iteration on Hartmann3 was 1.21 minutes for Improved-GP-EI compared to 1.37 minutes for standard GP-EI, with the gap widening at larger horizons due to reduced kernel matrix inversion sizes (inverting smaller local matrices instead of a global matrix).

5. Significance

This paper solves a long-standing open problem in the theoretical analysis of Expected Improvement for Gaussian Process bandits.

Bridging Theory and Practice: It proves that the most popular heuristic (EI) is theoretically grounded in noisy environments without requiring "oracle-like" knowledge of function smoothness or noise levels.
Practical Applicability: By eliminating dependence on unknown parameters ( $B$ and $R$ ), the proposed algorithms are immediately applicable to real-world scenarios (e.g., robotics, hyperparameter tuning) where these values are difficult or impossible to estimate a priori.
Algorithmic Advancement: Improved-GP-EI offers a robust, parameter-free alternative to UCB-based methods and demonstrates that EI-based strategies can achieve optimal regret rates while being more robust to kernel choice and parameter uncertainty.