🤖 machine learning

Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity

This paper introduces KENDO, a unified framework that enhances Bayesian optimization and active learning by replacing computationally expensive hyperparameter sampling with a kernel ensemble and disagreement-aware acquisition strategies, achieving superior performance and significant speedups over state-of-the-art methods.

Original authors: Heng Zhang, Haotian Xiang, Qin Lu, Konstantinos D. Polyzos, Tara Javidi

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Heng Zhang, Haotian Xiang, Qin Lu, Konstantinos D. Polyzos, Tara Javidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, many problems are like trying to find the highest peak in a vast, foggy mountain range without a map. You can only see the ground directly beneath your feet, and every step you take to explore costs time and energy. This is the challenge of optimizing expensive, mysterious functions, a task that powers everything from designing new drugs to tuning the settings of complex computer models. To navigate this fog, scientists use a strategy called Bayesian optimization. They build a probabilistic map, a kind of educated guess about the terrain, which helps them decide where to look next. This map relies on a mathematical tool called a kernel, which acts like a lens, determining how the system understands the relationship between two points in space. If the lens is too blurry or the wrong shape, the map becomes misleading, and the search for the best solution fails.

For years, researchers have struggled with a difficult choice. They could either pick a single lens and hope it fits the terrain, risking a poor guess, or they could try to account for every possible variation of that lens by running thousands of heavy, slow computer simulations. The first method is fast but often wrong; the second is accurate but so slow it becomes impractical. A team of researchers at the University of Georgia and the University of California San Diego has now proposed a new way forward. They call their approach KENDO, a system that stops trying to guess the perfect single lens and instead builds a team of different lenses, each with its own perspective. By letting these different views disagree and learning from that disagreement, the system can navigate the fog efficiently without the heavy computational cost of the old methods.

The core of this new approach is a shift in how the computer handles uncertainty. Traditionally, when a system wasn't sure about the shape of the terrain, it would run a massive simulation to sample thousands of possibilities, a process that takes a long time. The new method replaces this with a collection of distinct models, each using a different type of kernel, or lens. Instead of asking the computer to imagine thousands of slight variations of one lens, it simply asks it to compare a few different lenses entirely. The system then weighs these lenses based on how well they have performed so far. If one lens consistently predicts the terrain better than the others, the system gives it more influence. This creates a dynamic team where the best models lead the way, and the system learns which lens to trust as it gathers more data.

What makes this approach particularly clever is how it uses the disagreement between these models to guide the search. In the old methods, the system would look for places where it was most uncertain. The new system looks for places where the different lenses in the team strongly disagree with each other. If one lens thinks the ground is high and another thinks it is low, that spot is a prime candidate for investigation. By focusing on these points of conflict, the system learns not just where the best solution might be, but also which lens is the most reliable for the specific problem at hand. This dual learning process allows the system to correct its own mistakes as it goes, refining its map in real-time without needing to restart or run expensive simulations.

The researchers tested this idea on a wide variety of challenges, ranging from simple mathematical puzzles to complex, real-world engineering problems. In the realm of single-objective optimization, where the goal is to find the single best solution, their new system matched or beat the performance of the most advanced methods available today. More importantly, it did so while running up to five times faster. In the field of active learning, where the goal is to learn a function with as few data points as possible, the new method was even more dramatic, achieving results that were up to twenty-seven times faster than the previous standard. These speedups are significant because they mean that problems that once took hours or days to solve can now be tackled in minutes, opening the door to more complex and frequent applications.

The team also extended this method to handle situations where there is no single "best" answer, but rather a set of competing goals, such as trying to make a car both safer and more fuel-efficient. In these multi-objective scenarios, the system uses a technique called random scalarization, which temporarily combines the different goals into a single score to guide the search. This allows the system to explore the full range of trade-offs without getting bogged down in the complexity of solving multiple goals at once. The results showed that this approach could effectively map out the best possible compromises, outperforming existing methods on real-world benchmarks like vehicle safety design and antibiotic production.

Despite these successes, the researchers acknowledge that their method is not a magic bullet for every situation. The system relies on approximating the combined views of the different lenses, which might slightly underestimate the uncertainty if the lenses are wildly different from one another. Additionally, the team still needs to manually select the initial set of lenses to use, though they suggest that future work could automate this selection. Nevertheless, the findings represent a substantial step forward in making intelligent search systems both faster and more reliable. By turning the problem of model uncertainty into a strength through diversity and disagreement, the researchers have shown that a team of simple, specialized models can often outperform a single, overly complex one. This approach offers a practical path forward for scientists and engineers who need to find the best solutions in a world of limited time and resources.

Technical Summary: Enhancing Bayesian Optimization and Active Learning Through Kernel Diversity

Problem Statement

Bayesian Optimization (BO) and Bayesian Active Learning (BAL) rely on probabilistic surrogate models, typically Gaussian Processes (GPs), to guide sequential sampling. A critical bottleneck in these paradigms is hyperparameter selection (e.g., kernel choice and lengthscale parameters).

Point Estimation: Traditional methods maximize marginal likelihood to find a single set of hyperparameters. While computationally efficient, this approach is vulnerable to model misspecification and overfitting, particularly in early optimization stages.
Fully Bayesian Treatment: Fully Bayesian approaches integrate over hyperparameter posteriors using Markov Chain Monte Carlo (MCMC) sampling. While this accounts for uncertainty, it incurs prohibitive computational costs, especially when sampling from multimodal or high-dimensional posteriors.
Kernel Misspecification: Beyond hyperparameters, the choice of the kernel family itself is a source of misspecification. No single kernel (e.g., RBF, Matérn) is universally optimal across different problem landscapes. Existing ensemble methods often select a single GP model or fail to leverage the disagreement among models during the acquisition phase.

Methodology: KENDO

The paper proposes KENDO (Kernel ENsemble Disagreement-aware Operator), a unified framework that replaces hyperparameter sampling with an explicit Ensemble Gaussian Process (EGP) and adaptive Bayesian weighting. The core mechanism shifts the source of uncertainty from hyperparameter samples of a single kernel to the structural disagreement among a diverse set of kernels.

1. Ensemble Surrogate Model
Instead of sampling hyperparameters for a single kernel, KENDO maintains a dictionary of $M$ kernels $\mathcal{K} = \{\kappa_1, \dots, \kappa_M\}$ .

Each kernel $m$ has its own GP with hyperparameters $\hat{\theta}_m$ fitted via marginal likelihood maximization.
The ensemble forms a Gaussian mixture prior. The posterior is approximated via moment matching to yield a single Gaussian summary with mean $\bar{\mu}_t(x)$ and variance $\bar{\sigma}^2_t(x)$ .
The variance decomposition explicitly separates within-GP uncertainty (intrinsic noise) and between-GP disagreement (model uncertainty due to kernel choice).
Weights $w^m_t$ are updated adaptively via Bayesian Model Averaging based on the marginal likelihood of each kernel.

2. Disagreement-Aware Acquisition Functions
KENDO derives acquisition functions that measure the disagreement between the marginal ensemble posterior and conditioned posteriors.

KENDO-BO (Optimization):
- Adapts the SCoreBO concept of conditioning on sampled optimizers.
- For each kernel $m$ and sample $n$ , a function realization is drawn, its optimizer $x^*_{m,n}$ is found, and the GP is conditioned on this "phantom" optimum.
- The acquisition function $\alpha(x)$ aggregates the Hellinger distance between the marginal ensemble and each kernel's optimizer-conditioned posterior.
- Goal: Query points where knowing the location of the optimum would cause the ensemble's prediction to revise significantly, balancing exploration and model selection.
KENDO-AL (Active Learning):
- Removes the optimizer conditioning, as the goal is global function approximation rather than finding a specific optimum.
- Measures the Hellinger distance between the marginal ensemble and the posterior of each individual kernel component.
- Goal: Query points where the identity of the kernel has the maximal impact on predictions, prioritizing regions of high model uncertainty.

3. Extension to Multi-Objective Optimization (MOBO)
To extend KENDO to Multi-Objective BO (MOBO), the authors employ a random scalarization strategy:

Independent EGPs are maintained for each objective.
At each iteration, random weight vectors $\lambda_s$ are sampled from a symmetric Dirichlet distribution to scalarize the objectives.
A single optimizer is found for the scalarized objective using Random Fourier Features (RFF).
The acquisition function aggregates Hellinger distances across objectives and scalarization samples. This preserves the efficient single-optimizer conditioning structure while ensuring coverage of the Pareto front, avoiding the computational diffusion caused by conditioning on full Pareto sets.

Key Contributions

Unified Framework: Proposes KENDO, which unifies ensemble modeling and self-correcting optimization by replacing MCMC-based hyperparameter sampling with a discrete kernel ensemble.
Efficient Acquisition: Derives a computationally efficient acquisition function based on disagreement between marginal and conditioned ensemble posteriors, eliminating the need for expensive MCMC sampling.
MOBO Extension: Introduces a random scalarization approach for multi-objective optimization that maintains the single-optimizer conditioning structure, avoiding the complexity of multi-objective solvers.
Generalization to Active Learning: Demonstrates that the ensemble mechanism generalizes to Bayesian Active Learning (KENDO-AL), achieving superior predictive calibration over MCMC baselines.

Experimental Results

Extensive numerical tests were conducted on synthetic and real-world benchmarks across single-objective optimization, multi-objective optimization, and active learning.

Single-Objective BO: KENDO-BO achieves competitive or superior optimization performance (measured by simple regret) compared to state-of-the-art methods (including SCoreBO, PES, and JES) across nine benchmarks. It reduces computational overhead by up to 5× compared to MCMC-based methods.
Multi-Objective BO: KENDO-MO achieves the lowest log10 hypervolume difference on most benchmarks. Crucially, it outperforms a variant that conditions on sampled Pareto fronts (KENDO-MO-PF), validating the efficacy of the random scalarization strategy.
Active Learning: KENDO-AL achieves superior predictive calibration (lowest negative log-likelihood) on seven benchmarks compared to BALD, BQBC, and SAL. It achieves a 27× speedup over the MCMC-based SAL baseline.
Ablation Studies: Confirm that adaptive Bayesian weighting outperforms uniform averaging and that the ensemble exhibits "winner-take-all" behavior when a superior kernel exists, while maintaining diversity otherwise.

Significance and Claims

The paper claims that kernel family diversity is a critical, often overlooked source of model uncertainty. By explicitly modeling this diversity through an ensemble and leveraging disagreement for acquisition, KENDO offers a practical alternative to fully Bayesian treatments.

Efficiency vs. Accuracy: The framework demonstrates that one can achieve the robustness of fully Bayesian methods (handling model misspecification) without the prohibitive computational cost of MCMC.
Scalability: The method scales significantly better than MCMC-based approaches, making fully Bayesian-style uncertainty quantification feasible for larger budgets and real-world applications.
Unified Approach: It establishes a common self-correcting mechanism for both optimization (finding optima) and active learning (learning functions), driven by the same principle of measuring disagreement across model structures.

Limitations Noted by Authors:
The current framework relies on moment matching to approximate Gaussian mixture posteriors, which may underestimate uncertainty in cases with highly disparate kernel predictions. Additionally, the kernel dictionary requires manual specification, though the authors suggest automatic construction via kernel composition as a future direction.