NUBO: A Transparent Python Package for Bayesian Optimization

NUBO is a transparent, open-source Python package for Bayesian optimization that prioritizes user experience and code clarity through a modular design, enabling researchers from diverse disciplines to efficiently optimize expensive black-box functions without requiring expert programming knowledge.

The Gist

Imagine you are a chef trying to create the perfect recipe for a new dish. However, there's a catch: every time you try a new combination of ingredients, it costs you $1,000 in rare spices and takes three days to cook. You can't just taste-test every possible combination in the world; you simply don't have the money or time.

This is the problem NUBO solves.

NUBO (Newcastle University Bayesian Optimisation) is a free, open-source computer program designed to help scientists and researchers find the "perfect recipe" (the best solution) for expensive, complex problems with as few tries as possible.

Here is a simple breakdown of how it works, using everyday analogies:

1. The Problem: The "Black Box" Mystery

In science and engineering, researchers often deal with "Black Box" functions.

• The Analogy: Imagine a sealed, magical oven. You put ingredients in (inputs), and a cake comes out (outputs). You have no idea how the oven works inside. You can't see the gears or the temperature settings. You only know that if you put in this mix, you get a dry cake, and if you put in that mix, you get a burnt one.
• The Challenge: Testing every possible mix is too expensive. You need a smart strategy to guess the best mix without wasting money.

2. The Solution: The "Smart Map" (Surrogate Model)

NUBO uses a technique called Bayesian Optimisation. Think of this as building a smart, living map of the oven's behavior.

• The Map (Gaussian Process): Instead of guessing randomly, NUBO builds a 3D map based on the few tests you've already done. It doesn't just connect the dots; it draws a smooth, wavy surface that predicts what the oven would do in areas you haven't tested yet.
• The Confidence Interval: Crucially, this map also shows uncertainty. In areas where you haven't tested much, the map looks "fuzzy" or "wobbly" (high uncertainty). In areas you've tested a lot, the map is sharp and clear.

3. The Strategy: The "Explorer vs. The Miner" (Acquisition Functions)

How does NUBO decide where to test next? It balances two instincts, like a treasure hunter:

• The Miner (Exploitation): "I found a spot with a good cake last time. Let's dig right around there to see if we can find an even better one." (Focusing on what we know works).
• The Explorer (Exploration): "I haven't checked the back corner of the oven at all. It's fuzzy on the map. Maybe there's a gold mine of flavor back there!" (Checking areas of high uncertainty).

NUBO uses a mathematical formula (called an Acquisition Function) to decide exactly how much to "mine" vs. how much to "explore" at every step. It's like a GPS that tells you: "Go here, because it's either very likely to be good, OR it's a mystery we need to solve."

4. Why NUBO is Special: The "Transparent Toolkit"

There are other tools for this job (like BoTorch or SMAC3), but they are often like giant, complex Swiss Army knives with thousands of hidden blades. They are powerful, but if you aren't an expert engineer, you might get lost trying to figure out which blade to use, or you might accidentally break the handle.

NUBO is different:

• It's a Transparent Toolbox: The code is short, clean, and easy to read. It's like a clear plastic toolbox where you can see exactly how every screw and gear works. You aren't overwhelmed by options; it only gives you the essential tools you need.
• It's Flexible: You can build your own optimisation loop. It's like giving you the Lego bricks instead of a pre-built castle. You can build a single-person search, or a team of robots searching in parallel.
• It Handles "Asynchronous" Work: Imagine you have a team of 4 chefs. While Chef A is waiting for their cake to bake (which takes 3 hours), the other chefs don't have to stand around doing nothing. NUBO lets the other chefs start testing new recipes immediately, even while Chef A is still waiting. This saves huge amounts of time.

5. The Results: Fast and Accurate

The paper tested NUBO against other tools using mathematical "test cakes" (functions like the Levy and Hartmann functions).

• The Verdict: NUBO found the best solutions just as well as the massive, complex tools, but with a much simpler, easier-to-understand codebase.
• The Trade-off: NUBO takes a tiny bit more computer time to think about the next step (a few seconds), but since the actual "baking" (running the expensive experiment) takes hours or days, that few seconds of thinking time is totally worth it.

Summary

NUBO is a user-friendly, transparent guide for researchers. It helps them navigate the "foggy" landscape of expensive experiments, using a smart map to balance guessing new things with refining what works, ensuring they find the best solution without burning through their budget. It turns a complex statistical nightmare into a manageable, step-by-step process.

Technical Summary

1. Problem Statement

The optimization of expensive-to-evaluate black-box functions is a pervasive challenge across engineering, natural sciences, and computer science. These functions are characterized by:

• Unknown or non-analytical mathematical expressions.
• High evaluation costs (e.g., physical experiments, complex simulations).
• Lack of derivative information.

Traditional optimization algorithms (e.g., gradient-based methods like Adam or L-BFGS-B, or evolutionary algorithms) often require either derivative information or a prohibitively large number of function evaluations, making them unsuitable for expensive black-box problems. While Bayesian Optimization (BO) offers a sample-efficient alternative using surrogate models, existing Python implementations often suffer from:

• Lack of transparency: Complex, monolithic codebases (e.g., BoTorch with ~38k lines of code) that are difficult for non-experts to understand or modify.
• Limited flexibility: Many packages only support sequential single-point optimization, lacking support for parallel or asynchronous evaluation.
• Overwhelming complexity: An abundance of options that can confuse users without adding practical value.

2. Methodology

The paper introduces NUBO (Newcastle University Bayesian Optimisation), a lightweight, transparent Python package designed to make BO accessible to researchers from diverse disciplines.

Core Architecture

• Foundation: Built on the PyTorch ecosystem for tensor operations and GPU acceleration, utilizing GPyTorch for Gaussian Process (GP) implementation, and SciPy for deterministic optimizers (L-BFGS-B, SLSQP).
• Design Philosophy: Prioritizes transparency (clean code, ~1,300 lines) and modularity. Users can construct optimization loops using provided building blocks rather than relying on "black-box" high-level wrappers.

Key Algorithmic Components

1. Surrogate Model:

   • Uses Gaussian Processes (GP) with a Matérn 5/2 ARD (Automatic Relevance Determination) kernel and a constant mean function.
   • Supports noise estimation via GaussianLikelihood or explicit noise specification via FixedNoiseGaussianLikelihood.
   • Hyperparameters are estimated by maximizing the log-marginal likelihood.

2. Acquisition Functions:

   • Analytical: Supports Expected Improvement (EI) and Upper Confidence Bound (UCB) for sequential single-point optimization. These are optimized using deterministic solvers (L-BFGS-B for unconstrained, SLSQP for constrained).
   • Monte Carlo (MC): To support parallel and asynchronous optimization, NUBO approximates acquisition functions using Monte Carlo sampling. This involves drawing samples from the posterior distribution and averaging the acquisition values.
   • Optimization Strategies:
     • Sequential Single-Point: Standard BO loop.
     • Parallel Multi-Point: Uses multi_joint (joint optimization) or multi_sequential (greedy sequential selection) to generate batches.
     • Asynchronous: Handles pending evaluations by treating them as fixed points in the joint distribution.

3. Optimization Capabilities:

   • Bounded Optimization: Standard hyper-rectangle constraints.
   • Constrained Optimization: Supports equality and inequality constraints via the SLSQP optimizer.
   • Mixed Optimization: Handles discrete and continuous parameters by iterating over discrete combinations and optimizing continuous variables for each.

3. Key Contributions

• Transparency and Accessibility: NUBO is significantly smaller (1,322 lines of code) than major competitors like BoTorch (38,419 lines), making the underlying algorithms and logic easily inspectable and modifiable for non-statisticians.
• Comprehensive Optimization Modes: Unlike many lightweight packages that only support sequential optimization, NUBO natively supports sequential, parallel, and asynchronous modes, as well as constrained and mixed (discrete/continuous) spaces.
• Modular Design: The package allows users to swap components (e.g., changing the acquisition function or optimizer) without rewriting the entire loop, facilitating custom research implementations.
• Rigorous Validation: The package includes a suite of test functions and utilities (e.g., Latin Hypercube Sampling generation) to facilitate reproducible research.

4. Results

The authors conducted a comparative benchmark against BoTorch, bayes_opt, SMAC3, and pyGPGO using the 2D Levy and 6D Hartmann functions.

• Performance:
  • NUBO demonstrated competitive performance, converging to the global optimum in all test cases.
  • In the 6D Hartmann function, while NUBO required slightly more evaluations to reach the optimum compared to the more complex BoTorch, it ultimately achieved results with lower variance and closer proximity to the true optimum than other lightweight packages (pyGPGO, bayes_opt).
  • NUBO outperformed packages of similar complexity (pyGPGO, bayes_opt) and compared favorably against complex packages (BoTorch, SMAC3).
• Efficiency:
  • Computation Time: NUBO's iteration time was slightly higher (e.g., ~2.20s vs. ~0.19s for BoTorch in parallel 6D Hartmann) due to its lightweight implementation and lack of heavy pre-optimization. However, the authors argue this overhead is negligible compared to the time required to evaluate expensive physical experiments or simulators.
• Case Study: A detailed workflow optimizing a 6D Hartmann function with mixed discrete/continuous parameters and parallel batch evaluation (batch size 4) demonstrated NUBO's ability to efficiently explore the search space and approximate the global maximum (3.21 vs. true 3.32) significantly better than random sampling or Latin Hypercube Sampling.

5. Significance

NUBO addresses a critical gap in the scientific computing landscape by providing a transparent, user-friendly, yet powerful tool for Bayesian Optimization.

• Democratization of BO: By simplifying the codebase and documentation, it lowers the barrier to entry for researchers in fields like chemistry, physics, and engineering who need to optimize expensive experiments but lack deep expertise in statistics or machine learning.
• Flexibility for Research: The modular design empowers users to tailor optimization loops to unique, non-standard research problems without being constrained by the rigid structures of larger, more complex libraries.
• Future Outlook: The authors plan to extend NUBO to support multi-fidelity, multi-objective, and high-dimensional optimization problems, further expanding its utility in complex scientific domains.

In summary, NUBO proves that a simple, transparent implementation does not come at the cost of performance, offering a robust alternative for the optimization of expensive black-box functions where understanding the underlying mechanics is as important as the optimization result itself.