🤖 machine learning

Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets?

This paper demonstrates that Bayesian optimization applied within a low-dimensional random linear embedding of weight space enables efficient, gradient-free post-training of LLMs, achieving performance comparable to or better than existing methods like RandOpt while requiring five times fewer candidate evaluations.

Original authors: Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra, Jeremias Knoblauch, Jakob Zeitler

Published 2026-08-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nigel Bastian Cendra, Abdelhamid Ezzerg, Fernando Julio Cendra, Jeremias Knoblauch, Jakob Zeitler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot brain that has already read almost everything on the internet. This brain is a "Large Language Model." Usually, if you want to teach it a new trick—like solving math puzzles or playing a specific game—you have to use a method called "gradient-based optimization." Think of this like trying to find the top of a mountain in a thick fog by feeling the slope under your feet and taking tiny, careful steps. It works, but it's slow, expensive, and requires a lot of computing power.

Recently, scientists discovered a different way to tweak these brains without that heavy climbing gear. They found that if you just shake the brain's settings randomly a little bit, some of those random shakes actually make it smarter at a specific task. It's like shaking a box of LEGOs; most of the time, you just get a mess, but occasionally, you accidentally snap together a perfect little spaceship. This idea is called "Neural Thickets." The problem is, finding that perfect spaceship by just shaking the box randomly is like looking for a needle in a haystack while blindfolded. You have to try thousands of shakes, test each one, and hope you get lucky. This paper asks a simple question: Can we be smarter about our shaking? Instead of blindly guessing, can we use a "smart guesser" to figure out which direction to shake next, so we find the best result with far fewer tries?

The Smart Shaker: Finding the Needle Without the Blindfold

The researchers behind this study, working with models called Qwen2.5, decided to test a method called Bayesian Optimization. To understand what they did, let's imagine the robot brain's settings are a giant, multi-dimensional map. The "random shaking" method (called RandOpt) is like throwing darts at this map from a distance. You throw thousands of darts, see which ones hit the "good" spots, and keep the best ones. It works, but it's wasteful.

The authors proposed a different strategy: Bayesian Optimization (BO). Think of this as having a magical, invisible guide who has seen a few of your dart throws. After you throw a few darts, this guide builds a rough map of where the good spots might be. Instead of throwing darts randomly, the guide tells you exactly where to throw next to learn the most. It's like playing a game of "Hot and Cold," but the guide is so good at guessing that you find the treasure in just a few moves instead of hundreds.

To make this work, the team didn't look at the whole giant map. They realized that the "good" changes to the robot's brain probably happen in a very small, narrow hallway within that massive space. So, they built a tiny, 64-dimensional "slice" of the map to search. They then used their smart guide to explore this slice.

What They Found: Smarter, Not Harder

The results were quite exciting, but with some important caveats. When they tested this on reasoning tasks like Countdown (a number game), GSM8k (math word problems), and MATH500 (harder math problems), the "Smart Shaker" (BO) did something remarkable.

Using only 200 attempts (evaluations), the Bayesian Optimization method matched or even beat the performance of the "Random Shaker" (RandOpt) that used 1,000 attempts. That's a five-fold reduction in effort!

On the Countdown task with a 1.5-billion-parameter model, the Smart Shaker got a score of 15.05, while the Random Shaker with five times the effort only got 14.60.
On GSM8k, the Smart Shaker scored 67.78, beating the Random Shaker's 66.13.

This suggests that by using a smart guide to navigate the search space, you don't need to throw as many darts to find a strong, single expert model. You can get a "super-robot" that is ready to go alone, without needing to vote on answers from fifty different versions.

The Catch: The "Fake Treasure" Problem

However, the paper also warns us that this magic guide isn't perfect. There is a tricky part called the "winner's curse."

Imagine you are searching for the best apple in an orchard. If you pick the apple that looks the shiniest in the sun, it might just be lucky lighting, not actually the sweetest apple. The researchers found that when they pushed the search too hard, the "Smart Shaker" would find models that looked amazing on the test questions they were trained on (the "selection set"), but actually performed worse on new, unseen questions.

For example, on the MATH500 task, the Smart Shaker found models that scored incredibly high on the practice questions (over 63%), but when tested on real problems, they actually scored lower than the base model! The guide had gotten too confident in the "shiny apples" that were just lucky. The paper suggests that once you reach a certain point, the search is just memorizing the practice test rather than learning the actual skill.

The Bottom Line

This paper shows that Bayesian Optimization is a powerful tool for finding strong, single experts in neural networks without needing expensive back-and-forth calculations. It proves that you can find better solutions with five times less effort than random guessing, provided you stop before the search starts to "overfit" (memorize the practice test).

While the method doesn't work miracles on every single task (especially when the "good" solutions are very rare or the practice test is misleading), it suggests a promising future. Instead of brute-forcing our way through millions of random guesses, we can use intelligent, guided searches to find the best versions of our AI brains, saving time and energy while still getting top-tier performance. The authors suggest that with larger budgets and better ways to handle the "lucky lighting" problem, this approach could become a standard way to fine-tune the next generation of AI.

Technical Summary: Bayesian Optimization for Finding Strong Single Experts in Neural Thickets

Problem Statement

Gradient-free post-training has emerged as a cost-effective alternative to gradient-based methods (like PPO and GRPO) for Large Language Models (LLMs). Existing approaches, such as RandOpt (Neural Thickets), rely on sampling random Gaussian perturbations of pretrained weights, scoring them on a selection set, and ensembling the top $K$ models via majority voting. While effective, this approach faces two primary inefficiencies:

Search Cost: Evaluating a population of $N$ candidates requires $N \times |D_{sel}|$ autoregressive generations. To achieve strong results, RandOpt typically requires large populations (e.g., $N=5,000$ ), making the search computationally expensive.
Deployment Cost: The strongest results rely on ensembling $K$ models (e.g., $K=50$ ). Deploying a single query requires generating $K$ distinct outputs, and maintaining $K$ weight configurations is resource-intensive. Fallback to a single model ( $K=1$ ) results in significant accuracy drops (e.g., 11.2 points on MATH-500).

The central research question is: Given a fixed and modest evaluation budget, can a structured, gradient-free method identify a single strong expert that outperforms random sampling, thereby reducing both search and deployment costs?

Methodology

The authors propose applying Bayesian Optimization (BO) within a random linear embedding of the weight space. The method is designed to be entirely gradient-free, utilizing a Gaussian Process (GP) surrogate to guide the search.

Low-Dimensional Subspace Search: Instead of sampling isotropically in the full $D$ $D$ -dimensional weight space, the method constructs a random linear embedding $A \in \mathbb{R}^{D \times d}$ $A \in R^{D \times d}$ (where $d=64$ $d = 64$ ). Candidates are defined by a coefficient vector $\alpha \in \mathbb{R}^d$ $α \in R^{d}$ and a scale $\sigma \in \mathbb{R}_{>0}$ $σ \in R_{> 0}$ .
- The perturbation is formulated as: $\theta' = \theta_0 + \sigma \cdot \sqrt{D} \cdot \frac{A\alpha}{\|A\alpha\|}$ .
- This normalization ensures the perturbation norm matches that of standard RandOpt ( $\sigma\sqrt{D}$ ), allowing for fair comparisons of perturbation scale.
Surrogate-Guided Search:
- A Gaussian Process (GP) is fitted to the observed rewards (accuracy on the selection set $D_{sel}$ ).
- The next candidate $(\alpha, \sigma)$ is proposed by maximizing an acquisition function (e.g., logEI), balancing exploration and exploitation.
- Selection Rule: The final model is selected either by the highest observed reward or the highest GP posterior mean. The latter is intended to mitigate the "winner's curse" by penalizing candidates whose high scores may be due to noise or lucky draws on the selection set.
Evaluation Protocol: The method was tested on Qwen2.5-Instruct models (0.5B, 1.5B, 3B parameters) across three reasoning benchmarks: Countdown, GSM8k, and MATH500. The budget was fixed at 200 candidate evaluations for BO and Random Search, compared against RandOpt with budgets of 200 and 1,000.

Key Results

The experiments demonstrate that BO can achieve competitive or superior performance with significantly fewer evaluations:

Sample Efficiency: With 5x fewer candidate evaluations (200 vs. 1,000), BO matches or exceeds the performance of RandOpt-1000 at $K=1$ $K = 1$ (single expert) in most settings.
- Countdown-0.5B: BO (9.34%) vs. RandOpt-1000 (6.81%).
- GSM8k-1.5B: BO (67.78%) vs. RandOpt-1000 (66.13%).
- MATH500-1.5B: BO (40.43%) vs. RandOpt-1000 (39.80%).
Subspace vs. Surrogate: Within the same 64-dimensional subspace, BO consistently achieves higher selection rewards than random search, confirming that the gains are driven by the surrogate's efficiency rather than the dimensionality reduction itself.
Ensemble Performance ( $K=20$ ): BO matches alternatives at matched budgets (200 evaluations) for $K=20$ , though it trails behind RandOpt-1000, suggesting that ensemble quality benefits from a broader pool of candidates that concentrated search may not fully capture.
Limitations and Overfitting:
- MATH500 Anomaly: On MATH500, BO achieved the highest selection rewards but the lowest test accuracies. The authors attribute this to overfitting: when the "neighbourhood" contains no genuinely better models, the search fits the noise in the selection set ( $D_{sel}$ ) rather than the task.
- Saturation Threshold: Analysis of 200 candidates on GSM8k-1.5B revealed that selection rewards continue to climb even after validation and test accuracies plateau. The top 8 points of selection reward provided no measurable improvement on held-out sets, indicating a saturation point where the selection objective becomes uninformative.

Significance and Claims

The paper claims that surrogate-guided search can substantially reduce the evaluation cost of gradient-free post-training while producing stronger deployable single experts.

Efficiency over Feasibility: The work argues that the challenge is not whether gradient-free methods work, but how to do so efficiently. BO offers a mechanism to extract available gains from the weight space at a fraction of the budget required by random sampling.
Denoising Effect: The use of the GP posterior mean for selection provides a denoised estimate of true fitness, which helps avoid selecting candidates that are merely lucky on the selection set.
Modest Scope: The authors are careful to note that their results are contingent on the specific 64-dimensional slice chosen. They do not claim that this subspace is optimal, only that BO exploits it efficiently. Furthermore, they acknowledge that the method does not create gains that sampling cannot reach; it simply finds them faster. The difficulty lies in locating "good regions" within the subspace, as performance varies significantly across different random seeds and basis choices.

In summary, the paper establishes that Bayesian Optimization, when applied within a low-dimensional random embedding of weight space, is a viable and efficient strategy for identifying high-performing single experts in LLMs, offering a path toward cheaper training and deployment compared to current ensemble-based gradient-free methods.