🤖 AI

How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

This paper benchmarks various oracle-guidance methods for protein structure prediction, revealing that while no single approach dominates across all scenarios, Optimisation Over Outputs (O3) is most effective at low budgets while FK-steering and DPO excel as budgets increase, thereby providing practitioners with actionable guidance for allocating expensive biological oracle resources.

Original authors: Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics, Noam Ghenassia, Shikha Surana, Henry Moss, Paul Duckworth

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics, Noam Ghenassia, Shikha Surana, Henry Moss, Paul Duckworth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef trying to bake the perfect soufflé. You have a recipe book (a computer model) that tells you how to mix ingredients to create a structure. Usually, the recipe works great. But sometimes, the oven is tricky, and the soufflé collapses, burns, or turns into a weird, inedible blob. In the world of biology, scientists use these "recipe books" to predict the 3D shapes of proteins, the tiny machines that keep our bodies running. Getting these shapes right is crucial for designing new medicines, but the models aren't perfect. They might build a protein that looks okay on paper but falls apart in reality.

To fix this, scientists use a "taste-tester" called an oracle. This is a super-accurate, but incredibly slow and expensive, simulation that checks if a protein shape is stable or if it will bind to a virus. Think of the oracle as a famous, grumpy food critic who takes days to taste a single dish. Because this critic is so expensive to hire, you can't ask them to taste every single batch of cookies you bake. You have a strict "budget" of how many times you can ask them to taste. The big question is: if you only have money for ten tastings, how should you spend them? Should you bake a thousand cookies and ask the critic to pick the best ten? Should you tweak your recipe based on the first few tastes? Or should you try a different strategy entirely? This is the puzzle researchers are trying to solve to make protein design faster and cheaper.

This paper, titled "How to Spend Your Oracle Budget," acts as a practical guide for scientists trying to solve this exact problem. The authors tested four different strategies for using a limited number of expensive "taste-tests" (oracle calls) to improve protein predictions. They compared three established methods against a new approach called "Optimisation Over Outputs" (O3). Their findings suggest that there is no single "magic bullet" that works best in every situation; instead, the best strategy depends entirely on how much money (or computing power) you have in your budget.

The researchers discovered a clear trade-off. If you have a very tight budget—say, you can only ask the critic to taste 20 to 100 samples—the new O3 method is the clear winner. O3 works like a smart explorer who takes a few initial samples, maps out a small, manageable "neighborhood" of possibilities, and then uses a clever algorithm to find the best spot within that neighborhood without needing to check every single option. It's efficient and gets great results when you can't afford to check much.

However, as the budget grows larger (allowing for 200 to 1,000 taste-tests), the other methods start to catch up and eventually overtake O3. Two of these methods, called FK-steering and DPO, are more like "trial and error" learners. They don't map a small neighborhood; instead, they constantly adjust the recipe or the baking process based on feedback from the critic. These methods need a lot of data to learn effectively, so they perform poorly when the budget is small but shine when you can afford to run many more tests.

The paper also rules out the idea that one method is always superior. For instance, a simple strategy called "Best K-of-N" (bake 1,000, pick the top 10) is decent but gets stuck at a mediocre level of quality no matter how much you spend. The authors show that while O3 dominates in low-budget scenarios, FK-steering and DPO are necessary for high-budget scenarios to squeeze out the highest possible quality. They tested these ideas on real protein targets, specifically calmodulin and an enzyme from E. coli, using a scoring system called TM-score to measure how close the predicted shapes were to the real thing. The results indicate that for scientists with limited resources, the smartest move is to use O3, but for those with deep pockets, the older, more data-hungry methods are still the way to go.

Technical Summary: How to Spend Your Oracle Budget for Protein Structure Prediction

Problem Statement

Foundation models for protein structure prediction (e.g., AlphaFold3, Chai-1, Boltz-2) have demonstrated high accuracy but remain unreliable on specific targets, potentially collapsing into single conformations, producing physically implausible geometries, or misassembling complexes. To mitigate this, researchers employ external "oracles" (e.g., molecular dynamics simulations or binding energy calculations) to score and flag unreliable predictions. However, these biological oracles are computationally expensive and often non-differentiable, creating a critical constraint known as the oracle budget.

While existing guidance methods—such as inference-time steering (FK-steering), fine-tuning (DPO), and Best K-of-N sampling—utilize oracle scores to improve outputs, there is no systematic comparison of how these methods perform under varying budget constraints. The central problem addressed is determining which guidance strategy is most effective given a limited number of oracle evaluations ( $N$ ).

Methodology

The authors benchmark four guidance strategies applied to the protein structure prediction model Boltz-2 across two protein targets: calmodulin (1CLL) and E. coli aspartate transcarbamoylase (9EEH). The methods are evaluated under a fixed budget of $N$ oracle calls to return a batch of $K$ candidate structures.

1. Bayesian Optimisation via O3 (Optimisation Over Outputs)

This approach, adapted from Willis et al. (2025), treats the guidance problem as a low-dimensional black-box optimization task.

Subspace Construction: The method first samples $M$ initial structures and selects the top $d$ seeds based on oracle scores. It constructs a low-dimensional subspace $\mathcal{U}$ within the model's latent space using these seeds.
Mapping: A surrogate chart $\phi$ maps points in $\mathcal{U}$ to the latent space via a Knothe–Rosenblatt transform (to generate weights on a simplex) and a LOL projection (to map weights back to the latent space).
Optimization: A Gaussian Process (GP) surrogate model is fitted within $\mathcal{U}$ using the initial seeds and random samples. Bayesian Optimization (specifically Log Expected Improvement) is then used to query the remaining $N-M$ budget to find optimal points in this subspace.

2. FK-Steering (Feynman-Kac)

An inference-time method that alters the sampling trajectory without modifying model weights.

Mechanism: It generates $M$ interacting particles (stochastic processes) through the generative process.
Resampling: At intermediate denoising steps, particles are resampled proportionally to weights derived from the oracle reward signal. The reward difference between steps is used to favor particles with increasing rewards.
Budget Allocation: The total budget $N$ is distributed across $M$ particles, determining the number of resampling steps.

3. Direct Preference Optimisation (DPO)

A fine-tuning approach that updates model parameters based on oracle-scored pairs.

Mechanism: The model is fine-tuned to increase the likelihood of preferred structures ( $x_w$ ) over rejected ones ( $x_l$ ) using a KL-regularized objective.
Variants: The paper evaluates both offline DPO (training on a fixed batch of $N$ samples) and online DPO (interleaving sampling and training, resetting the reference model at each epoch).
Approximation: Since diffusion log-likelihoods are expensive to compute, the method approximates the log-likelihood ratio using differences in training losses (LDSM).

4. Best K-of-N Sampling

A baseline approach where $N$ samples are drawn independently, and the top $K$ structures with the highest oracle scores are returned.

Experimental Setup

Oracle: The primary oracle is the TM-score (Template Modeling score) against a ground-truth PDB structure (1CLL), chosen for computational efficiency. A secondary experiment uses the MolProbity oracle on 9EEH.
Budgets: Evaluations span six configurations of $(N, K)$ : $(20, 2), (50, 5), (100, 10), (200, 20), (500, 50), (1000, 100)$ .
Metrics: Performance is measured by Mean-of-K (average quality of the batch) and Max-of-K (quality of the single best structure).

Key Results

Performance Across Budgets

The study reveals that no single method dominates across all budget sizes; effectiveness is highly dependent on the available oracle budget:

Low-to-Mid Budgets ( $N \le 1000$ ): O3 significantly outperforms all other methods. At $N=20$ , FK-steering and DPO perform poorly (near baseline levels), while O3 achieves a mean-of-K TM-score of $\sim0.81$ at $N=1000$ , plateauing well above the Best K-of-N baseline ( $\sim0.60$ ).
High Budgets: FK-steering and DPO show steady improvement as $N$ increases. FK-steering reaches $\sim0.73$ at $N=1000$ , surpassing DPO ( $\sim0.71$ ) but remaining below O3's performance in the tested range.
Best K-of-N: This baseline remains relatively flat across all budgets, serving as a lower bound for more sophisticated guidance methods at low budgets.

Method-Specific Findings

O3: The performance gain stems from the combination of the low-dimensional subspace construction and the Bayesian optimizer. The optimal subspace dimension ( $d$ ) varies with the budget (e.g., $d=6$ is optimal for $N=20$ , while $d=10$ is better for $N=200$ ), indicating a trade-off between subspace richness and optimization difficulty.
FK-Steering: Increasing the reward scaling coefficient ( $\lambda$ ) improves performance but may reduce output diversity. Increasing the number of resampling steps (at the cost of particle count) generally yields better results at higher budgets.
DPO: Online DPO (resetting the reference model) offers a trade-off: it softens KL regularization but weakens the influence of the pre-trained model. Within the tested settings, DPO did not outperform FK-steering or O3.

Significance and Claims

The paper claims to provide the first practical reference for selecting guidance strategies based on oracle budget constraints in protein structure prediction.

Practical Guidance: The authors distill their findings into actionable recommendations: practitioners with scarce oracle budgets (common in biological wet-lab settings) should utilize O3 (Optimisation Over Outputs), as it leverages a small number of high-quality seeds to define an efficient search space. Conversely, if the budget is larger, FK-steering becomes a competitive alternative.
Methodological Contribution: The work extends the O3 framework, originally proposed for general generative models, to the specific domain of protein structure prediction, demonstrating its efficacy in a high-dimensional, non-differentiable oracle setting.
Limitations: The study acknowledges that while TM-score was used for extensive benchmarking, the trade-offs identified apply to expensive, non-differentiable biological oracles. The results are specific to the Boltz-2 model and the tested protein targets, though the authors suggest the budget-dependent trade-off is a general phenomenon.

In summary, the paper argues that the choice of guidance method is not universal but is critically dependent on the cost of the oracle, with O3 offering a distinct advantage in resource-constrained scenarios.