Technical Summary: Regime-Conditioned Evaluation in Multi-Context Bayesian Optimization

Problem Statement

Current evaluations of transfer Bayesian Optimization (BO) and acquisition functions suffer from a fundamental measurement flaw: they report Average Treatment Effects (ATE) over hidden regime variables rather than Conditional Average Treatment Effects (CATE) specific to the practitioner's context. The paper argues that the performance ranking of acquisition functions (e.g., Greedy vs. UCB) is not an intrinsic property of the algorithm but a function of the evaluation regime, specifically defined by the budget ratio ($B/|A|$), the quality of the prior ($\rho$), the context count ($K$), and the metric type.

An audit of 40 recent papers (2022–2025) reveals that 98% do not vary the budget ratio $B/|A|$ as a controlled axis, and 80% report only a single metric family. Consequently, existing leaderboards are "regime-uncontrolled." The authors demonstrate that identical benchmarks can yield opposite conclusions solely based on the budget: on the GDSC2 drug-response benchmark, Greedy outperforms UCB by 0.050 Hit@1 at $B=50$, while UCB outperforms Greedy by 0.035 Hit@1 at $B=100$. These are not contradictions but different CATEs. The paper posits that without controlling for these regime variables, any reported ranking is a property of the measurement setup, not the algorithm.

Methodology

1. The Portable Regime Score (PRS)

To make the regime observable, the authors define the Portable Regime Score (PRS):
$$PRS = \frac{B}{|A|} (1 - \rho)$$
Where:

• $B$: Per-context budget.
• $|A|$: Size of the action space.
• $\rho$: Spearman rank correlation between prior means and true action values (measurable from a pilot run).

The PRS serves as a one-dimensional order parameter. A high PRS indicates a large budget relative to the action space and/or a weak prior, suggesting that exploration is likely to pay off. A low PRS indicates a strong prior or limited budget, favoring exploitation (Greedy).

2. Theoretical Framework

The paper introduces the No-Free-Leaderboard Proposition (Proposition A.11). It proves that if the CATE of an acquisition choice changes sign across a regime variable (as observed with PRS), then the ATE reported on any benchmark is a free parameter determined by the benchmark mixture. By choosing evaluation conditions appropriately, any ranking (including ties or reversals) can be achieved without fabrication. This formalizes why regime-agnostic leaderboards are unreliable decision tools.

3. REGIMEPLANNER

To demonstrate that the regime view is exploitable, the authors propose REGIMEPLANNER, an adaptive algorithm that:

1. Estimates $\hat{\rho}$ online within each context using cached prior means and observed outcomes.
2. Computes the dynamic PRS ($\hat{PRS}_t$).
3. Switches between Greedy and UCB based on a threshold $\theta = 0.10$.
   • If $\hat{PRS}_t < \theta$: Switch to Greedy (exploitation).
   • If $\hat{PRS}_t \ge \theta$: Switch to UCB (exploration).

This allows the planner to adapt to regime shifts within a single context, a capability fixed policies lack.

4. Empirical Audit and Validation

• Audit: A systematic review of 40 transfer-BO papers (NeurIPS, ICML, ICLR, etc.) quantified the lack of regime control (98% do not sweep $B/|A|$).
• Benchmarks: Experiments were conducted on three primary benchmarks:
  • Buchwald-Hartwig: A chemistry reaction optimization task with shuffled priors to isolate prior quality effects.
  • GDSC2: A drug-response biology benchmark with noisy observations.
  • HPO-B: A community hyperparameter optimization benchmark across 16 search spaces.
• Analysis: The authors analyzed 79 conditions spanning these benchmarks, measuring Hit@1 and Discovery AUC.

Key Results

1. Regime Predictability

The PRS strongly predicts the "exploration advantage" (the performance gap between exploration and exploitation strategies).

• Across 79 conditions, PRS correlates with exploration advantage with a hierarchical model coefficient $\beta = 0.50$ ($p = 1.1 \times 10^{-9}$).
• Within-benchmark correlations are strong: GDSC2 ($r=0.96$), HPO-B ($r=0.83$), and Buchwald ($r=0.75$).
• PRS outperforms its individual components ($B/|A|$ alone or $1-\rho$ alone) as a predictor.

2. Reversal of Rankings

The study confirms that acquisition rankings flip entirely based on the regime:

• Budget Reversal: On GDSC2, Greedy wins at low budgets ($B=50$), while UCB/REIGN win at high budgets ($B=100$).
• Prior Quality Reversal: On Buchwald, with a weak prior, exploration wins; with a strong structured prior, Greedy dominates by a large margin (0.481 vs 0.389 Hit@1), compressing the value of acquisition choice.
• Metric Reversal: On GDSC2, Greedy dominates Discovery AUC at all budgets, while UCB/REIGN dominate Hit@1 only above the regime threshold.

3. Performance of REGIMEPLANNER

• GDSC2: At $B=50$, REGIMEPLANNER achieves 0.676 Hit@1, outperforming all fixed planners and exceeding the matched {Greedy, UCB} per-context oracle by 18%.
• HPO-B: At $B=100$, REGIMEPLANNER wins on all 16 search spaces against Greedy (mean +0.103 Hit@1) and ranks #1 overall, whereas Greedy ranks last.
• Adaptivity: REGIMEPLANNER succeeds by tracking the evolution of $\hat{\rho}$ within a context. On GDSC2, $\hat{\rho}$ rises from ~0.12 to ~0.66 during an episode, allowing the planner to switch from exploration to exploitation mid-stream, a feat impossible for fixed policies.

4. Failure Modes and Equivalence Zones

• Accuracy: PRS predicts the empirical winner in 74.7% of 79 conditions.
• Equivalence Zone: 19% of conditions fall in an equivalence zone ($|advantage| < 0.01$ Hit@1) where differences are practically indistinguishable, yet these "non-differences" often generate publishable rankings.
• Boundary Failures: Predictions degrade near the threshold ($PRS \in [0.05, 0.15)$) and in cases with very low context counts ($K < K_{min}$) where the prior has not yet differentiated between planners.
• Pre-registered Validation: On 40 held-out conditions, the method achieved 67.5% accuracy, below the 90% target, highlighting that the threshold is not universally domain-agnostic without noise estimation.

Significance and Claims

The paper claims three primary contributions:

1. Predictability: The regime is predictable before any comparison runs using the PRS, which orders exploration advantage across diverse domains (chemistry, biology, HPO).
2. Exploitability: The CATE is exploitable online. REGIMEPLANNER demonstrates that an adaptive rule based on observable quantities ($B/|A|$ and $\hat{\rho}$) can outperform fixed policies and even per-context oracles in specific regimes.
3. Diagnostic Transparency: The "No-Free-Leaderboard" proposition formalizes why regime-agnostic rankings are unreliable. The paper argues that omitting $B/|A|$, $\rho$, $K$, and metric type produces "measurement claims, not algorithmic ones."

The authors conclude that the field has been arguing over non-differences (equivalence zones) and that the "best" acquisition function is a property of the regime in which it was evaluated. They propose a new reporting protocol: acquisition comparisons must report the prior condition, budget ratio, metric type, and context count to be interpretable. The work serves as a diagnostic tool (PRS) and an existence proof (REGIMEPLANNER) that regime-conditioned evaluation is necessary for reliable algorithmic assessment.