Technical Summary: Safe Bayesian Optimization with Counterfactual Policies

Problem Definition

The paper addresses the challenge of performing Safe Bayesian Optimization (SafeOpt) in settings where safety constraints depend on counterfactual outcomes. In many sequential decision-making scenarios (e.g., clinical trials), a new intervention $x_t$ is deemed safe only if its outcome $Y(x_t)$ does not fall significantly below the outcome of an established standard-of-care policy, $Y(x_{soc,t})$.

The core difficulty arises because $x_{soc,t}$ is a counterfactual: the standard-of-care policy is not applied at time $t$, so $Y(x_{soc,t})$ is unobserved. Traditional SafeOpt methods rely on observed constraint values to maintain a safe set of actions. When the constraint $q(x_t)$ depends on an unobserved counterfactual, simple point estimates of $Y(x_{soc,t})$ are insufficient, as they can lead to safety constraint violations exceeding the user-specified tolerance rate $\alpha$.

Methodology: SafeOpt-CPC

The authors propose SafeOpt with Counterfactual Policy Constraints (SafeOpt-CPC), a framework that integrates split conformal prediction into the SafeOpt loop to handle unobserved counterfactuals.

1. Conformal Estimation of Counterfactuals

Instead of using a point estimate for the counterfactual outcome, the method constructs a valid uncertainty interval (specifically, a one-sided upper bound $\hat{U}^{soc}_t$ or lower bound $\hat{L}^{soc}_t$) using split conformal prediction.

• Data Splitting: Pre-collected data is split into a training set $D_{train}$ (to train a point predictor $\hat{Y}$) and a calibration set $D_{cal}$.
• Scoring: A score function $S(x, Y(x))$ is defined as the difference between the true outcome and the prediction (e.g., $Y(x) - \hat{Y}(x)$ for maximization problems).
• Interval Construction: The upper bound is calculated as:
  $$\hat{U}^{soc}_t(x_{soc,t}) = \hat{Y}(x_{soc,t}) + Q_{soc}\left(1 - \epsilon; \sum \tilde{w}_i \delta_{S_i} + \tilde{w}_t \delta_{+\infty}\right)$$
  where $Q_{soc}$ is the quantile of the scores, $\epsilon$ is a user-specified failure rate, and $\tilde{w}$ are normalized weights.

2. Handling Covariate Shift

The method accounts for distributional shifts between the calibration data and the test environment (where SafeOpt is deployed) using weighted conformal prediction:

• Scenario 1 (Different Policy): If calibration data was collected under a different policy than the standard of care, Inverse Propensity Scoring (IPS) weights are used: $w_i = \pi_{soc}(x_i) / \pi_{cal}(x_i)$.
• Scenario 2 (Covariate Shift): If the test population differs from the calibration population (e.g., different hospital demographics), weights are derived from the ratio of test and calibration covariate densities: $w_i = P_{test}(x_i) / P_{cal}(x_i)$.
• Non-stationarity: The paper outlines algorithms (Algorithms 3 and 4) to handle changepoints (sudden shifts) and drift (gradual shifts) by re-calculating weights or applying time-decaying weights, respectively.

3. Safety Guarantee Adjustment

Because the method nests conformal prediction within SafeOpt, the target violation rate must be adjusted. Let $\epsilon'$ be the combined failure probability of the conformal estimates (derived via a union bound for multiple estimated terms). The algorithm targets a stricter internal violation rate $\alpha'$ to ensure the overall violation rate remains below the user-defined $\alpha$:
$$\alpha' \leq \frac{\alpha - \epsilon'}{1 - \epsilon'}$$
This adjustment accounts for the probability that the true counterfactual outcome falls outside the conformal interval.

4. Integration with SafeOpt

At each timestep $t$:

1. If the proposed action $x_t$ is the standard of care, the constraint is trivially satisfied.
2. Otherwise, the algorithm computes the conformal upper bound $\hat{U}^{soc}_t$ for the counterfactual outcome.
3. The safety constraint is evaluated as $q(x_t) = Y(x_t) - \hat{U}^{soc}_t(x_{soc,t}) + \omega \geq 0$, where $\omega$ is a tolerance parameter.
4. Standard SafeOpt logic (using Gaussian Processes and online conformal updates for $\beta_t$) selects the next action to maximize the objective while maintaining the safe set.

Key Contributions

1. SafeOpt-CPC Framework: A novel method that ensures safe Bayesian optimization actions do not perform worse than a known safe policy by more than a tolerance $\omega$ for $100(1-\alpha)\%$ of samples, even when the policy's outcomes are unobserved.
2. Robustness to Covariate Shift: The method demonstrates how to adapt conformal estimates to different covariate shift scenarios (policy mismatch and population drift) through re-weighting mechanisms.
3. Formal Safety Proofs: The paper provides a proof (Theorem 4.1) that the method satisfies safety constraints under the assumption of weighted exchangeability. It also bounds the coverage gap when this assumption is violated (Appendix B).
4. Empirical Validation: Experiments on a chemical reaction simulator (where true counterfactuals are known) and the MovieLens dataset (where counterfactuals are simulated) validate that the method keeps violation rates below $\alpha$ while optimizing the objective.

Results

• Violation Rates: In experiments, SafeOpt-CPC consistently maintains the true violation rate below the specified threshold $\alpha$ (e.g., 0.1). In contrast, standard Bayesian optimization without safety constraints violates the threshold significantly.
• Comparison to Oracle: SafeOpt-CPC is slightly more conservative than an "oracle" baseline (which knows the true counterfactual), resulting in a slightly lower maximum objective value, but it guarantees safety where the oracle does not.
• Tolerance Sensitivity: As the tolerance parameter $\omega$ increases, the algorithm reverts to the standard-of-care policy less frequently and achieves higher objective values.
• Non-stationarity: The method successfully adapts to changepoints and drift when weights are correctly re-calculated. Ignoring these shifts leads to a breakdown in coverage guarantees.
• Sensitivity: The method is robust to noisy data and estimator bias/variance, though these factors increase the number of fallback steps to the standard-of-care policy.

Significance and Claims

The paper claims that SafeOpt-CPC provides a rigorous solution for high-risk applications (such as medical treatments) where safety is defined relative to a standard of care rather than an absolute threshold. The authors emphasize that:

• Controlling the violation rate is critical for safety, but the method is designed to be distribution-free regarding the safety constraint observations.
• The approach is generalizable to any constraint requiring an estimated quantity, not just counterfactuals.
• The method operates under minimal assumptions (weighted exchangeability) and provides formal bounds on safety violations.

The authors modestly note limitations: if the conformal quantiles are too wide or the tolerance $\omega$ is too low, the algorithm may become overly conservative, frequently reverting to the standard-of-care policy. Additionally, if multiple estimated terms are present in the constraint, the effective violation rate $\alpha'$ becomes very small, potentially restricting exploration. The paper concludes that while it focuses on violation rate control, the framework can be extended to other constraint functions, a direction left for future work.