Anytime-Valid Inference in Adaptive Experiments: Covariate Adjustment and Balanced Power

1. Problem Statement

Adaptive experiments, such as Multi-Armed Bandits (MABs), offer significant efficiency gains over traditional Randomized Controlled Trials (RCTs) by dynamically allocating samples to optimal treatments. However, they present two critical challenges for causal inference:

1. Invalid Inference: The dependence of treatment assignment $W_t$ on historical data $\{(W_1, Y_1), ..., (W_{t-1}, Y_{t-1})\}$ violates the independence assumptions required by standard Average Treatment Effect (ATE) estimators (e.g., difference-in-means, inverse propensity scoring). This renders standard confidence intervals invalid and inflates Type I error rates.
2. Imbalanced Statistical Power: Adaptive algorithms naturally concentrate samples on the optimal treatment arm to maximize cumulative welfare. Consequently, sub-optimal arms are often underpowered, making it difficult to draw robust statistical conclusions about them.

While recent work (e.g., Liang and Bojinov, 2024) introduced the Mixture Adaptive Design (MAD) to enable anytime-valid inference (using Confidence Sequences, or CSs) that remains valid under arbitrary adaptive designs, MAD still suffers from two limitations:

• Low Precision: ATE estimates often have high variance, resulting in wide confidence sequences.
• Power Imbalance: It does not actively correct the sample allocation bias toward optimal arms, leaving sub-optimal treatments underpowered.

2. Methodology

The authors propose two extensions to the MAD framework to address these issues: MADCovar and MADMod. Both methods retain the core MAD mechanism, which mixes a fully random assignment policy with an adaptive policy $A$ (e.g., Thompson Sampling) via a deterministic sequence $\delta_t$.

A. MADCovar: Covariate-Adjusted Estimation

MADCovar improves the precision of ATE estimates by incorporating covariate adjustment into the MAD framework.

• Estimator: It utilizes a doubly robust, covariate-adjusted estimator for individual treatment effects ($\hat{\tau}_i$):
  $$\hat{\tau}_i := \hat{\mu}_1(X_i) - \hat{\mu}_0(X_i) + \frac{\mathbb{I}\{W_i=1\}(Y_i - \hat{\mu}_1(X_i))}{p^{MAD}_i(1)} - \frac{\mathbb{I}\{W_i=0\}(Y_i - \hat{\mu}_0(X_i))}{p^{MAD}_i(0)}$$
  Where $\hat{\mu}_w(X_i)$ is an arbitrary estimator of the conditional mean outcome $E[Y_i(w)|X, \mathcal{F}_{i-1}]$.
• Theoretical Guarantee: Under bounded outcome model assumptions, the authors prove that the sequence $(\hat{\bar{\tau}}_t \pm \hat{V}_t)$ forms a valid $(1-\alpha)$ asymptotic Confidence Sequence (CS). This allows researchers to monitor the ATE continuously and stop data collection once a desired precision or significance is met, while maintaining Type I error control.
• Robustness: The method is shown to be robust even when including many irrelevant covariates in the outcome model.

B. MADMod: Power-Modified Allocation

MADMod addresses the issue of imbalanced statistical power by dynamically reallocating samples to underpowered arms.

• Mechanism: It introduces importance weights $w_k(t)$ for each treatment arm $k$.
  • If an arm has not yet reached statistical significance (i.e., its CS includes 0), $w_k(t) = 1$.
  • If an arm reaches significance, $w_k(t)$ begins to decay according to a researcher-defined function (e.g., polynomial decay).
• Re-weighting: The adaptive assignment probabilities $p^A_t(k)$ are scaled by these weights. The "lost" probability mass from significant arms is redistributed proportionally to the remaining weights of the underpowered arms.
• Goal: This ensures that while the experiment remains efficient (favoring optimal arms), sufficient samples are allocated to sub-optimal arms to ensure they are well-powered for inference.

3. Key Contributions

1. MADCovar: A novel estimator that combines covariate adjustment with anytime-valid inference. It significantly reduces the width of confidence sequences without sacrificing the validity guarantees of the MAD framework.
2. MADMod: An algorithmic modification to adaptive sampling that balances efficiency with statistical power. It allows researchers to trade a small amount of efficiency (allocating more samples to sub-optimal arms) for a substantial gain in the power to detect effects across all treatments.
3. Theoretical Rigor: The paper establishes that both methods preserve the anytime-valid property, meaning researchers can stop data collection at any time based on the data observed without inflating Type I error rates.
4. Practical Implementation: The authors provide an open-source Python package (pyssed) implementing these methods, along with a replication package for all figures and tables.

4. Empirical Results

The authors validated their methods through extensive simulations and a large-scale real-world application.

• Simulations (MADCovar):
  • In settings with medium to high covariate signal strength, MADCovar reduced the width of Confidence Sequences by 40–70% compared to standard MAD.
  • These gains persisted even when the outcome models included a high number of irrelevant covariates (up to 50 covariates).
• Large-Scale RCT (MADCovar):
  • Applied to a political science mega-study (Voelkel et al., 2024) with ~32,000 participants.
  • MADCovar achieved 60–75% precision gains relative to the non-covariate-adjusted baseline, demonstrating that the simulation results translate to real-world data.
• Simulations (MADMod):
  • In a 4-arm bandit setting, MADMod significantly reduced Type II errors for sub-optimal treatments (arms 1 and 2) compared to standard MAD.
  • While MADMod allocated more samples to sub-optimal arms (sacrificing some cumulative reward efficiency), it resulted in narrower confidence bands and more reliable inference for all arms, ensuring no treatment was left underpowered.

5. Significance and Impact

• Practicality for Applied Research: The proposed methods make adaptive experiments viable for domains requiring rigorous causal inference (e.g., healthcare, education, social policy) where standard MABs are often rejected due to inferential concerns.
• Ethical Efficiency: By improving precision without requiring larger sample sizes, these methods reduce costs and limit participant exposure to inferior treatments.
• Comprehensive Inference: MADMod specifically addresses the "winner-take-all" bias of adaptive designs, ensuring that researchers can confidently evaluate the effectiveness of all interventions, not just the optimal one.
• Flexibility: Unlike some theoretical approaches that require specific MAB algorithms, MADMod is designed to be integrated with virtually any existing adaptive design (e.g., Thompson Sampling, UCB), offering researchers control over the trade-off between efficiency and power.

In conclusion, Molitor and Gold provide a robust toolkit that resolves the tension between the efficiency of adaptive sampling and the rigor of causal inference, enabling researchers to conduct experiments that are both statistically powerful and ethically sound.