Assumption-lean weak limits and tests for two-stage adaptive experiments

1. Problem Statement

Adaptive experiments (e.g., batched bandits, subgroup enrichment designs) are increasingly used to maximize efficiency and welfare by dynamically allocating resources based on interim data. However, the data-dependent nature of these designs violates the independent and identically distributed (i.i.d.) assumptions required for classical statistical inference.

Key Challenges:

• Non-Normal Limits: Standard estimators (like sample means or Inverse Probability Weighted (IPW) estimators) often exhibit non-normal sampling distributions under adaptive data collection, particularly when the signal strength is weak or zero.
• Assumption Sensitivity: Existing asymptotic theories often rely on strong assumptions, such as specific outcome distributions (e.g., Gaussian), strong signal strengths, or contiguous alternatives. Misapplying normal approximations in these settings can lead to severe Type-I error inflation.
• Lack of Unified Framework: Previous results are fragmented, often failing to cover the full spectrum of signal regimes (from zero signal to strong signal) or specific adaptive designs like "drop-the-loser" experiments.

2. Methodology

The authors propose a framework for assumption-lean weak convergence and a simulation-based inference procedure for two-stage adaptive experiments.

A. Experimental Setup

• Two-Stage Design:
  1. Pilot Stage: Collect $N_1$ i.i.d. observations ($D_P$).
  2. Selection: A selection algorithm $S$ processes $D_P$ to determine treatment assignment probabilities for the follow-up stage.
  3. Follow-up Stage: Collect $N_2$ observations ($D_F$) conditionally i.i.d. given the pilot data.
• Estimator: The paper focuses on the Weighted Inverse Probability Weighted (WIPW) estimator, which combines data from both stages. It considers various weighting schemes ($m=0, 1/2, 1$) and scaling (normalized vs. unnormalized).

B. Theoretical Contributions (Weak Limits)

The authors derive new weak convergence results for WIPW estimators under minimal assumptions:

• Minimal Assumptions: The theory requires only moment conditions (bounded 4th moments) and mild restrictions on the sampling function (Lipschitz continuity or step-function structure). It does not assume a specific parametric form for the outcome distribution.
• Signal-Dependent Limits: The limiting distribution is explicitly characterized as a weighted sum of two dependent Gaussian vectors ($A^{(1)}$ and $A^{(2)}$).
  • Strong Signal Regime ($c = -\infty$): The limit converges to a standard Normal distribution.
  • Zero/Weak Signal Regime ($c \in (-\infty, 0]$): The limit is non-normal due to the randomness in the selection algorithm affecting the second-stage weights and covariance.
• Phase Transitions: The paper establishes a smooth transition of the limiting distribution across signal regimes, unifying previously disjoint results.
• Quantitative Bounds: The authors provide quantitative convergence rates in bounded-Lipschitz distance, revealing a trade-off between aggressive exploitation (small clipping rates) and inferential stability.

C. Simulation-Based Inference

Since the limiting distributions are often non-normal and depend on unknown nuisance parameters, the authors propose a computationally efficient, simulation-based testing procedure:

1. Algorithm: The method simulates the asymptotic experiment by sampling from the derived limiting distribution (a weighted sum of dependent Gaussians).
2. Implementation: It estimates nuisance parameters (variances, covariances, sampling probabilities) from the observed data and generates $B$ simulation samples to estimate critical values (quantiles).
3. Efficiency: The procedure has time complexity $O(B)$, independent of the sample size $N$, making it highly scalable.
4. Validity: The authors prove that the resulting tests control the Type-I error rate asymptotically and provide convergence rates for the error.

3. Key Contributions

1. Assumption-Lean Weak Limits: Derivation of weak convergence results for WIPW estimators that hold across all signal regimes (null, weak, strong) without requiring parametric outcome distributions.
2. Unified Framework: A single theoretical lens that connects existing fragmented results and explains the phase transition from non-normal to normal limits based on signal strength.
3. Practical Testing Procedure: A provably valid, nonparametric, and scalable simulation-based method to obtain critical values for hypothesis testing, overcoming the intractability of non-normal limits.
4. Quantitative Analysis: Establishment of bounded-Lipschitz convergence rates, explicitly quantifying the trade-off between the "clipping rate" (exploration) and inferential accuracy.

4. Results

The authors validate their theory through extensive numerical simulations and a semi-synthetic analysis using the SPRINT (Systolic Blood Pressure Intervention Trial) dataset.

• Calibration: The proposed simulation-based tests maintain correct Type-I error control across various distributions (Gaussian, Bernoulli, Poisson, Student, Mixture Normal) and signal strengths, whereas methods relying on normal approximations often suffer from error inflation in weak-signal regimes.
• Power Analysis:
  • No Uniform Dominance: Neither normality-based tests (e.g., Batched Difference-in-Means) nor non-normal limit tests uniformly dominate in power.
  • Outcome Structure Matters:
    • Under continuous outcomes (e.g., Gaussian), adaptive weighting (non-normal limit tests) generally achieves higher power.
    • Under discrete outcomes (e.g., Bernoulli), normal-limit tests can be more powerful unless the outcomes are sparse (low event rates), where adaptive IPW methods outperform.
  • Adaptive Weighting: Tests using adaptive weighting ($m=1/2$ or $m=1$) consistently outperform constant weighting, especially when the selection algorithm leads to significant downsampling of one arm.
• Semi-Synthetic Data: In the SPRINT analysis (sparse binary outcomes), the proposed adaptive methods controlled Type-I error effectively, while sample-splitting methods suffered from inflation due to sparsity.

5. Significance

• Robustness: The framework enables valid inference in real-world adaptive experiments where outcome distributions are complex, unknown, or non-standard (e.g., single-cell CRISPR screens, clinical trials with rare events).
• Practical Utility: The simulation-based procedure provides a "plug-and-play" solution for practitioners to conduct hypothesis tests without deriving complex analytical limits for every new experimental design.
• Design Guidance: The results offer clear insights for experimental design, highlighting that the choice of testing method should depend on the sparsity and structure of the outcome distribution, not just the sample size.
• Theoretical Advancement: By extending classical tools (CLT, Continuous Mapping Theorem) to conditional settings with adaptive dependence, the paper bridges the gap between theoretical statistics and modern adaptive data collection practices.

In summary, this paper provides a rigorous, assumption-lean foundation for inference in two-stage adaptive experiments, offering both theoretical guarantees on the behavior of estimators and a practical, scalable algorithm for hypothesis testing that adapts to the underlying data structure.