📊 statistics

Design Stability in Adaptive Experiments: Implications for Treatment Effect Estimation

This paper establishes central limit theorems and asymptotic variances for inverse propensity weighted and augmented inverse propensity weighted estimators of the average treatment effect under sequentially adaptive assignment mechanisms by introducing the concept of design stability, thereby enabling valid statistical inference in sequential experiments.

Original authors: Saikat Sengupta, Koulik Khamaru, Suvrojit Ghosh, Tirthankar Dasgupta

Published 2026-04-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Saikat Sengupta, Koulik Khamaru, Suvrojit Ghosh, Tirthankar Dasgupta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to figure out if a new medicine works. You have a group of patients, and you need to decide who gets the real medicine (Treatment) and who gets a sugar pill (Control).

In the old days, scientists would flip a fair coin for every single patient. Heads = Medicine, Tails = Sugar. This is simple, but sometimes, by pure bad luck, you might end up with 90% of your "sick" patients in the sugar pill group. That makes it hard to tell if the medicine actually works or if the groups were just unbalanced to begin with.

To fix this, modern scientists use Adaptive Designs. Instead of a fair coin, they use a "smart coin." If too many people have already gotten the medicine, the smart coin becomes heavier on the "Sugar" side to balance things out. If the groups are balanced, it flips fairly.

The Problem:
While this "smart coin" sounds great for balancing the groups, it creates a statistical nightmare. Because the coin changes based on what happened before, the data points are no longer independent. It's like trying to predict the weather when every day's forecast depends on the previous day's weather in a complex, shifting way. Traditional math tools break down here, making it hard to know if your results are real or just a fluke.

The Solution: "Design Stability"
The authors of this paper, Saikat Sengupta and his team, propose a new way to handle this chaos. They introduce a concept called Design Stability.

Think of Design Stability as the "calm after the storm."

Strong Stability: Imagine the smart coin eventually stops changing its mind. After 1,000 patients, it settles into a rhythm where it gives the medicine 50% of the time, no matter what happened before. It becomes predictable.
Weak Stability: Imagine the coin never settles on a single number. It might flip 60% heads today and 40% tomorrow. However, if you look at the average of all those flips over a long time, it settles into a predictable pattern. The average behavior is stable, even if the individual flips are wild.

The paper proves that as long as your experiment has one of these two types of "stability," you can still do the math correctly.

The Tools: Two New Estimators
The authors provide two mathematical "lenses" (estimators) to look at the data and calculate the true effect of the treatment:

The IPW Lens (Inverse Propensity Weighted):
- Analogy: Imagine you are counting votes in an election where some voters were harder to reach than others. If you only counted the easy-to-reach voters, your results would be biased. The IPW lens says, "Okay, this voter was hard to reach, so their vote counts more to balance the scale." It weighs every patient's result by how likely they were to get the treatment.
- Pros: It's simple and unbiased.
- Cons: It can be "noisy" (high variance), meaning your confidence intervals (the range where you think the truth lies) are very wide.
The AIPW Lens (Augmented IPW):
- Analogy: This is the IPW lens with a "co-pilot." It uses the IPW method but also looks at the patient's history to make a better guess about what would have happened. It's like a weather forecaster who looks at the barometer (IPW) and the satellite images (the patient's history) to make a more accurate prediction.
- Pros: It is much more precise. It gives you a tighter, more accurate range for your results.
- Cons: It's mathematically more complex to build.

The "Safety Net" (Conservative Variance)
One of the biggest challenges in these adaptive experiments is that you can't see the "what if" scenarios. You know what happened to the patient who got the medicine, but you don't know what would have happened if they got the sugar pill (and vice versa).

Because of this missing information, the authors propose Conservative Variance Estimators.

Analogy: Imagine you are building a bridge. You don't know the exact weight of every truck that will cross it in the future. So, instead of calculating for the average truck, you design the bridge to hold the weight of a heavy truck. You might be over-engineering it (making the bridge wider than strictly necessary), but you guarantee it won't collapse.
In statistics, this means your confidence intervals might be slightly wider than they could be, but they are guaranteed to contain the true answer. You won't get a false positive.

Real-World Examples
The paper tests these ideas on two famous "smart coin" designs:

Wei's Design: This design is like a thermostat. If the room gets too hot (too many treatments), it cools down. The paper proves this design has Strong Stability (it settles down nicely).
Efron's Design: This design is like a strict referee. If one team is winning by too much, the referee forces a penalty to the other team. This design is a bit more chaotic and doesn't settle on a single number, but it has Weak Stability (the average behavior is predictable).

Why This Matters
This paper is a "user manual" for the future of clinical trials and online experiments (like A/B testing on websites).

It tells us that Adaptive Designs are safe to use as long as they are "stable."
It gives us the math tools to analyze them without making up fake assumptions about how the data is generated.
It shows that the AIPW method (the lens with the co-pilot) is generally the best choice because it gives more precise answers, saving time and money in medical trials.

In a Nutshell:
The authors took a messy, complex problem (adaptive experiments where the rules change as you go) and showed us that if the rules eventually settle into a predictable pattern (Stability), we can use specific, robust math tools to get accurate, trustworthy results. They gave us a way to build a "safety net" so we never accidentally claim a medicine works when it doesn't.

1. Problem Statement

The paper addresses the challenge of estimating the Average Treatment Effect (ATE) in finite-population settings where treatment assignments are sequentially adaptive.

Context: Unlike classical Completely Randomized Designs (CRD) where assignment probabilities are fixed and independent, modern experiments (e.g., adaptive clinical trials, online A/B testing) often assign treatments based on the history of previous assignments and observed outcomes.
The Challenge: These adaptive mechanisms introduce dependencies between units, violating the i.i.d. assumptions required for standard asymptotic theory. Consequently, traditional estimators (like the difference-in-means) may become biased or inefficient, and standard confidence intervals may be invalid.
Goal: To develop a rigorous inferential framework (estimators, asymptotic normality, and valid confidence intervals) for ATE under sequential adaptive designs without assuming a parametric model for the potential outcomes (design-based inference).

2. Methodology and Framework

A. Potential Outcomes Framework

The authors adopt the Neyman-Rubin potential outcomes framework. For $N$ units, each unit $i$ has fixed potential outcomes $Y_i(0)$ (control) and $Y_i(1)$ (treatment). The target parameter is the finite-population ATE: $\bar{\tau} = \frac{1}{N}\sum_{i=1}^N (Y_i(1) - Y_i(0))$ .

Assumption: The potential outcomes are fixed; all randomness arises solely from the treatment assignment mechanism.

B. Sequential Assignment Mechanism

The assignment of the $i$ -th unit, denoted by $K_i \in \{0, 1\}$ , follows a Bernoulli distribution with a success probability $p_i$ that depends on the history $\mathcal{F}_{i-1}$ (past assignments and outcomes):
$P(K_i = 1 | \mathcal{F}_{i-1}) = p_i$
This allows for arbitrary adaptive rules, such as balancing treatment arms based on current imbalances.

C. Estimators

The paper analyzes two estimators:

Inverse Propensity Weighted (IPW) Estimator: A Horvitz-Thompson type estimator that reweights observed outcomes by the inverse of their assignment probabilities.
$\hat{\tau}_{IPW} = \frac{1}{N} \sum_{i=1}^N \left( \frac{K_i Y_i}{p_i} - \frac{(1-K_i)Y_i}{1-p_i} \right)$
Augmented IPW (AIPW) Estimator: A model-free, finite-population analogue of the standard AIPW estimator. It uses sequential estimates of the conditional means of potential outcomes to reduce variance.
$\hat{\tau}_{AIPW} = \frac{1}{N} \sum_{i=1}^N \left[ \left( \frac{K_i(Y_i - \hat{Y}_{i-1}(1))}{p_i} + \hat{Y}_{i-1}(1) \right) - \left( \frac{(1-K_i)(Y_i - \hat{Y}_{i-1}(0))}{1-p_i} + \hat{Y}_{i-1}(0) \right) \right]$
where $\hat{Y}_{i-1}(\ell)$ are running weighted averages of observed outcomes for treatment arm $\ell$ .

D. The Core Concept: Design Stability

The theoretical breakthrough of the paper is the definition of Design Stability, a condition required to establish Central Limit Theorems (CLT) for these estimators. The authors propose two levels:

Strong Design Stability: The assignment probability $p_i$ $p_{i}$ itself converges in probability to a non-random constant $p^*$ $p^{*}$ .
$p_i \xrightarrow{P} p^*$
- Implication: The design "freezes" asymptotically.
Weak Design Stability: The assignment probabilities $p_i$ $p_{i}$ need not converge individually, but the sample averages of the inverse propensity scores must converge to fixed limits.
$\frac{1}{N}\sum_{i=1}^N \frac{1}{p_i} \xrightarrow{P} \frac{1}{p^*_1} \quad \text{and} \quad \frac{1}{N}\sum_{i=1}^N \frac{1}{1-p_i} \xrightarrow{P} \frac{1}{p^*_2}$
- Implication: This acts as a Law of Large Numbers for the variance process, allowing for CLTs even if $p_i$ fluctuates (e.g., Efron's design).

3. Key Contributions and Results

A. Asymptotic Normality (Central Limit Theorems)

The authors prove that under both Strong and Weak Design Stability, both $\hat{\tau}_{IPW}$ and $\hat{\tau}_{AIPW}$ are asymptotically normal:
$\sqrt{N}(\hat{\tau} - \bar{\tau}) \xrightarrow{d} \mathcal{N}(0, V)$

Variance Expressions: Explicit formulas for the asymptotic variances ( $V_{IPW}$ and $V_{AIPW}$ ) are derived.
Efficiency: The AIPW estimator is shown to be more efficient (lower asymptotic variance) than the IPW estimator ( $V_{AIPW} \leq V_{IPW}$ ).

B. Variance Estimation and Confidence Intervals

A major hurdle in causal inference is that the cross-moment term (covariance between $Y_i(0)$ and $Y_i(1)$ ) is unobservable.

Conservative Estimation: The authors propose variance estimators that use the Cauchy-Schwarz inequality to bound the unobservable covariance term. This results in conservative variance estimators (asymptotically positively biased), ensuring that confidence intervals have at least the nominal coverage probability ( $1-\alpha$ ).
Exact Coverage Conditions: The estimators become asymptotically exact (non-conservative) if the potential outcomes satisfy specific homogeneity conditions:
- For IPW: Log-additivity ( $Y_i(1) = c Y_i(0)$ ).
- For AIPW: Generalized treatment effect homogeneity ( $Y_i(1) - \bar{Y}(1) = \lambda(Y_i(0) - \bar{Y}(0))$ ).

C. Application to Classic Designs

The theory is applied to two seminal adaptive designs:

Wei's Adaptive Coin Design: Proven to satisfy Strong Design Stability (converges to $p^*=0.5$ ).
Efron's Biased Coin Design: Proven to satisfy Weak Design Stability but not Strong Stability (probabilities oscillate between $\eta$ and $1-\eta$ ). The paper derives the specific limiting constants for Efron's design, demonstrating the utility of the Weak Stability framework.

4. Significance and Impact

Bridging Theory and Practice: The paper provides the first rigorous finite-population inferential framework for a broad class of adaptive designs, moving beyond the restrictive assumptions of superpopulation models often used in bandit literature.
Robustness: By defining "Weak Design Stability," the authors extend valid inference to popular designs (like Efron's) that were previously difficult to analyze theoretically because they do not converge to a single probability.
Practical Utility: The proposed conservative variance estimators allow practitioners to construct valid confidence intervals for ATE in adaptive trials without needing to model the outcome distribution.
Superiority of AIPW: The results empirically and theoretically confirm that the AIPW estimator offers better efficiency and stability in adaptive settings compared to standard IPW, particularly when treatment effects are heterogeneous.

5. Conclusion

Sengupta et al. successfully establish that sequential adaptive experiments can be analyzed using standard asymptotic tools provided the concept of Design Stability is met. Their work unifies the analysis of various adaptive randomization schemes, offering a robust, model-free methodology for estimating treatment effects with valid uncertainty quantification in finite populations.