📊 statistics

Selective Randomization Inference for Adaptive Experiments

This paper proposes a general framework called selective randomization inference that uses conditional post-selection inference and directed acyclic graphs to perform valid statistical testing and construct confidence intervals for adaptive experiments without relying on modeling assumptions or i.i.d. data.

Original authors: Tobias Freidling, Qingyuan Zhao, Zijun Gao

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Tobias Freidling, Qingyuan Zhao, Zijun Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery. In a standard investigation, you decide exactly what questions to ask and how to gather evidence before you start looking at the clues. You stick to that plan, and at the end, you calculate your odds of being right based on that fixed plan.

But in the real world, science often works differently. It's more like a detective who looks at the first few clues, realizes the suspect is hiding in a specific neighborhood, and then decides to focus all their energy on that neighborhood. This is called an adaptive experiment. It's smart because it saves time and resources, but it creates a tricky statistical problem: Selection Bias.

The Problem: The "Unfair" Comparison

The paper explains that if you change your plan based on what you see, you can't just use the old rules to judge your results.

The Analogy of the Dice:
Imagine you are rolling dice to see if a new game is fair.

Round 1: You roll the dice 10 times. You see that the number "6" keeps coming up.
The Decision: Because "6" looks promising, you decide, "Okay, from now on, I'm only going to care about the number 6. I'm going to ignore all other numbers."
Round 2: You roll the dice 10 more times, but this time you only count the "6"s.

If you try to calculate the odds of getting so many "6"s using standard math, you will be wrong. Why? Because you didn't just roll the dice; you chose to focus on "6" because you saw it happen in Round 1. You "cherry-picked" the data. If you compare your results to a standard dice roll (where you didn't pick a number), you are comparing apples to oranges. You are essentially cheating by changing the rules after seeing the first half of the game.

The Solution: "Selective Randomization Inference"

The authors, Tobias Freidling, Qingyuan Zhao, and Zijun Gao, propose a new way to do the math called Selective Randomization Inference.

Think of it as a "What-If" Simulation that respects your detective work.

Instead of asking, "What are the odds of this happening if I had stuck to my original plan?" (which is unfair because you didn't stick to the plan), they ask:

"If I had rolled the dice differently in Round 1, but still ended up deciding to focus on the number 6, how likely would my results be?"

How it works in the paper's language:

The Setup: They use a "Directed Acyclic Graph" (DAG), which is just a fancy flowchart showing how one thing leads to another (like: See Data $\to$ Choose Subgroup $\to$ Assign Treatment).
The Condition: They only compare your actual experiment to other possible experiments that would have led to the exact same decision to focus on that specific subgroup.
The Result: This gives you a "Selective P-value." It tells you the odds of your result happening given that you made the specific choice you made.

Why This is Better Than "Data Splitting"

Before this paper, the common advice for this problem was Data Splitting.

The Old Way: Throw away the first half of your data (the part that made you change your mind) and only use the second half to do the math.
The Metaphor: It's like a chef tasting a soup, deciding it needs more salt, and then throwing away the first half of the soup to prove the salt works. It's safe, but it's wasteful. You lose a lot of information.

The authors' new method is like Data Carving.

The New Way: You keep all the data. You just acknowledge that some of the "flavor" (information) was used to make the decision to add salt. You mathematically "carve out" the part used for the decision and only compare the remaining "leftover" information against similar scenarios.
The Benefit: You get a more powerful test. You use all the data you have, but you don't get fooled by the fact that you made a choice along the way.

The "Hold-Out" Trick

The paper also mentions a practical trick to make the math easier and the results smoother. Sometimes, if you use all the data to make your decision, the math gets jagged and unstable (like a bumpy road).

To fix this, they suggest using a "Hold-Out" group.

The Metaphor: Imagine you are a teacher grading a class. You look at the first 10 students' tests to decide which topic is hardest. Then, you decide to focus your final exam on that topic. But, to grade the final exam fairly, you only look at the last 10 students' scores, who weren't part of the group you used to make the decision.
In the paper, they show that if you use a "hold-out" group (data not used for the decision), the math becomes much smoother and more reliable, while still being very powerful.

Summary of Claims

The paper claims that:

Adaptive experiments are common (in medicine, social science, etc.) but hard to analyze because the rules change based on the data.
Standard methods fail because they ignore the fact that the researcher made a choice based on the data, leading to false positives (thinking a treatment works when it doesn't).
Their new method (Selective Randomization Inference) fixes this by only comparing the real experiment to "parallel universes" where the researcher would have made the same choice.
It requires no assumptions about the data distribution (it doesn't assume the data follows a bell curve or is independent), relying only on the randomness of the treatment assignment.
It is more powerful than throwing away data (data splitting) because it uses all the available information, just with a more careful mathematical filter.

In short, the paper gives scientists a new, fair rulebook for analyzing experiments where they change their strategy mid-game, ensuring that their conclusions are solid and not just a lucky accident of how they chose to look at the data.

Technical Summary: Selective Randomization Inference for Adaptive Experiments

Problem Statement

Adaptive experiments, which utilize preliminary data analyses to inform subsequent stages of recruitment and treatment assignment, are increasingly common in medical and social sciences. However, standard statistical inference methods often fail in these settings because the null hypothesis and experimental design become data-dependent. This "selection" introduces bias, rendering naive randomization tests invalid as they ignore the fact that future design choices were conditioned on past outcomes. While existing solutions exist, they are often limited to specific adaptive designs (e.g., multi-arm multi-stage trials) or rely on strong parametric assumptions and asymptotic approximations that may not hold in finite populations. Furthermore, simple data-splitting approaches, while valid, are inefficient as they discard the information contained in the initial stage used for selection.

Methodology

The authors propose Selective Randomization Inference (SRI), a general, model-free framework for analyzing adaptive experiments. The core methodology integrates conditional post-selection inference with the principles of randomization inference.

1. Causal Framework and Assumptions

The authors model adaptive experiments with $K$ stages using Directed Acyclic Graphs (DAGs). They define a general setup involving:

Recruitment ( $R_k$ ): Modeled as a random event where the set of recruited units can depend on previous data.
Treatment Assignment ( $Z_k$ ): Randomized by the experimenter but potentially dependent on covariates and past outcomes.
Selection Statistics ( $S_k$ ): Deterministic functions of observed data that dictate the design of subsequent stages (e.g., which subgroups to recruit).

Three minimal assumptions are required for validity:

(A1) Known Sequential Assignment: The distribution of treatment assignments $q(z | r, x, y)$ is known.
(A2) Conditional Independence: Treatment assignment at stage $k$ is conditionally independent of potential outcomes given past observed data and current covariates.
(A3) Selection Sufficiency: The selection statistic $S_{k-1}$ captures all information from previous stages that influences the recruitment and treatment assignment of stage $k$ .

2. The Selective Randomization P-value

The central contribution is the definition of a selective randomization p-value ( $P_{sel}$ ). Unlike a naive p-value which compares the observed treatment assignment $Z$ against all possible random assignments, $P_{sel}$ conditions on the specific selection event that occurred.

Formally, let $Z^*$ be a counterfactual treatment assignment drawn from the sequential assignment distribution. The p-value is defined as:
$P_{sel} = P\left( T(Z^*, \dots) \leq T(Z, \dots) \mid R, X_R, Y_R(\cdot), Z, S(Z^*) = S(Z), G(Z^*) = G(Z) \right)$
where:

$T$ is the test statistic.
$S(Z^*) = S(Z)$ ensures the counterfactual assignment leads to the same design selection as the observed data.
$G(Z^*) = G(Z)$ is an additional conditioning statistic required for "partially sharp" null hypotheses to ensure unobserved potential outcomes can be imputed.

The authors prove that this p-value stochastically dominates the uniform distribution, thereby controlling the selective type-I error (the error rate conditional on the selection event).

3. Computational Approaches

Since the support of the conditional distribution (assignments satisfying $S(Z^*)=S(Z)$ ) is often intractable to enumerate, the authors propose Monte Carlo approximations:

Rejection Sampling: Simple but potentially inefficient if the probability of satisfying the selection constraint is low.
Random Walk Metropolis-Hastings (RWM): A Markov Chain Monte Carlo (MCMC) algorithm that constructs a chain on the space of feasible assignments. The authors provide theoretical guarantees that the chain converges to the correct stationary distribution and that the resulting p-value controls the selective type-I error, even if the chain is restricted to a specific communication class.

4. Estimation and Confidence Intervals

The framework extends to constructing confidence intervals for homogeneous treatment effects by inverting the selective hypothesis tests. The authors note that conditioning on selection can lead to non-monotonic p-value curves, complicating interval construction. To mitigate this, they suggest using hold-out units (data not used for selection) to smooth the p-value curve and ensure well-behaved confidence sets.

Key Results

The paper validates the method through simulation studies and a hypothetical analysis of real-world data (SPRINT trial):

Validity: In simulations where the null hypothesis is true, the naive randomization test (RT) exhibits inflated type-I error rates (under-coverage of confidence intervals), whereas the proposed SRT and the data-splitting approach (RT 2nd) correctly control the selective type-I error.
Power: The SRT generally demonstrates higher statistical power than the data-splitting approach (RT 2nd). By conditioning on the selection event rather than discarding the first stage, SRT utilizes "left-out" information from the initial stage, resulting in tighter confidence intervals and higher rejection probabilities for false nulls.
Computation: While rejection sampling is straightforward, it can be computationally expensive when the selection constraint is rare. The RWM sampler offers a more efficient alternative, with performance largely insensitive to window size choices in the tested scenarios.
Real Data Application: In a hypothetical two-stage enrichment trial derived from the SPRINT trial, the SRT yielded a smaller p-value (0.048) compared to the data-splitting approach (0.314) for the selected subgroup, demonstrating its ability to detect effects that data-splitting might miss due to reduced sample size.

Significance and Claims

The paper claims to provide a general, model-free framework for inference in adaptive experiments. Its significance lies in several key aspects:

Generality: Unlike previous methods restricted to specific designs (e.g., response-adaptive randomization or multi-arm trials), this approach applies to a wide class of adaptive designs, including enrichment designs and complex recruitment rules.
Model-Free Nature: The method relies solely on the randomness of treatment assignment, requiring no assumptions about the distribution of outcomes, independence of data points, or parametric models. This makes it a robust "default" test for randomized experiments.
Efficiency: It overcomes the inefficiency of data-splitting by utilizing all available data while rigorously correcting for selection bias.
Regulatory Relevance: The authors argue that controlling the conditional type-I error is crucial for regulatory perspectives, as it treats the selected hypothesis as given, allowing for objective evaluation of evidence in a randomized experiment.

The authors position this work as a complement to existing super-population frameworks, offering a finite-population alternative that minimizes modeling assumptions. They acknowledge that while the theoretical framework is established, practical extensions regarding optimal test statistic selection and handling drop-outs remain areas for future research.