📊 statistics

Semiparametric Efficiency in Sequential Experiments: Characterization and Design via Average Propensity

This paper establishes a semiparametric efficiency benchmark for sequential experiments based on an induced average propensity score and proposes implementable batched adaptive designs that utilize regression adjustment or covariate balancing to achieve this optimal precision under various operational constraints.

Original authors: Jiachun Li, David Simchi-Levi

Published 2026-07-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiachun Li, David Simchi-Levi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive online platform. You have developed several new AI features (let's call them "AI Assistants") and you want to know which one works best for your users. To find out, you run an experiment: you show different users different assistants and measure the results.

In the old days, you would just flip a coin for every user. This is called "random assignment." It's fair, but it's not very smart. If you have a user who is very tech-savvy and another who isn't, a coin flip might accidentally give the tech-savvy user the "hard-to-use" assistant and the non-tech user the "easy" one. This creates "noise" in your data, making it harder to tell which assistant is actually better.

This paper is about how to run these experiments smarter, especially when you have to make decisions one by one (sequentially) and can't wait for all the data to come in before making a change.

Here is the core idea, broken down into simple concepts:

1. The Problem: The "Moving Target"

In modern experiments, you can't just flip a coin once and stick with it. You might need to:

Adapt: If Assistant A seems to be failing, you might want to stop showing it to new users.
Balance: You might want to make sure you have an equal number of tech-savvy and non-tech-savvy users in each group.
Follow Rules: You might have a budget limit (only 100 people can see Assistant B) or fairness rules.

These rules make the data messy. The users aren't independent anymore; what happened to User #1 affects who gets what treatment for User #2. Standard statistical tools, which assume everyone is independent, break down.

2. The Big Discovery: The "Average Recipe"

The authors found a way to simplify this mess. They realized that no matter how complex your rules are (adaptive, balanced, budgeted), they all boil down to one simple number: The Average Propensity Score.

Think of this like a recipe.

Imagine you are baking cookies. You have a complex set of instructions: "If the kitchen is hot, add less sugar. If the oven is old, bake longer."
The authors say: "Don't worry about the complex instructions. Just look at the final average amount of sugar you actually used across all the cookies you baked."
That "average amount of sugar" is the Average Propensity Score.

The Magic Claim: The paper proves that the precision of your experiment (how clearly you can see the truth) depends only on this average recipe. It doesn't matter if your rules were complicated; if your average recipe is good, your experiment will be efficient. If your average recipe is bad, no amount of fancy math can save you.

3. The Goal: The "Perfect Recipe"

If you knew exactly how every user would react, you could calculate the Perfect Recipe (called the "Oracle Benchmark"). This recipe tells you exactly how many users of each type should get each assistant to get the clearest possible answer with the fewest number of people.

The paper asks: Can we design an experiment that gets us close to this Perfect Recipe, even if we don't know the answers in advance?

4. The Solution: Two Ways to Cook

The authors propose two practical methods to get close to that Perfect Recipe. Both methods use a strategy called "Batching." Instead of changing the rules every second (which is chaotic and hard to manage), you change the rules every "batch" (e.g., every 1,000 users).

Method A: The "Smart Adjuster" (Regression Adjustment)

How it works: You run a batch of users. Then, you look at the data and use a computer model (like a smart calculator) to guess which users responded well to which assistant. You use this guess to tweak the "recipe" for the next batch.
The Catch: This method relies on the computer model being very accurate. If the model is slightly wrong, it can mess up the results. The paper shows this works well, but only if the model is good enough.
Analogy: It's like a chef tasting the soup, guessing what's missing, and adding spices. If the chef's taste buds are off, the soup might still be salty.

Method B: The "Balanced Scale" (Covariate Balancing)

How it works: Instead of guessing the answer with a model, this method focuses on forcing balance during the assignment. It ensures that within each batch, the groups are perfectly matched (e.g., exactly the same number of tech-savvy users in every group).
The Benefit: Because the groups are perfectly balanced, you don't need a fancy computer model to fix the data later. You can use a simple, robust math formula (like a simple average) to get the answer.
The Catch: It's harder to balance perfectly if you have too many different types of users (high dimensions).
Analogy: It's like a chef who doesn't taste the soup but instead carefully measures every ingredient to ensure the ratios are perfect from the start. The soup turns out right because the ingredients were balanced, not because the chef guessed the taste.

5. Real-World Proof

The authors tested these ideas in two ways:

Simulations: They created fake data with different levels of complexity (some easy, some very hard with many variables). They found that their methods consistently beat the old "coin flip" method.
Real Data (AI Medical Assistants): They applied this to a real study evaluating AI medical assistants. They had to compare four different AI assistants.
- They found that by using their "batched" methods, they could get the same level of accuracy with fewer users (or better accuracy with the same number of users).
- They also showed that for the "Balanced Scale" method, it helped to focus on just the most important user traits (like age and scenario type) rather than trying to balance every single detail.

Summary

This paper provides a new "rulebook" for running modern experiments.

The Rule: Don't worry about the complex rules you use to assign treatments. Just focus on the average distribution of those treatments.
The Strategy: Run experiments in batches.
The Tools: You can either use a smart model to adjust the recipe (Method A) or use strict balancing to ensure fairness (Method B).
The Result: You get more accurate answers about what works, faster, and with fewer resources, even when the experiment is complex and changing in real-time.

It's like moving from flipping a coin to using a GPS that recalculates your route every few miles to ensure you get to the destination as efficiently as possible.

Technical Summary: Semiparametric Efficiency in Sequential Experiments via Average Propensity

1. Problem Formulation

Modern experimental settings, particularly in the evaluation of AI-enabled services and platform interventions, frequently deviate from classical independent and identically distributed (i.i.d.) sampling. Assignments are often adaptive, balanced across covariates, or constrained by operational limits such as exposure caps, treatment budgets, and fairness requirements. These features induce dependence across observations, rendering classical i.i.d.-based efficiency theory insufficient as a guide for experimental design.

The paper addresses two central questions:

What is the fundamental efficiency benchmark for estimating a target causal estimand in a sequential experiment with dependent assignments?
How can implementable designs approach this benchmark in practice, particularly under constraints like delayed feedback and the need for operational stability?

The authors formalize the setting using a non-anticipating design framework. Units are indexed by an assignment order $t=1,\dots,n$ , and the treatment assignment $A_t$ depends only on the history $\mathcal{F}_{t-1}$ and the current covariate $X_t$ , never on future outcomes or covariates. This framework unifies i.i.d. randomization, offline stratified designs, and sequential adaptive designs.

2. Methodology and Theoretical Framework

2.1 The Induced Average Propensity Score

The core theoretical contribution is the identification of the induced average propensity score as the sufficient statistic for the efficiency benchmark. For any non-anticipating design $\text{ALG}$ , the induced average propensity $\bar{e}_a(x)$ is defined as:
$\bar{e}_a(x) := \frac{1}{n} \sum_{t=1}^n \mathbb{E}_{\text{ALG}}[e_t(a | x, \mathcal{F}_{t-1})]$
where $e_t$ is the conditional assignment probability at step $t$ .

2.2 Semiparametric Lower Bound

The paper establishes a semiparametric lower bound for regular, locally unbiased estimators. Theorem 1 proves that for any non-anticipating design, the asymptotic variance of an estimator is bounded below by the i.i.d. efficiency benchmark evaluated at the induced average propensity $\bar{e}$ :
$\text{Var}(T) \geq \frac{1}{n} V(\bar{e}) = \frac{1}{n} \mathbb{E}[\phi_{\bar{e}}(O)^2]$
where $\phi_{\bar{e}}$ is the efficient influence function (EIF) under the fixed propensity $\bar{e}$ .

This result implies that the statistical complexity of adaptive, balanced, or constrained designs collapses into a single object: the average treatment exposure allocated across covariate values. Operational constraints (e.g., budget limits) are incorporated by restricting the admissible set of average propensities.

2.3 Generalized Neyman Allocation

For vector-valued targets (e.g., multi-treatment effects), the paper derives a generalized Neyman allocation. The optimal design minimizes the trace of the covariance matrix of the EIF, leading to an allocation rule where treatment arms with higher conditional variance in the efficient influence function receive greater exposure. For linear functionals of outcome means, this reduces to the classical Neyman allocation rule.

3. Constructive Approaches to the Benchmark

The paper proposes two complementary, implementable batched adaptive designs that approach the theoretical benchmark. Both methods update the assignment policy only a logarithmic number of times ( $B \asymp \log n$ ), ensuring compatibility with delayed feedback and operational auditability.

3.1 Route 1: Batched Adaptive Regression Adjustment

This approach focuses on estimator-side correction.

Mechanism: The experiment is divided into batches. Within each batch, the assignment rule is fixed. At the end of each batch, nuisance parameters (outcome regressions and second-moment functions) are estimated using cross-fitting on accumulated data. The assignment rule for the next batch is updated via plug-in minimization of the oracle design objective.
General Estimands: For general smooth estimands, the method achieves the oracle benchmark provided the nuisance estimators converge at a rate faster than $n^{-1/4}$ . This is a known bottleneck in semiparametric estimation due to second-order plug-in bias.
Linear Functionals: For linear functionals of outcome means (e.g., Average Treatment Effects, Policy Values), the plug-in bias cancels exactly due to the structure of the EIF. Consequently, the $n^{-1/4}$ bottleneck vanishes. The paper establishes a sharp second-order convergence rate of $O(n^{-1-2\alpha} \vee n^{-1-2\beta})$ , where $\alpha$ and $\beta$ are the convergence rates of the outcome regression and second-moment estimators, respectively. A matching lower bound proves this rate is minimax-optimal.

3.2 Route 2: Batched Adaptive Covariate Balancing

This approach focuses on design-side control to bypass nuisance estimation bottlenecks.

Mechanism: This route uses a simple moment-based estimator (no regression adjustment in the final step). The design enforces local balance of covariate-indexed moment functions within each batch using spatial binning and grouped permutations. The target propensity is updated across batches based on estimated second-moment proxies.
Efficiency: By controlling the "balancing remainder" (the deviation from the target propensity) within batches, this method attains the semiparametric benchmark $V^*/n$ without requiring nonparametric nuisance estimation in the estimation step.
Trade-offs: The balancing remainder decays at a dimension-dependent rate $O(m^{-1/(d+4)})$ for a batch size $m$ . Thus, this route is most effective in lower-dimensional settings or when the effective dimension of the heterogeneity is low.

4. Key Results and Empirical Validation

4.1 Theoretical Results

Unified Framework: The induced average propensity score unifies previously disparate efficiency bounds for adaptive, stratified, and local-randomization designs under a single non-anticipating framework.
Resolution of Conjectures: The framework resolves a conjecture by Cytrynbaum (2021) regarding the efficiency bounds of two-stage experimental designs.
Minimax Sharpness: For linear functionals, the paper provides the first minimax characterization of the convergence rate to the semiparametric efficiency bound for general sequential experiments.

4.2 Numerical and Empirical Studies

The authors validate their methods through synthetic simulations and a real-world application:

Synthetic Experiments: Simulations with varying dimensions ( $d \in \{2,3,8\}$ ) and heteroskedasticity show that both adaptive routes significantly outperform uniform randomization. The regression adjustment (RA) route is robust to high dimensions, while the covariate balancing (CB) route performs best when the balancing partition aligns with the low-dimensional structure of the data (effective dimension).
AI Medical Assistant Evaluation: Using data from the HELPMed study (Bean et al., 2026), the authors evaluated multiple AI assistants against a baseline. The adaptive designs reduced the normalized mean squared error (MSE) by approximately 5–8% compared to uniform randomization. The study highlighted that using a data-adaptive, low-dimensional partition for the CB route significantly improved performance in high-dimensional settings.

5. Significance and Claims

The paper claims to provide a unified framework for characterizing and designing efficient sequential experiments. Its primary significance lies in:

Reframing Design: It shifts the focus of experimental design from optimizing complex, path-dependent assignment mechanisms to learning or choosing an efficient average propensity score.
Operational Feasibility: By demonstrating that the efficiency benchmark can be approached with only a logarithmic number of policy updates, the paper bridges the gap between theoretical optimality and operational constraints (delayed feedback, auditability, stability).
Practical Flexibility: The two complementary routes (regression adjustment and covariate balancing) offer practitioners trade-offs: RA is flexible for high-dimensional settings but requires fast nuisance estimation; CB is robust and simple but requires careful handling of the covariate space dimension.

The authors conclude that the average propensity score serves as a common language for understanding statistical limits in sequential experimentation, enabling the design of efficient multi-treatment experiments in modern applications like AI service evaluation.