📈 economics

Prior-Free Sample Size Design for Test-and-Roll Experiments

This paper proposes a prior-free Worst-case Marginal Benefit (WMB) rule for designing test-and-roll experiments that resolves the limitations of standard minimax regret by establishing a practical rule-of-thirds benchmark, suggesting an optimal sample size of approximately one-third of the total population for both Gaussian and Bernoulli outcomes.

Original authors: Kentaro Kawato, Shosei Sakaguchi

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kentaro Kawato, Shosei Sakaguchi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a company with a fixed number of employees, say N people. You have two new software tools, Tool A and Tool B, and you need to pick the one that makes everyone more productive. You don't know which one is better yet.

You have two choices for how to proceed:

The "Rollout" Approach: Just pick one tool and give it to everyone immediately. (Fast, but risky if you pick the wrong one).
The "Test-and-Roll" Approach: First, you run a small experiment on a few employees (let's call this number m). You give Tool A to half of them and Tool B to the other half. You see which one works better. Then, you give the winning tool to the rest of the employees.

The Big Dilemma: How many people should be in the experiment?

This is the core question of the paper. It's a classic tug-of-war between exploration (learning) and exploitation (using what you know).

If you test too few people (small m): You might get lucky and pick the right tool, but you also risk picking the wrong one. If you pick the wrong tool, you have to give it to the entire rest of the company, causing a huge loss in productivity.
If you test too many people (large m): You will almost certainly know which tool is better. However, during the test, half the people in the experiment are forced to use the worse tool. If you test 90% of your company, you are wasting the productivity of 45% of your workforce just to be sure.

The paper asks: What is the perfect number (m) to test so that the total happiness (welfare) of the whole company is maximized?

The Old Way: The "Scared" Manager

The authors first looked at how statisticians usually solve this using a method called "Absolute Minimax Regret." Think of this as a manager who is terrified of the worst possible scenario.

This manager thinks: "What if the two tools are almost identical, but one is just a tiny bit better? If I test too many people, I'm wasting time. But if I test too few, I might pick the wrong one!"

Because this method is so focused on the absolute worst-case scenario (where the tools are nearly identical and the cost of being wrong is high), it gets overly cautious. It tells the manager: "Don't test anyone! Just guess and roll out immediately."

The paper argues this is silly. In the real world, we know we need some data to make a good decision. The "scared" method leads to experiments that are too small to be useful.

The New Way: The "Marginal" Manager

The authors propose a new rule called the Worst-case Marginal Benefit (WMB) rule.

Instead of asking, "What is the worst thing that could happen to the whole project?", this manager asks a simpler, step-by-step question:
"Is it worth adding one more pair of people to the experiment?"

The Cost: Adding two more people means two more people have to use the potentially worse tool during the test.
The Benefit: Adding two more people gives us slightly more information, which lowers the chance that we will make a mistake when we roll out the tool to the rest of the company.

The manager keeps adding people to the test as long as the Benefit (saving the future rollout from a mistake) is greater than the Cost (wasting the current test subjects). As soon as the cost of adding one more pair outweighs the benefit, they stop.

The Magic Number: The "Rule of Thirds"

After doing some heavy math (which the paper details with complex formulas and Gaussian approximations), the authors found a surprisingly simple answer.

Regardless of whether the data is "Yes/No" (like a coin flip) or "Numbers" (like test scores), the optimal strategy is almost always:

Test 1/3 of your population, and Roll out to the remaining 2/3.

If you have 300 people: Test 100. Roll out to 200.
If you have 3,000 people: Test 1,000. Roll out to 2,000.

Why is this important?

No Guessing Required: You don't need to guess how good the tools are beforehand (no "priors"). You just need to know how many people you have in total.
It's Robust: It works even if you don't know the exact details of the problem, as long as you aren't in a weird, extreme edge case (like a tool that never works).
It Balances the Scale: It stops the "scared" manager from testing too little and the "obsessive" manager from testing too much.

Summary

The paper says that when you have a fixed group of people and need to choose between two options, the best way to balance learning and doing is to spend one-third of your group on the experiment and spend the other two-thirds on the final decision. This rule is simple, doesn't require complex guesses, and protects the group from both wasted effort and bad decisions.

Technical Summary: Prior-Free Sample Size Design for Test-and-Roll Experiments

Problem Formulation
This paper addresses the sample-size design problem for "test-and-roll" experiments conducted within a finite population of size $N$ . In this setting, a decision-maker first conducts a randomized controlled trial (RCT) on a subset of $m$ units (with $m/2$ assigned to treatment and $m/2$ to control). Based on the observed outcomes, the decision-maker assigns the remaining $N-m$ units to the treatment arm that performed better in the experiment.

The objective is to maximize the expected total welfare over the entire population, $E[\sum_{i=1}^N Y_i]$ . This creates a fundamental exploration–exploitation tradeoff:

Exploration Cost: Increasing $m$ improves the accuracy of identifying the superior treatment but imposes welfare losses on the experimental units, as half are inevitably assigned to the inferior treatment ex post.
Exploitation Risk: Decreasing $m$ reduces the immediate welfare loss on experimental units but increases the risk of making an erroneous rollout decision, thereby assigning the inferior treatment to the larger remaining population ( $N-m$ ).

The central challenge is that true treatment effects are unknown ex ante. The paper seeks a design rule that is prior-free, avoiding the sensitivity to subjective Bayesian priors or arbitrary hyperparameters that characterizes existing welfare-based approaches.

Methodology and Critique of Existing Criteria
The authors first analyze the standard Absolute Minimax Regret criterion, which selects the sample size $m$ to minimize the worst-case regret relative to an oracle that knows the true treatment effects.

Limitation: The paper demonstrates that this criterion leads to implausibly small experimental sample sizes (e.g., $m/N \approx 2.3\%$ for $N=10,000$ ).
Mechanism of Failure: The worst-case adversary in the absolute minimax framework concentrates on parameter values that maximize the absolute treatment effect ( $|\tau|$ ). In such extreme cases, the exploration cost ( $m/2 \cdot |\tau|$ ) dominates the regret function, causing the criterion to over-penalize exploration and suppress the sample size to near-zero.

To resolve this, the authors propose the Worst-case Marginal Benefit (WMB) rule.

Core Concept: Instead of minimizing total worst-case regret, the WMB rule evaluates the tradeoff at the margin. It compares the worst-case marginal benefit of adding one more matched pair to the experiment against the corresponding marginal exploration cost.
Stopping Rule: Experimentation continues as long as the worst-case marginal benefit of an additional pair exceeds the marginal cost. The optimal sample size $\hat{m}_{WMB}$ is the smallest even $m$ where the worst-case marginal benefit ratio $\eta(m, \mu_1, \mu_0)$ drops to or below 1.
Definition:
$\hat{m}_{WMB} = \min \left\{ m \in \{0, 2, \dots, N-2\} : \max_{(\mu_1, \mu_0) \in [0,1]^2} \eta(m, \mu_1, \mu_0) \leq 1 \right\}$
where $\eta$ measures the reduction in expected rollout errors (saved units) per unit of added exploration cost.

Key Results

The Rule-of-Thirds Benchmark:
The paper establishes a simple, prior-free benchmark for the optimal sample size:
$\hat{m} \approx \frac{N}{3}$
This result holds under two distinct conditions:
- Bernoulli Outcomes: Through a Gaussian approximation, the WMB criterion yields $\hat{m} \approx N/3$ . The authors rigorously show that the exact Bernoulli problem requires excluding a "boundary rare-event pathology" (where one arm has zero variance and the other has a vanishing success probability of order $O(1/N)$ ). In this pathological case, the optimal sample size approaches $N/2$ . However, for regular interior states (where treatment effects are non-degenerate), the least favorable parameters localize near the diagonal ( $\mu_1 \approx \mu_0$ ), and the Gaussian approximation becomes valid, leading to the $N/3$ rule.
- Gaussian Outcomes (Known Variance): For outcomes following a Gaussian distribution with a known common variance, the $N/3$ benchmark arises exactly without asymptotic approximation or the need to exclude boundary cases. The continuous relaxation of the marginal ratio function yields a closed-form solution where the worst-case marginal benefit is $\leq 1$ if and only if $m \geq N/3$ .
Behavior of the Adversary:
Under the WMB rule, the worst-case adversary behaves differently than under absolute minimax regret. Instead of choosing extreme treatment effects to maximize total regret, the adversary chooses parameter values that maximize the marginal benefit of continued experimentation. In the interior regime, these least favorable states concentrate near the diagonal ( $\mu_1 \approx \mu_0$ ), where distinguishing between treatments is most difficult.
Numerical Validation:
Numerical studies confirm that for small $N$ , boundary effects can push the sample size toward $N/2$ . As $N$ increases, the least favorable parameters move away from the boundary, and the optimal sample size stabilizes near $N/3$ .

Significance and Contribution
The paper claims to provide a practically implementable, prior-free guide for welfare-based sample-size design.

Practicality: The resulting rule ( $m \approx N/3$ ) depends only on the total population size $N$ . It does not require the decision-maker to specify subjective priors, prior variances, or other tuning parameters, which are often difficult to justify in practice.
Theoretical Robustness: By shifting from a total-regret minimization to a marginal-benefit analysis, the paper avoids the "over-penalization" of exploration inherent in standard minimax regret criteria.
Welfare Focus: Unlike conventional power analysis (which controls Type I/II errors for hypothesis testing), this approach directly optimizes the welfare objective of the finite population, explicitly balancing the cost of experimentation against the risk of a suboptimal rollout.

The authors conclude that the WMB rule offers a robust solution to the exploration–exploitation tradeoff in finite-population test-and-roll experiments, providing a clear, theoretically grounded stopping rule that is independent of subjective prior beliefs.