📊 statistics

A closed-form sample size correction for always-valid inference with optional stopping

This paper introduces a closed-form sample size correction factor that adjusts fixed-sample calculations for always-valid sequential tests, enabling accurate power estimation without simulations and reducing the required sample budget by 8–20% compared to conservative last-point rules.

Original authors: Mårten Schultzberg

Published 2026-06-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mårten Schultzberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a case (an A/B test) to see if a new clue (a treatment) is better than the old one. You want to be sure you don't make a mistake, but you also want to stop as soon as you have enough evidence.

The Problem: The "Last-Stop" Trap

In the past, detectives used a rule called the "Last-Stop Rule." They would say, "I will look at the evidence every day, but I will only officially declare a winner on Day 100. I need to make sure that on Day 100 specifically, I have a 95% chance of catching the criminal if they are there."

To be safe, they would calculate how many days they needed to reach that 95% chance. But here's the catch: because they were allowed to peek at the evidence every single day before Day 100, the criminal could have been caught on Day 10, Day 50, or Day 99. The "Last-Stop Rule" ignored all those earlier chances.

Because it ignored the early chances, the rule was being overly cautious. It told the detective, "You need 100 days to be safe," when in reality, they might have been safe with only 85 days. This meant wasting time, money, and resources on a longer investigation than necessary.

The Solution: The "Magic Multiplier"

The author of this paper, Mårten Schultzberg, has found a closed-form correction factor (a fancy math term for a simple "magic number" you can calculate without running complex computer simulations).

Think of this as a magic multiplier (called $k^*$ ). Instead of guessing how long to run the experiment, you take your standard "fixed" calculation and multiply it by this magic number.

Old Way: "We need 1,000 people to be safe." (This is actually too many).
New Way: "We need 1,000 people, but multiply that by 0.85. So, we only need 850 people."

This new method accounts for the fact that you are checking the evidence continuously. It realizes that because you are looking all the time, you don't need to wait as long to be sure.

How It Works (The Metaphor)

Imagine the evidence is a ball rolling up a hill.

The Boundary is a fence at the top of the hill. If the ball crosses the fence, you stop and declare victory.
The Old Rule assumed the ball only needed to cross the fence at the very end of the track. To be safe, they built a very long track.
The New Method realizes the ball could cross the fence anywhere along the track.
The author uses a clever trick: they draw a straight line (a tangent) that just touches the top of the fence at the end point. Because the fence is curved (it gets harder to cross as you go), this straight line sits slightly above the actual fence.
By calculating the odds of the ball crossing this straight line instead of the curved fence, the math becomes simple and solvable with a calculator. This straight-line approximation is so accurate that it tells you exactly how much shorter the track can be.

The Results: Saving Money and Time

The paper tested this "magic multiplier" against three different types of fences (statistical boundaries) used by companies like Spotify.

The Savings: By using this new formula, companies can save 8% to 20% of the sample size they would have otherwise collected.
- Analogy: If you were planning to bake a cake for 100 people, this new method tells you, "Actually, you only need ingredients for 85 people, and you'll still be 95% sure the cake is good."
Accuracy: When they ran thousands of computer simulations, the new method hit the target success rate almost perfectly (within about 3 percentage points).
Real-World Proof: The author tested this on real data from Spotify's experimentation platform. Across 713 different tests, the method saved a median of 9.5% in sample size. That's a lot of saved money and time.

Important Limits

The paper notes that this "magic number" works best when the data behaves nicely (like a bell curve) and when you have a reasonable amount of data to start with (called "burn-in"). If you are testing something very rare (like a disease that happens to 1 in 10,000 people), the math gets a bit shaky, and the savings might not be as reliable.

Summary

In short, this paper gives experimenters a simple, pre-calculated "discount code" for their sample sizes. It stops them from over-preparing for experiments that allow continuous monitoring, saving them roughly 10–20% of their budget without sacrificing the safety of their conclusions.

Technical Summary: A Closed-Form Sample Size Correction for Always-Valid Inference with Optional Stopping

Problem Statement
Sequential tests that allow continuous monitoring (always-valid inference) are widely deployed in A/B experimentation platforms to enable early stopping when evidence is clear. However, determining the appropriate sample size for these tests is computationally expensive. Traditional power calculations for sequential tests require extensive simulations that do not scale well across the thousands of metrics monitored on large experimentation platforms.

A common heuristic, the "last-point rule," attempts to size these experiments by inflating the fixed-sample size ( $n_z$ ) until the marginal rejection probability at the planned endpoint reaches the target power ( $1-\beta$ ). The paper identifies a critical flaw in this approach: always-valid (AV) power is defined as the probability of a boundary crossing at any time during the run, not just at the endpoint. Consequently, the last-point rule is overly conservative. At standard operating characteristics ( $\alpha=0.05, 1-\beta=0.80$ ), this rule results in empirical power between 0.86 and 0.88, effectively oversizing experiments by 7–9 percentage points.

Methodology
The authors propose a closed-form correction factor, $k^*(\alpha, \beta, t_0)$ , to adjust the fixed-sample size $n_z$ such that the total sample size $n = k^* \cdot n_z$ approximately achieves the target AV power without simulation.

The methodology relies on the following components:

Rescaled Time and Brownian Motion Approximation: The problem is mapped to a rescaled time variable $t = n/n_z$ . Under the alternative hypothesis ( $H_1$ ), the standardized cumulative process $Y_t$ is approximated as a Brownian motion with drift $\mu_d = z_\alpha + z_\beta$ and unit variance rate.
Tangent Linearization: The core of the approximation involves replacing the curved sequential boundary $b(t)$ with its tangent line at the planned endpoint $t=k$ . This linear surrogate allows for the application of the Bachelier first-passage formula.
Analytical Derivation: The authors derive the probability of crossing the linearized boundary over the interval $[t_0, k]$ $[t_{0}, k]$ , where $t_0 = m/n_z$ $t_{0} = m / n_{z}$ is the burn-in fraction (the point where monitoring begins). The derivation integrates over the initial value of the process at $t_0$ $t_{0}$ using Gaussian completion-of-the-square and the bivariate normal cumulative distribution function (CDF), $\Phi_2$ $Φ_{2}$ .
- The resulting power approximation $\pi_{closed}(k)$ consists of three terms: the immediate rejection probability at $t_0$ , the probability of crossing the linear boundary during $(t_0, k]$ , and a reflection correction term.
Boundary Specificity: The correction factor depends on the specific sequential boundary only through its value $b(k)$ $b (k)$ and slope $b'(k)$ $b^{'} (k)$ at the endpoint. The paper explicitly derives these for three common boundaries:
- The Waudby-Smith et al. [2023] (WSKR) confidence sequence.
- The Maharaj et al. [2023] confidence sequence.
- The Johari et al. [2022] mixture sequential probability ratio test (mSPRT).

Key Contributions

Closed-Form Correction Factor: The paper provides an explicit formula for $k^*$ that can be evaluated using elementary functions and the bivariate normal CDF, eliminating the need for simulation-based power calculations.
Generalizability: The approximation is applicable to any smooth, concave boundary, provided the value and slope at the endpoint are known.
Burn-in Sensitivity: The analysis accounts for the burn-in parameter $m$ , showing how the correction factor varies with the start of the monitoring window.
Empirical Validation: The authors validate the correction across three distinct boundary families and a wide grid of $(\alpha, \beta)$ parameters using Monte Carlo simulations with 50,000 replications per cell.

Results

Accuracy: Setting the sample size to $k^* \cdot n_z$ hits the target empirical power within approximately 3 percentage points of the target across the tested grid. In Gaussian simulations, the empirical power is often slightly above $1-\beta$ , a conservative bias attributed to the tangent line lying above the concave boundary.
Efficiency Gains: The correction factor yields significant sample size savings compared to the last-point rule. Across the operating range, the savings range from 8% to 20%.
- For example, at $\alpha=0.05$ and $1-\beta=0.80$ , the savings are approximately 10–14% depending on the boundary type.
Production Validation: The authors applied the correction to 713 metrics from Spotify's experimentation platform (using the Maharaj boundary). The median sample size reduction observed was 9.5%, with a mean of 9.8%.
Burn-in Impact: The paper notes that while realistic burn-in values ( $m \ll n_z$ ) increase the total sample budget (often 2–3 times the fixed-sample size) due to wider confidence sequences, the correction factor still provides substantial savings relative to the uncorrected last-point rule.
Limitations: The authors note that at very low base rates (e.g., $p_C \le 0.01$ ), the Central Limit Theorem (CLT) approximation degrades, and the closed-form factor may undershoot the target power.

Significance
The paper claims that the last-point rule, while simple, is inefficient for sequential testing, leading to unnecessary resource expenditure. By providing a closed-form correction factor, the authors offer a scalable solution for experimentation platforms to size always-valid tests accurately. This allows platforms to reduce sample sizes by 8–20% without compromising statistical validity, thereby accelerating decision-making and reducing the cost of experimentation. The work bridges the gap between theoretical sequential analysis and practical, large-scale implementation in industry settings.