📊 statistics

Bayesian Inference Procedures for A/B Testing: An Overview

This paper provides a systematic three-tier hierarchy of Bayesian A/B testing configurations, demonstrating that while many commercial platforms use uncalibrated methods, proper selection of priors and stopping rules—specifically Bayes factor stopping and empirical Bayes—is essential to control distinct risks like false positive rates, estimation error, and regret.

Original authors: Mårten Schultzberg, Mattias Frånberg

Published 2026-08-14

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mårten Schultzberg, Mattias Frånberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: "Did this new clue actually solve the case, or was it just a lucky guess?" In the world of science and business, this is called A/B testing. It's a way to compare two versions of something (like a red button vs. a blue button) to see which one works better. But here's the tricky part: if you keep checking the results while the experiment is running, you might get fooled by random noise. It's like flipping a coin and stopping the moment you see three heads in a row; you might think the coin is magic, but it's just luck. This is the danger of "peeking."

To solve this, scientists use Bayesian inference, a method that updates beliefs as new data arrives, kind of like how you update your opinion about a movie as you watch more scenes. However, there isn't just one way to do this. Some methods are like guessing based on a hunch, while others are like using a strict rulebook to avoid false alarms. The big question is: which method actually keeps you safe from making bad decisions, and which one just feels safe? This paper dives into the messy reality of these different methods to figure out which ones are actually reliable and which ones are just fancy ways of getting tricked by chance.

The Three Tiers of Truth

The authors, Mårten Schultzberg and Mattias Frånberg from Spotify, realized that people often talk about "Bayesian A/B testing" as if it were a single, magic tool. They discovered it's actually a family of different tools with very different superpowers (and weaknesses). To make sense of this, they organized these tools into a three-tier hierarchy, like a video game with three levels of difficulty and safety.

Level 1: The "Hunch" Tier (Posterior Coherence)

Imagine you are playing a game where you just want to feel confident about your guess. You look at the data and say, "I'm 95% sure the blue button is better!" This is Level 1. It's mathematically consistent, meaning your belief updates make sense internally. But here's the catch: it offers zero protection against false alarms if you keep checking the results.

The paper shows that if you use a "flat" prior (basically starting with no opinion) and just stop whenever you feel 95% sure, you are doing the exact same thing as a "simple peeker." It's like checking your coin flip every second and stopping the moment you see three heads. The paper's simulations confirm that this method has a false positive rate of about 30% (way higher than the 5% people usually want). It feels good, but it's dangerous. Even if you try to use a "decision-theoretic" approach (calculating the cost of a mistake), if you don't have a strict rule for stopping, you might still end up shipping a bad feature half the time.

Level 2: The "Guarded" Tier (Bayes Factor Stopping)

Now, let's add a security guard. Level 2 uses something called a Bayes Factor. Think of this as a score that compares how much the data supports "Version B is better" versus "Version B is the same as A."

The paper finds that if you use a proper prior (a reasonable starting guess) and stop only when this score crosses a specific line, you get a guarantee. No matter how many times you peek, the chance of a false alarm stays below your limit (usually 5%). This is a huge deal. The authors show that for many real-world business costs, this method is almost the best possible way to decide when to stop. It's like having a bouncer who checks your ID every time you try to enter, ensuring that only the truly lucky (or truly skilled) get in.

Interestingly, the paper argues that the choice between "Bayesian" and "Frequentist" (the other main statistical school of thought) often doesn't matter as much as people think. If you set up a Bayesian test with a specific cost model, it often ends up looking exactly like a Frequentist test. The math is different, but the outcome is often the same.

Level 3: The "Master" Tier (Empirical Bayes)

This is the VIP section. Level 3 adds a secret weapon: history. Instead of guessing what a "reasonable" starting point is, this method looks at hundreds of past experiments from the same company to learn the truth about how big effects usually are.

Imagine you are a chef. In Level 2, you guess how much salt to add based on a recipe book. In Level 3, you look at a logbook of every dish you've cooked in the last year to see exactly how much salt the customers actually liked. This is called an Empirical Bayes prior.

The paper shows that when you use this history-based prior, you get two amazing benefits:

Automatic Correction: It naturally fixes the problem of checking too many things at once (multiplicity). If you test 10 buttons, you don't need to do extra math to avoid false alarms; the history does the work for you.
Calibrated Shrinkage: It pulls extreme results back to reality. If a test shows a massive 50% improvement, but your history says improvements are usually tiny, this method says, "That's probably a fluke," and pulls the estimate down. This prevents the "Winner's Curse," where you celebrate a huge win that was actually just noise.

But there's a trap. The paper warns that this Level 3 magic only works if your history is honest and representative.

If you only look at your past wins (ignoring the failures), your history is broken, and the method fails.
If you mix data from two different types of experiments (like mixing cake recipes with car engine tests), the history gets confused, and the protection disappears.
You need a decent amount of history (around 200 past experiments) for this to work reliably.

The Verdict: It's About the Risk, Not the Label

The authors ran simulations to test these tiers against each other and against standard Frequentist methods. Here is what they found:

Level 1 (Flat Prior + Peeking): It's a disaster for error control. It produces false positives about 30% of the time, just like simple peeking.
Level 2 (Bayes Factor): It keeps false positives low (under 5%) and is very flexible. It doesn't require you to decide in advance how long the experiment will run.
Level 3 (Empirical Bayes): When the history is good, it produces the most accurate estimates (lowest error) of all methods. It shrinks wild guesses back to reality better than anyone else.
The "Expected Loss" Trap: Some people try to stop based on "how much money we might lose." The paper shows this only works if shipping a bad feature is free. If there is any cost to shipping a bad feature (like fixing code or annoying users), this method starts shipping bad features way too often.

The paper concludes that the "best" method isn't about being Bayesian or Frequentist. It's about what risk you are trying to control.

If you need to be absolutely sure you aren't making false claims, you need Level 2 or 3.
If you have a lot of history and want the most accurate numbers, Level 3 is the king.
If you don't have enough history, stick to Level 2 and use strict rules to avoid false alarms.

In the end, the paper suggests that the most important step isn't picking a fancy statistical name. It's asking: "What happens if I'm wrong?" If the cost of being wrong is high, you need the guardrails of Level 2 or 3. If you treat the method like a magic wand without understanding the rules, you might end up celebrating a victory that was just a lucky guess.

Technical Summary: Bayesian Inference Procedures for A/B Testing

1. Problem Statement

Bayesian inference for A/B testing is frequently treated in literature and practice as a monolithic method. However, it is actually a family of configurations defined by different combinations of priors and stopping rules, possessing fundamentally different statistical properties. This conflation leads to misleading claims where properties true for specific configurations (e.g., error rate control) are incorrectly attributed to the framework as a whole. Furthermore, many commercial experimentation platforms operate at a "lowest tier" of statistical guarantees by default, often failing to control false positive rates (FPR) under continuous monitoring or to calibrate effect estimates. There is a lack of a systematic overview organizing these configurations, their guarantees, and their relationship to frequentist methods.

2. Methodology and Framework

The paper organizes common Bayesian configurations into a three-tier guarantee hierarchy, ordered by the strength of their error-rate control. The analysis relies on a standard two-group normal model where the average treatment effect $\delta = \mu_T - \mu_C$ is estimated from sequential data.

The Three-Tier Hierarchy

Tier 1: Posterior Coherence (No Error Control)

Definition: Any procedure producing a proper posterior. The posterior represents a valid belief update regardless of the stopping rule (Likelihood Principle).
Mechanisms: Includes posterior-probability stopping (e.g., $P(B > A) > \gamma$ ) and expected-loss stopping (minimizing expected regret).
Limitations: These configurations provide no frequentist error-rate guarantees. Under a flat prior, posterior-probability stopping is algebraically identical to "simple peeking" (stopping when a p-value < 0.05), resulting in inflated Type I error rates. Expected-loss stopping optimizes for a specific loss function but does not bound FPR.

Tier 2: Bounded FPR via Bayes Factor (BF) Stopping

Definition: Achieves unconditional control of the False Positive Rate ( $FPR \le \alpha$ ) even under optional (continuous) stopping.
Mechanism: Combines a proper prior with a Bayes Factor thresholding rule ( $BF_{10} > 1/\alpha$ ).
Theoretical Basis: Under the null hypothesis, the sequence of Bayes Factors forms a non-negative martingale. By Ville's inequality, the probability of the BF ever exceeding $1/\alpha$ is bounded by $\alpha$ , regardless of when the experiment stops.
Optimality: For cost functions with terminal payoffs affine in the posterior, BF stopping is near-optimal for sequential decision-making. The choice between a decision-theoretic formulation and a frequentist formulation is largely a matter of parameterization.

Tier 3: Bounded FDR and Calibrated Shrinkage via Empirical Bayes (EB)

Definition: Adds control over the False Discovery Rate (FDR) across multiple metrics and provides calibrated shrinkage of effect estimates.
Mechanism: Requires an Empirical Bayes prior estimated from a corpus of historical experiments. This prior is typically a two-component mixture: a point mass at zero (representing null effects) and a normal distribution (representing non-null effects).
Guarantees:
- FDR Control: The mixture prior encodes the historical non-null rate ( $p$ ). When combined with BF stopping, the $(1-\hat{p})$ floor in the posterior odds bounds the Bayesian FDR without requiring explicit multiplicity corrections (e.g., Bonferroni).
- Calibrated Shrinkage: The posterior mean is shrunk toward zero based on the program's actual effect-size distribution, correcting for the "winner's curse" (Type M error) and reducing estimation variance.
Constraints: Tier 3 guarantees depend entirely on the quality and representativeness of the historical corpus. Failure modes include "winner-selected" corpora (excluding nulls), pooled programs with heterogeneous effect distributions, or insufficient corpus size (stabilization requires $\approx 200$ historical experiments).

3. Key Contributions

Systematic Taxonomy: The paper provides the first systematic overview organizing Bayesian A/B testing configurations into a three-tier hierarchy based on error-rate guarantees, clarifying that "Bayesian" is not a single method but a spectrum of statistical properties.
Equivalence of Frameworks: It demonstrates that for many practical configurations, the distinction between Bayesian and frequentist approaches is superficial.
- Flat-prior posterior inference is numerically identical to frequentist z-tests.
- The Bayes Factor is identical to the mixture Sequential Probability Ratio Test (mSPRT) statistic.
- Cost-motivated stopping rules often converge to BF thresholds, linking decision-theoretic and frequentist error rates.
Empirical Validation of Risks: Through simulations, the paper quantifies the risks of different configurations:
- Tier 1 (Flat Prior + Posterior Stopping): Reproduces simple peeking with an FPR of $\approx 0.30$ (for $\alpha=0.05$ ).
- Tier 2 (BF Stopping): Controls FPR strictly below $\alpha$ but may have lower power than Group Sequential Tests (GST) for marginal signals due to logarithmically increasing thresholds.
- Tier 3 (EB + BF): Achieves the lowest Mean Squared Error (MSE) among all configurations due to calibrated shrinkage, but this advantage vanishes if the prior is misspecified (e.g., from a winner-selected corpus).
Decision-Theoretic Trade-offs: The paper analyzes the "regret" of different methods. Expected-loss stopping minimizes regret only when the cost of shipping a null-effect variant is effectively zero. Once shipping incurs any non-metric cost (maintenance, complexity), methods requiring statistical evidence (BF stopping) achieve lower total regret.

4. Simulation Results

The authors present four simulation settings to validate the hierarchy:

Setting A (FPR Control): Confirms that flat-prior posterior stopping fails to control FPR (FPR $\approx 0.30$ ), while BF stopping with proper priors maintains FPR $\le 0.05$ .
Setting B (Sequential Comparison): Compares BF stopping against frequentist GST and always-valid confidence sequences (WS-R).
- All valid sequential methods control FPR.
- GST achieves the highest power but requires a pre-committed sample size ( $n_{max}$ ).
- BF stopping offers flexibility (no $n_{max}$ ) and, when paired with an Oracle EB prior, achieves the lowest MSE.
- Misspecified EB priors (e.g., from winner-selected data) lose the MSE advantage and can inflate error rates.
Setting C (Corpus Quality): Demonstrates that Tier 3 properties (FDR control, shrinkage) are highly sensitive to corpus quality. A corpus of $\approx 200$ experiments is needed for stability. "Winner-selected" or "pooled" corpora lead to persistent FPR/FDR inflation that does not converge with more data.
Setting D (Regret Analysis): Shows that expected-loss stopping is optimal only when shipping costs are zero. As shipping costs increase, BF stopping (even with loose calibration) becomes superior in minimizing expected regret.

5. Significance and Claims

The paper argues that the appropriate method for an experimentation program should be dictated by the risks the program needs to control, not by a preference for a specific statistical framework.

Risk-Based Configuration:
- If the goal is to minimize expected cost per decision and shipping nulls is cheap, decision-theoretic (expected-loss) rules may be appropriate.
- If the goal is to bound the False Positive Rate or False Discovery Rate across a program, Tier 2 (BF stopping) or Tier 3 (EB + BF) is required.
The Necessity of Calibration: The paper emphasizes that the benefits often attributed to "Bayesian methods" (automatic multiplicity correction, shrinkage, low estimation error) are not inherent to the use of a posterior. They are specific to Tier 3, requiring a calibrated Empirical Bayes prior derived from a representative historical corpus. Without this, a Bayesian configuration offers no more protection against multiplicity than a frequentist one.
Practical Guidance: Many platforms currently operate at Tier 1, inadvertently replicating frequentist peeking errors. The authors suggest that programs without a large, representative historical corpus should operate at Tier 2 (using BF stopping with a weakly informative prior) and apply explicit multiplicity corrections, rather than attempting Tier 3 with an underfitted prior.

In conclusion, the paper asserts that the choice of configuration (prior and stopping rule) is more critical than the choice of framework (Bayesian vs. Frequentist). The "Bayesian" label alone does not guarantee error control; only specific configurations within the hierarchy provide the rigorous guarantees necessary for large-scale experimentation.