📊 statistics

Nobody Puts Bonferroni in a Corner

This paper argues that the Bonferroni correction is an underappreciated yet superior choice for online experimentation due to its simplicity, tractability for sample size calculations, and ability to control false positives effectively when applied only to success metrics, thereby minimizing power loss compared to more sophisticated methods.

Original authors: Mårten Schultzberg

Published 2026-04-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mårten Schultzberg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive ship (Spotify) with a crew of thousands. Every day, you want to test new ideas: maybe a new song recommendation algorithm, a different button color, or a new way to organize playlists. These are your experiments.

To decide if an idea is good, you look at the data. But here's the problem: The more things you look at, the more likely you are to see a "ghost."

The Problem: The "Ghost" in the Machine

If you flip a coin once, getting "Heads" is normal. If you flip it 100 times, you will almost certainly get "Heads" 10 times in a row by pure luck.

In data science, this is called the Multiple Testing Problem. If you test 20 different metrics (like "time spent listening," "skip rate," "new followers") on one experiment, and you use a standard rule to say "this is a success," you will accidentally declare about 1 of them a success just by random chance.

If you keep shipping these "ghost" successes, you'll eventually build features that do nothing but waste engineering time and confuse your users. You lose trust in your data.

The Old Solution: The "Brute Force" Lockdown

For years, statisticians have said: "To stop the ghosts, you must be incredibly strict." The most famous strict rule is called Bonferroni.

Think of Bonferroni like a super-strict security guard at a club.

The Rule: If you want to get in (declare a success), you need a VIP pass. But because there are 20 people trying to get in, the guard divides the number of VIP passes by 20. Now, you need a pass that is 20 times harder to get.
The Result: It's very safe. Almost no ghosts get in. But it's also very hard for real stars (good ideas) to get in. Many good ideas get rejected because the bar is set impossibly high.

Because of this, many companies started using "smarter" guards (like Holm or Hommel) who are slightly more lenient, or even "FDR" guards who say, "It's okay if a few ghosts get in, as long as most people are real."

The Paper's Big Idea: "Nobody Puts Bonferroni in a Corner"

Mårten Schultzberg, the author, argues that everyone has been judging Bonferroni unfairly. He says Bonferroni is actually the best guard for this specific job, but only if you understand how the club actually works.

Here are the four reasons why, explained simply:

1. The "Guardrail" Misunderstanding (The Most Important Point)

Most people think Bonferroni is too strict because they count every single metric against the guard.

Success Metrics: These are the things you want to go up (e.g., "Did people listen more?"). If these go up, you ship the feature.
Guardrail Metrics: These are the things you don't want to go down (e.g., "Did the app crash?" or "Did people get annoyed?"). You only ship if these stay safe.

The Analogy: Imagine you are hiring a new employee.

Success Metric: "Can they code?" (You need this to be YES).
Guardrail Metric: "Do they have a criminal record?" (You need this to be NO).

If the candidate has a criminal record, you reject them. But if you accidentally think they have a criminal record when they don't (a false alarm), you just don't hire them. You don't hire a criminal! A false alarm on a guardrail metric just makes you more cautious; it doesn't make you hire a bad person.

The Paper's Insight: You only need to be super-strict about the Success Metrics. The Guardrails don't need the same strict math because a "false alarm" on a guardrail just stops a good idea, it doesn't ship a bad one.

Old Way: Count 10 Success Metrics + 10 Guardrails = 20 things to check. Bonferroni divides your strictness by 20. (Too strict!)
New Way: Count only the 10 Success Metrics. Bonferroni divides by 10. (Much fairer!)

When you do this, Bonferroni becomes much less "punishing" and much more practical.

2. The "Map" vs. The "Compass" (Confidence Intervals)

When you find a "winner," you need to know how good it is.

Bonferroni gives you a clear, honest map. It says, "We are 95% sure the improvement is between 1% and 5%." This map is valid for every metric, even the ones that didn't win.
The "Smarter" Methods (Holm/Hommel) are like a compass that only works if you are already standing on the treasure island. They can tell you the direction for the winners, but they can't give you a reliable map for the losers. If you try to use them for everything, the map gets blurry and confusing.

For a company like Spotify, having a clear, honest map for everything is more valuable than a slightly sharper compass for just the winners.

3. Planning the Trip (Sample Size)

Before you run an experiment, you need to know how many users to test.

Bonferroni is like a simple calculator. You plug in the numbers, and it tells you exactly how long to run the test.
The "Smarter" Methods are like a complex weather simulation. To use them, you have to guess how many "ghosts" are in the sky before you even start. Since you can't know that in advance, companies often just ignore the math and guess. This leads to running tests that are too short or too long.

4. The "Real World" Test

The author ran a simulation and looked at 1,296 real experiments from Spotify.

The Result: When they used the "Guardrail" logic (ignoring guardrails in the strict count), the "smarter" methods (Holm/Hommel) only shipped about 4-5% more experiments than Bonferroni.
The Catch: In the real world, most experiments are "null" (nothing happens). When nothing happens, the "smarter" methods don't actually find more winners; they just lower the bar a tiny bit.
The Trade-off: Is shipping 4% more experiments worth losing the clear, honest map (Confidence Intervals) and the easy planning? The author says: Probably not.

The Conclusion

The paper isn't saying "Stop using corrections." It's saying: "Stop being scared of Bonferroni."

If you organize your experiments correctly (separating "Success" from "Guardrails"), Bonferroni is:

Simple: Easy to explain to your boss.
Honest: It gives you a clear map for every result.
Fair: It's not actually that strict once you stop counting the guardrails.

The "smarter" methods sound fancy, but they are like a Ferrari that gets stuck in traffic. Bonferroni is a reliable Toyota Camry that gets you to the destination every time, with a clear view of the road.

In short: Don't let the fear of being "too strict" make you ship bad features. Use the simple, honest tool, but make sure you're only being strict about the things that actually matter.

1. Problem Statement

In online experimentation (A/B testing), practitioners often test multiple hypotheses simultaneously. This inflates the Family-Wise Error Rate (FWER), leading to a high probability of false positives (Type I errors).

The Status Quo: The industry often views the Bonferroni correction as overly conservative and "naive," preferring more sophisticated methods like Holm, Hommel, or Benjamini-Hochberg (BH) (which controls the False Discovery Rate, FDR).
The Core Issue: This preference often ignores the specific structural constraints of product experimentation:
1. Decision Logic: Product decisions rely on a mix of "success" metrics (need improvement) and "guardrail" metrics (must not degrade).
2. Inference Needs: Practitioners require unconditional simultaneous confidence intervals (CIs) for all metrics to assess effect sizes and uncertainty, not just binary reject/fail decisions.
3. Planning: Pre-experiment sample size calculations are difficult with complex methods.
4. Reality of Nulls: Unlike genomics (where many true effects exist), product experimentation has a high base rate of true nulls (approx. 70% of changes have no effect). In this regime, FDR methods (like BH) lower the bar for shipping, increasing false positives without necessarily capturing more true effects.

2. Methodology

The author argues for the re-evaluation of Bonferroni based on four pillars, supported by theoretical analysis, simulation studies, and empirical data from Spotify's experimentation platform, Confidence.

A. Theoretical Framework

Decision Framework Re-architecture: The paper proposes a specific decision logic where metrics are categorized:
- Success Metrics: Union-Intersection (UI) logic. At least one must be significant to ship. These require FWER control.
- Guardrail/Quality Metrics: Intersection-Union (IU) logic. All must pass (non-inferiority). A false positive here (thinking a metric is fine when it isn't) makes shipping easier, not harder. Therefore, they do not inflate the FWER of the shipping decision.
- Implication: The Bonferroni denominator should be the count of Success Metrics ( $S$ ) only, not the total number of metrics ( $m$ ).
Confidence Interval Coherence: Bonferroni allows for the construction of honest, unconditional simultaneous CIs for every metric using a simple $\alpha/S$ adjustment. Other methods (Holm, Hommel, BH) either lack simultaneous CIs for rejected hypotheses or require complex, conditional frameworks (False Coverage Rate) that are rarely implemented in practice.
Sequential Testing: Bonferroni allows each metric to run its own Group Sequential Test (GST) with independent spending functions, avoiding the complexity of shared information fractions required by correlation-exploiting methods.

B. Empirical & Simulation Analysis

Simulation: Monte Carlo simulations ( $N=1,000$ , 10,000 replications) varying the number of non-null metrics ( $k$ ), correlation structures ( $\rho$ ), and effect sizes.
Real-World Data: Analysis of 1,296 experiments (1,840 treatment-control comparisons) from Spotify.
- Metrics: Median 2 success metrics, 4 guardrail metrics per experiment.
- Comparison: Applied Bonferroni, Holm, Hommel, BH, and Benjamini-Yekutieli (BY) under two conditions:
  1. Naive Family: All metrics included in correction.
  2. Success-Only Family: Only success metrics included in correction (guardrails evaluated at raw $p$ -values).

3. Key Contributions

1. The "Success-Only" Denominator Argument

The most significant theoretical contribution is demonstrating that guardrail metrics do not contribute to the FWER inflation of the shipping decision.

Result: When guardrails are correctly excluded from the correction family, the Bonferroni denominator drops from the total metric count (e.g., 12) to the success metric count (e.g., 2).
Impact: This makes the effective significance level ( $\alpha/2 = 0.025$ ) much less conservative than the naive approach ( $\alpha/12 \approx 0.004$ ), rendering Bonferroni practically mild.

2. Coherent Confidence Intervals

The paper highlights that Bonferroni is the only broadly implementable FWER method that provides unconditional simultaneous CIs for all metrics.

Methods like Holm and Hommel produce CIs that are "non-informative" for rejected hypotheses (they only state the effect is in the rejection region, not the magnitude).
BH requires conditional CIs (FCR) which are not standard in industry platforms, leading to misleadingly narrow intervals if standard CIs are reported.

3. Tractability in Sample Size Planning

Bonferroni allows for straightforward pre-experiment power calculations ( $n \propto (z_{\alpha/2S} + z_\beta)^2$ ).

Methods like BH require assumptions about the proportion of true nulls ( $\pi_0$ ) and joint distributions, which are unknown at design time, making honest power statements impossible.

4. Empirical Power Analysis

The study quantifies the "power cost" of Bonferroni:

Sparse Regime (Typical Product Exp): When few metrics are truly non-null (the common case), the power gap between Bonferroni and sophisticated methods (Holm, Hommel, BH) is near zero.
Correct Family Specification: When using the "Success-Only" family, Holm and Hommel gain only 4–5 percentage points in "ship rate" (fraction of experiments deployed) over Bonferroni.
Naive Family: If guardrails are incorrectly included, Holm and Hommel are nearly indistinguishable from Bonferroni (gain < 1 pp).

4. Key Results

Metric	Finding
Ship Rate (Empirical)	With a Success-Only family, Holm/Hommel ship 4.5–4.6% more often than Bonferroni. With a Naive family, the gain is only 0.5–0.6%.
Power Gap (Simulation)	In sparse scenarios (1 non-null metric out of 8), the power difference between Bonferroni and BH/Hommel is < 1%. The gap widens only when many metrics are non-null and highly correlated.
Variance Reduction	Variance reduction techniques slightly reduce the advantage of correlation-exploiting methods (Holm/Hommel) over Bonferroni, as they decorrelate metrics.
FWER vs. FDR	In high-null environments, FDR methods (BH) produce higher ship rates primarily by relaxing the threshold (increasing false positives), not by capturing more true effects.

5. Significance and Conclusion

The paper argues that the reputation of Bonferroni as "too conservative" is largely a myth in the context of well-specified product experimentation.

Trust and Scale: Bonferroni is the most robust choice for building trust in experimentation platforms because it offers coherent inference (CIs + Decisions) without hidden assumptions.
Design Implication: The "cost" of Bonferroni is minimized not by switching methods, but by improving experimental design:
1. Limiting the number of Success Metrics (keeping $S$ small).
2. Correctly categorizing Guardrail Metrics so they are not penalized by the correction.
Final Verdict: For teams running experiments where the goal is to avoid shipping broken features (high cost of false positives) and where the number of truly effective changes is low, Bonferroni is the optimal balance of simplicity, interpretability, and statistical rigor. The marginal power gain of more complex methods is often negligible and comes at the cost of interpretability and planning tractability.

Quote from Conclusion: "The call is not to avoid corrections. It is to worry less about using Bonferroni."