📊 statistics

The Benjamini--Hochberg Procedure Can Fail to Control the FDR for Correlated Two-Sided Gaussian Tests

This paper disproves a long-standing conjecture by demonstrating that the Benjamini–Hochberg procedure fails to control the false discovery rate at its nominal level for correlated two-sided Gaussian tests, a result rigorously proven via interval arithmetic and verified by the author.

Original authors: Edgar Dobriban

Published 2026-07-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Edgar Dobriban

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to catch a group of liars in a room full of honest people. You have a special rulebook (the Benjamini–Hochberg procedure) that tells you how to shout "Gotcha!" without accidentally accusing too many innocent bystanders. For decades, statisticians believed this rulebook was unbreakable. They thought that even if the people in the room were whispering secrets to each other (correlated data), the rulebook would still keep the number of false accusations below a strict limit, say 1% (0.01).

But a new paper, written with the help of a super-smart AI named GPT-5.6 Pro, has just pulled the rug out from under that belief.

The Big Discovery: The Rulebook Has a Loophole
The authors show that for a specific type of tricky situation—where the "tests" are like two-sided Gaussian numbers (think of them as measurements that can wiggle up or down from zero)—the rulebook can actually fail. It turns out that when these measurements are correlated in a very specific, sneaky way, the rulebook lets the false accusation rate creep up just a tiny bit above the limit.

Instead of staying safely at 1%, the rate of false alarms can jump to 0.0104. That's a small number, but in the world of strict math, it's a giant crack in the dam. The paper doesn't just guess this; it builds a rigorous "certificate" (like a mathematical safety inspector's report) that proves, with absolute certainty, that the rate must be higher than 0.01 for large groups of tests.

The "Sneaky Factor" Model
How did they find this loophole? They built a fictional world, a "factor model," to test the rulebook. Imagine a giant room with three groups of people:

The Honest Crowd (96% of the room): These are the true nulls. They are supposed to be innocent.
The Signal Group A (1% of the room): These are the "real" signals, moving in one direction.
The Signal Group B (3% of the room): These are also real signals, but moving in a different direction.

Here's the twist: Everyone in the room is listening to a single, invisible conductor (a "latent factor" called Z).

The Honest Crowd moves with the conductor.
The Signal Groups move against the conductor, but at different speeds.

When the conductor waves their baton in a certain way, the signals get louder, making the rulebook shout "Gotcha!" more often. But at the exact same time, the "Honest Crowd" gets a bit jittery, making their tails (the edges of their behavior) look suspiciously like the signals. This creates a perfect storm where the rulebook gets confused, rejects too many innocent people, and the false discovery rate spikes just enough to break the 1% rule.

How Sure Are They?
This isn't a "maybe" or a "we think." The authors are incredibly sure.

The Proof: The core argument was generated by an AI and then carefully checked by the human author. They used a method called "interval arithmetic," which is like doing math with a safety margin that guarantees the answer is never wrong, even with rounding errors. They proved that for a large enough number of tests, the rate is strictly greater than 0.0104.
The Simulation: To back this up, they ran computer experiments (Monte Carlo simulations). When they tested with 200 groups of data (totaling 20,000 tests), the computer measured the false discovery rate at 0.010359. This is statistically higher than the 1% limit, confirming the theory in a real-world simulation.

What This Means (and What It Doesn't)
This paper explicitly rules out the idea that the Benjamini–Hochberg procedure is safe for all correlated two-sided Gaussian tests. For twenty years, experts believed it was safe; this paper says, "Actually, it's not."

However, the paper is careful not to say the rulebook is useless. The violation is tiny (0.0104 vs 0.01), and it happens in a very specific, constructed scenario with a large number of tests. The authors admit they don't know if this happens with smaller numbers of tests or if there's a universal cap on how bad it can get. They also didn't test this on real-world data like genes or stocks yet; they built a mathematical model to prove the point.

So, the next time you hear that a statistical method is "proven" to work under any condition, remember this paper: sometimes, the most trusted tools have a tiny, hidden crack that only a super-smart detective (and a bit of AI help) can find.

Technical Summary: The Benjamini–Hochberg Procedure Can Fail to Control the FDR for Correlated Two-Sided Gaussian Tests

Problem Statement
The paper addresses a long-standing open question in multiple hypothesis testing: whether the Benjamini–Hochberg (BH) procedure controls the False Discovery Rate (FDR) at its nominal level $\alpha$ under arbitrary correlation structures for two-sided Gaussian tests. While the BH procedure is known to control FDR under independence (Benjamini and Hochberg, 1995) and under the Positive Regression Dependence on a Subset (PRDS) condition (Benjamini and Yekutieli, 2001), the case of correlated two-sided tests remained unresolved.

Prior literature, including empirical studies (Reiner-Benaim, 2007; Kim and van de Wiel, 2008) and influential reviews (Benjamini, 2010; Sarkar, 2023), widely conjectured that FDR control holds for two-sided $z$ -tests with any correlation structure. The paper challenges this conjecture, positing that the "folding" of two tails in two-sided tests ( $\{P_i \le t\} = \{X_i \ge c_t\} \cup \{X_i \le -c_t\}$ ) disrupts the monotone-regression and total-positivity arguments that work for one-sided statistics, potentially leading to FDR inflation.

Methodology and Construction
To disprove the conjecture, the authors construct a specific Gaussian factor model involving $m_N = 100N$ hypotheses, partitioned into three blocks:

True Nulls ( $96N$ ): $X^{(0)}_i = \frac{3}{10}Z + \frac{\sqrt{91}}{10}\epsilon_i$ .
Non-nulls Block 1 ( $N$ ): $X^{(1)}_j = \frac{12}{5} - \frac{3}{10}Z + \frac{\sqrt{91}}{10}\eta_j$ .
Non-nulls Block 2 ( $3N$ ): $X^{(2)}_k = \frac{22}{5} - \frac{18}{25}Z + \frac{\sqrt{301}}{25}\xi_k$ .

Here, $Z$ is a common latent factor, and $\epsilon, \eta, \xi$ are independent standard normals. The means of the non-null blocks are tuned to move against the latent factor $Z$ , while the null block moves with it. This creates a scenario where, for specific realizations of $Z$ , the non-null signals increase the rejection count (raising the BH threshold) precisely when the conditional null distribution develops heavier two-sided tails.

The proof relies on an asymptotic analysis of the empirical cumulative distribution function (CDF) of $p$ -values. The authors utilize a "threshold bracketing" technique:

They define a limiting conditional CDF $G_z(t)$ for the $p$ -values given $Z=z$ .
They identify a range of $z$ values where the BH threshold $\tau_N$ is strictly bounded between two values $v$ and $w$ (where $v < w < \alpha$ ).
Within this range, the conditional False Discovery Proportion (FDP) is shown to exceed $\alpha$ .

Key Contributions and Proof Mechanism
The primary contribution is a rigorous counterexample proving that FDR control fails for correlated two-sided Gaussian tests. The proof is distinguished by two innovations:

Avoidance of Limiting Uniqueness: The argument does not require a unique limiting crossing point or differentiability of the CDF at the crossing. Instead, it uses two strict sign conditions to bracket the threshold.
Rigorous Interval Arithmetic: The proof employs a finite collection of inequalities verified by "outward-rounded interval arithmetic" (using the Arb library). By partitioning the latent factor space and the $p$ -value threshold space into rational rectangles, the authors certify that specific inequalities hold strictly. This eliminates floating-point errors, providing a mathematical certificate that the FDR exceeds the nominal level.

The proof was generated by the AI model GPT-5.6 Pro and subsequently verified by the author, who also conducted the numerical simulations and drafted the manuscript.

Results

Theoretical Result: For the constructed model at level $\alpha = 0.01$ , the authors prove that for all sufficiently large $N$ , the FDR satisfies $\text{FDR}_N > 0.0104$ . This strictly violates the nominal level of 0.01.
Numerical Certificate: The interval arithmetic certificate sums contributions across 1,000 bins of the latent factor $Z$ , yielding a certified lower bound of approximately $0.0104168$.
Monte Carlo Experiments: Stratified Monte Carlo simulations (stratified on $Z$ $Z$ to handle variance) were conducted for $N=50, 100, 200$ $N = 50, 100, 200$ .
- At $N=200$ , the estimated FDR was $0.010359$, with a 95% confidence interval $[0.010155, 0.010563]$ , which lies entirely above the nominal level.
- The results are consistent with the asymptotic theoretical finding, showing a failure of control in finite samples as well.

Significance and Claims
The paper claims to disprove a conjecture that has been widely believed for twenty years. It demonstrates that the BH procedure is not universally robust to arbitrary correlation structures in two-sided Gaussian tests, specifically when the correlation induces a coupling between the signal strength and the null distribution's tail behavior.

The authors note that the violation is slight (approximately 4% inflation over the nominal level) and occurs in a model with a large number of tests. They explicitly state that it remains an open question whether a universal bound exists on the possible inflation of FDR or if control is guaranteed for smaller numbers of tests. The work highlights the limitations of current theoretical guarantees for the BH procedure in complex, dependent settings and underscores the necessity of dependence-adjusted alternatives in such scenarios.

Technical Summary: The Benjamini–Hochberg Procedure Can Fail to Control the FDR for Correlated Two-Sided Gaussian Tests

More like this