📊 statistics

Improved null proportion estimators for multiple discrete tests with plug-in FDR control

This paper introduces a general class of null proportion estimators and proposes new strategies that leverage null distribution information to reduce conservativeness and improve efficiency in multiple discrete hypothesis testing while maintaining valid plug-in FDR control.

Original authors: Sebastian Döhler, Iqraa Meah

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sebastian Döhler, Iqraa Meah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive case involving hundreds of suspects (hypotheses). Your job is to figure out which ones are actually guilty (reject the null hypothesis) and which ones are innocent (keep the null hypothesis).

In the world of statistics, there's a famous rule called the Benjamini-Hochberg (BH) procedure. Think of this as a standard "guilt threshold." If a suspect's evidence (a p-value) is strong enough to pass this threshold, you declare them guilty. This rule is great because it controls the "False Discovery Rate" (FDR)—essentially, it promises that if you arrest 100 people, only a small, controlled percentage of them will actually be innocent.

However, there's a catch. The standard rule assumes that the "evidence" for innocent people looks like a perfectly smooth, continuous slide from 0 to 1 (like a ramp). But in the real world, especially when dealing with counts (like counting patients or genes), the evidence often looks like a staircase. You can only stand on specific steps, not anywhere in between.

The Problem: The "Staircase" Trap

When the evidence is a staircase (discrete data) instead of a smooth ramp (continuous data), the standard detective rule gets too scared. It becomes overly cautious.

The Analogy: Imagine a security guard at a club who is supposed to let in 10% of the people who look suspicious. If the people are walking on a smooth ramp, the guard does a good job. But if they are walking on a staircase, the guard gets confused. Because the steps are "jumpy," the guard thinks, "This person is almost at the top step, but maybe they aren't quite there yet." To be safe, the guard lets in almost no one.
The Result: The guard (the statistical test) becomes too conservative. They miss a lot of actual "guilty" suspects (low power) because they are terrified of accidentally letting in an innocent one.

The Solution: New Estimators

The authors of this paper, Iqraa Meah and Sebastian Döhler, say, "We can do better." They propose a new set of tools (estimators) to help the detective count how many suspects are actually innocent ( $\pi_0$ ) more accurately, even when the evidence is on a staircase.

They introduce a "General Class" of tools that includes old, famous tools (like Storey's and Pounds-Cheng's) but fixes them for the staircase world. They offer three creative ways to fix the "over-cautious" problem:

1. The "Customized Ruler" (Rescaling)

The Idea: The old tools used one giant ruler to measure everyone. But on a staircase, every step might be a different height.
The Fix: Instead of using one ruler, they give each suspect their own custom ruler that fits their specific step size perfectly. This ensures the measurement is fair and doesn't overestimate how many people are innocent.

2. The "Mid-Step" Guess (Conditional Mean)

The Idea: If you are standing on a step, the old tools assume you are at the very top edge of that step. But statistically, you might be somewhere in the middle.
The Fix: They use a "mid-p-value" approach. Instead of guessing you are at the top of the step, they guess you are right in the middle of the step. This smooths out the staircase, making it look more like the smooth ramp the original rules were designed for. It's like filling in the gaps between the stairs with a ramp just for the calculation.

3. The "Magic Dice" (Expected Randomization)

The Idea: Sometimes, the steps are so jagged that even the middle guess isn't perfect.
The Fix: Imagine rolling a magic die for every suspect to see if they "slide" up or down slightly within their step. You do this thousands of times in a computer simulation and take the average result. This creates a "smooth" version of the jagged data.
The Catch: While this is the most accurate mathematically, it requires a lot of computer power (like rolling dice a million times), so the authors suggest using it carefully.

What Did They Prove?

The paper claims three main things:

Safety First: All these new methods still guarantee that the "False Discovery Rate" stays under control. The detective won't accidentally arrest too many innocent people.
Better Efficiency: Because these methods stop being overly cautious, they catch more of the actual "guilty" suspects. They are more powerful.
Fixing Old Tools: They showed that even the old Pounds-Cheng tool, which was previously unproven for this specific type of control, can be slightly tweaked to work perfectly with their new safety guarantees.

Real-World Testing

The authors tested these ideas in two ways:

Simulations: They created fake data with "staircases" (discrete tests) and saw that their new methods found more "guilty" suspects without letting in more innocent ones compared to the old methods.
Real Data: They applied this to real biological data (from the International Mouse Phenotyping Consortium), where they were looking at how gene changes affect mouse traits. The results were the same: the new methods worked better, finding more significant connections between genes and traits than the old, overly cautious methods.

Summary

In short, the paper says: "Stop treating jagged, step-like data as if it were smooth. By adjusting your measuring tools to fit the steps (using rescaling, mid-steps, or randomization), you can be less afraid of making mistakes, which allows you to find more true discoveries without breaking the rules of statistical safety."

Technical Summary: Improved Null Proportion Estimators for Multiple Discrete Tests with Plug-in FDR Control

Problem Statement
The Benjamini-Hochberg (BH) procedure is a standard method for controlling the False Discovery Rate (FDR) in multiple hypothesis testing. While "plug-in" or adaptive BH procedures, which incorporate estimators of the proportion of true null hypotheses ( $\pi_0$ ), often yield higher statistical power, their theoretical guarantees and performance are heavily dependent on the distribution of p-values. Existing literature has largely focused on continuous test statistics where null p-values follow a uniform distribution. However, in discrete testing scenarios (e.g., based on contingency tables or count data), p-values are "super-uniform" (stochastically larger than uniform) and possess heterogeneous supports.

In this discrete setting, classical plug-in estimators (such as those by Storey and Pounds–Cheng) become overly conservative. This conservativeness leads to inflated estimates of the number of null hypotheses ( $m_0$ ), resulting in a loss of power for the resulting adaptive BH procedures. Furthermore, while some estimators like Storey's have established FDR control proofs, others, such as the Pounds–Cheng estimator, have had gaps in their theoretical justification, particularly regarding plug-in FDR control in both continuous and discrete settings.

Methodology
The authors propose a unified framework to address these limitations through three main methodological developments:

A General Class of Estimators: The paper introduces a general class of estimators ( $\mathcal{F}$ ) for the number of null hypotheses $m_0$ . These estimators are defined as sums of p-values transformed by bounded, non-decreasing functions. This class encompasses classical estimators (Storey's and a modified Pounds–Cheng) and allows for the incorporation of specific null distribution functions ( $F_i$ ) known for each test.
- The class is defined as: $\hat{m}_0 = \frac{1}{\min(\nu_1, \dots, \nu_m)} + \sum_{i=1}^m \frac{g_i(p_i)}{\nu_i}$ , where $g_i$ are non-decreasing functions and $\nu_i$ are normalization constants.
- Theoretical justification relies on the "Inverse Moment Criterion" (IMC), a sufficient condition for plug-in FDR control established by Blanchard and Roquain (2009).
Strategies to Mitigate Conservativeness: To adapt these estimators to the discrete setting, the authors propose three specific strategies that utilize knowledge of the null distribution functions $F_i$ :
- Rescaling Constants: Replacing global normalization constants with individual constants $\nu_i^{resc} = E_{F_i}[g(p_i)]$ to ensure the expectation of the transformed p-value under the null equals 1, thereby eliminating "null-bias."
- Conditional Mean Transformations: Replacing the global transformation function $g$ with a "deflated" function $\bar{g}_{F_i}$ (e.g., using mid-p values or conditional expectations) such that $E[\bar{g}_{F_i}(p_i)] = E[g(U)]$ where $U$ is uniform. This mimics the behavior of continuous p-values.
- Expected Randomization: Utilizing randomized p-values (which are uniformly distributed) and taking the conditional expectation of the estimator with respect to the randomization. This yields a non-randomized estimator that is theoretically less conservative than the standard estimator.
Optimization for Transformation Functions: The paper sketches approaches for selecting the transformation functions $g$ to minimize bias in the discrete setting and Mean Squared Error (MSE) in the continuous setting. This involves formulating optimization problems (e.g., quadratically constrained linear programs) to find optimal weights for step-function transformations.

Key Contributions

Unified Theoretical Framework: The paper establishes a general class of estimators ( $\mathcal{F}$ ) and proves that any estimator within this class satisfies the IMC, thereby guaranteeing plug-in FDR control under independence and super-uniformity assumptions.
Resolution of the Pounds–Cheng Gap: By showing that a slightly modified version of the Pounds–Cheng estimator fits within the proposed general class, the authors provide the first rigorous proof of plug-in FDR control for this estimator in the continuous case.
Discrete Adaptations: The paper introduces and validates three specific modifications (rescaling, conditional mean, and expected randomization) that adapt classical estimators to discrete data. These modifications are proven to be less conservative (yielding smaller estimates of $m_0$ ) than their unadjusted counterparts while maintaining valid FDR control.
Empirical Validation: The proposed methods are tested on simulated data (two-sample binary response problems using Fisher's Exact Tests) and real data from the International Mouse Phenotyping Consortium (IMPC).

Results

Simulation Studies: In simulations with varying levels of discreteness (controlled by sample size $N$ ), the discrete-adjusted estimators significantly outperform classical estimators. The adjusted estimators provide more accurate estimates of $\pi_0$ (avoiding values $>1$ which indicate severe over-conservatism) and result in higher statistical power for the plug-in BH procedure.
Robustness to Dependence: The discrete estimators remain robust when p-values exhibit positive dependence (modeled via Gaussian copulas), maintaining FDR control well below the nominal level.
Real Data Application: Analysis of IMPC gene-phenotype data confirms that accounting for discreteness improves both the estimation of $\pi_0$ and the number of discoveries (rejections) compared to standard continuous-based methods. The discrete versions of the Pounds–Cheng estimator perform comparably to the Storey estimator, whereas the unadjusted Pounds–Cheng estimator is inferior.
Comparison of Strategies: The three proposed adjustment strategies (rescaling, mean, randomization) yield essentially similar performance gains in terms of efficiency and power. The "expected randomization" estimator is theoretically the most efficient but requires numerical approximation (Monte Carlo) in practice.

Significance and Claims
The paper claims to fill critical gaps in the literature regarding plug-in FDR control for discrete hypothesis testing. Specifically:

It provides a rigorous theoretical foundation for the Pounds–Cheng estimator, which previously lacked a proof of plug-in FDR control.
It offers a general, flexible methodology to construct conservative estimators that are provably less conservative in discrete settings without sacrificing FDR control.
It demonstrates that the loss of power in discrete testing due to the super-uniformity of p-values can be effectively mitigated by incorporating specific null distribution information into the estimation of $\pi_0$ .

The authors maintain a modest tone, noting that while their methods improve efficiency, the rejection capacity of the procedure is still ultimately limited by the inherent super-uniformity of discrete p-values. They also acknowledge that finding the optimal transformation function $g$ for minimal bias or MSE is a complex optimization problem left for future work, though they provide a heuristic approach for the continuous case.