📊 statistics

Nonparametric methods controlling the median of the false discovery proportion

This paper proposes a nonparametric multiple testing procedure that controls the median of the false discovery proportion by leveraging the symmetry of test statistics, offering superior power for scenarios where most alternative hypotheses are expected to be false (such as noninferiority or equivalence tests) without requiring independence assumptions.

Original authors: Jesse Hemerik

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jesse Hemerik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive crime scene with hundreds of clues. In the world of statistics, these "clues" are hypotheses (guesses about whether something is true or false). Usually, when we look at hundreds of clues, we expect most of them to be red herrings (false leads). We use standard detective tools to find the few real criminals while trying not to accuse innocent people.

But what if the situation is flipped? What if you are a food safety inspector checking a new batch of genetically modified crops? You expect that almost every single chemical in the crop is safe (i.e., most hypotheses are "false" in the sense that they are not dangerous). Your goal isn't to find the few bad apples; it's to prove that the vast majority of the batch is good.

This is the problem the paper tackles. The author, Jesse Hemerik, proposes a new, super-efficient way to handle these "mostly good" situations.

Here is the breakdown using simple analogies:

1. The Problem: The "Bad Apple" vs. The "Good Batch"

Most statistical methods are designed like a metal detector at an airport. They assume most people are innocent, and they are trying to find the few terrorists (false discoveries) without flagging too many innocent travelers. They control the average number of mistakes (called the False Discovery Rate or FDR).

However, in safety testing or equivalence testing, we are looking at a batch of cookies. We expect 99% of them to be delicious. We want to say, "Look, almost all these cookies are good!"

The Old Way: If we use the standard "metal detector" logic, we might be too cautious. We might say, "I can't be 100% sure this cookie is good," and reject it, even though it's delicious. We lose power.
The Goal: We want a method that says, "I am confident that most of these rejected hypotheses (the ones we say are 'safe') are actually correct."

2. The New Tool: The "Mirror Trick"

The author's secret weapon is a property called Symmetry.

Imagine you have a group of people standing on a seesaw. If the group is balanced (symmetric), and you flip the seesaw upside down, the pattern of people looks exactly the same, just mirrored.

The Analogy: The paper assumes that the "noise" in our data (the random errors) is perfectly balanced. If you flip the data upside down (multiply by -1), the pattern of errors stays the same.
The Trick: The author uses this symmetry to create a "mirror image" of the data.
- He looks at the real data to see how many "good" things he found.
- He looks at the "mirror" data to see how many "bad" things would have been found if the data were flipped.
- By comparing the real world to the mirror world, he can estimate how many mistakes he is making without needing to know the exact mathematical shape of the data (which is why it's "nonparametric").

3. The Metric: The "Median" vs. The "Average"

Standard methods try to keep the average number of mistakes low.

Analogy: If you have 100 days, and on 99 days you make 0 mistakes, but on 1 day you make 100 mistakes, your average is 1 mistake per day. That sounds okay, but that one bad day is a disaster.

The author proposes controlling the median (the middle value).

Analogy: In the same 100 days, the median is 0. This means that if you run this test 100 times, at least 50 of those times, you will make zero mistakes (or very few).
Why this matters: In safety testing, you don't want an "average" safety record. You want to be sure that most of the time, your conclusion is solid. Controlling the median gives you a "50% confidence" guarantee that your list of "safe items" is mostly correct.

4. Why It's Better (The "Speed and Power" Boost)

The paper compares this new method to existing ones (like "SAM" or "Benjamini-Hochberg").

The Old Methods: Like trying to count every single grain of sand on a beach to find the shells. They are slow, computationally heavy, and often too conservative (they reject too many good cookies because they are afraid of making a mistake).
The New Method: Like using a metal detector that is specifically tuned for the "mostly good" scenario.
- Faster: It doesn't need to run thousands of simulations (permutations). It just looks at the data and its mirror image. It's lightning fast.
- More Powerful: It finds more "good cookies" (rejects more null hypotheses) while still keeping the error rate low. It exploits the fact that you already know most things are safe.

5. Real-World Example: The House Prices

The author tested this on real data about house prices in Ames, Iowa.

The Question: "Do these 31 different house features (like lot size, number of bathrooms) have a negative effect on price?" (We expect the answer to be "No" for most of them).
The Result:
- The standard method (Benjamini-Hochberg) said: "I can only confirm 29 of these are safe."
- The new method said: "I can confirm 30 of these are safe!"
- Even when the data was "noisier" (smaller sample size), the new method found 27 safe features, while the standard method only found 16.

Summary

This paper introduces a smart, fast, and non-parametric way to test hundreds of things at once when you expect most of them to be true (or safe).

Instead of trying to be perfect (controlling the average error), it aims to be reliable most of the time (controlling the median error). It uses a clever mirror trick based on symmetry to estimate errors without heavy math, making it perfect for fields like food safety, drug equivalence, and genetics where we expect the "good stuff" to dominate.

1. Problem Statement

In many multiple testing scenarios, particularly in fields like food safety (e.g., testing metabolite concentrations for equivalence or non-inferiority) and genomics, researchers often have strong a priori expectations that most null hypotheses are false. Specifically, they expect parameters to lie within a specific interval (equivalence testing) or in a specific direction (non-inferiority testing).

Standard multiple testing methods, such as those controlling the False Discovery Rate (FDR, the mean of the False Discovery Proportion) or the Family-Wise Error Rate (FWER), are often conservative or lack power in these specific settings because they do not fully exploit the prior knowledge that the majority of signals are present. Furthermore, existing methods for controlling the False Discovery Exceedance (FDX) or the median FDP (mFDP) often rely on computationally expensive permutation procedures (like SAM+CT) or assume independence that may not hold.

The paper addresses the need for a powerful, nonparametric (or semiparametric) method that:

Controls the median of the False Discovery Proportion (mFDP), ensuring $P(\text{FDP} \leq \gamma) \geq 0.5$ .
Exploits the symmetry of test statistics to achieve high power when most hypotheses are indeed false.
Remains valid even if the user's prior expectations are incorrect, though power will be lower in that case.
Avoids the computational bottlenecks of permutation-based closed testing.

2. Methodology

The proposed approach relies on a symmetry property of the test statistics rather than data permutations.

2.1 Core Assumption: Symmetry

The method assumes that the vector of test statistics corresponding to the true null hypotheses, $(T_j - \mu_j : j \in N)$ , is symmetric about zero. That is:
$(T_j - \mu_j : j \in N) \stackrel{d}{=} (-(T_j - \mu_j) : j \in N)$
This assumption holds for many finite-sample nonparametric tests (e.g., sign-flipping, permutation tests) and is satisfied asymptotically for multivariate normal statistics. Crucially, the method does not require the test statistics to be independent.

2.2 Median Unbiased FDP Estimation

The paper introduces a novel estimator for the number of false discoveries ( $V$ ) and the FDP.

Directional Testing (One-sided): For hypotheses $H_j: \mu_j \leq \delta_j$ , the method rejects if $T_j > \delta_j + t$ . The estimator for the number of false discoveries is:
$\tilde{V}(t) = R(t) \wedge R^-(t)$
where $R(t)$ is the number of rejections and $R^-(t)$ is the number of test statistics falling in the "reflected" region ( $T_j < \delta_j - t$ ).
Equivalence Testing: For hypotheses $H_j: |\mu_j| \geq \delta_j$ , the method rejects if $|T_j| < \delta_j - t$ . The estimator is similarly defined using the count of statistics in the "outer" region ( $|T_j| > \delta_j + t$ ).

Theoretical Guarantee: Under the symmetry assumption, these estimators are median unbiased. That is, $P(V \leq \tilde{V}) \geq 0.5$ . This implies that $\tilde{FDP}$ is a 50% confidence upper bound for the true FDP.

2.3 mFDP Control Procedure

To control the mFDP at a level $\gamma$ (e.g., 0.1), the procedure:

Defines a rejection threshold $s^+$ based on the estimated FDP curve $\tilde{FDP}(t)$ .
Selects $s^+$ such that $\tilde{FDP}(t) \leq \gamma$ for all $t \geq s^+$ .
Rejects all hypotheses with statistics exceeding this threshold.

The paper proves (Theorems 5.1 and 5.2) that this procedure guarantees $P(\text{FDP} \leq \gamma) \geq 0.5$ under specific conditions (e.g., independence between true and false hypotheses, or asymptotically as sample size increases).

2.4 Admissibility and Connection to Closed Testing

The authors link their estimator to the theory of closed testing. They demonstrate that the proposed bound is equivalent to a closed testing procedure with specific local tests. By introducing a randomized coin-flip adjustment in the case of ties, they prove the estimator is admissible, meaning it cannot be uniformly improved upon by another valid median-unbiased estimator.

3. Key Contributions

Novel mFDP Control: A new, powerful procedure for controlling the median FDP that is specifically tailored for settings where most hypotheses are false (e.g., equivalence testing).
Computational Efficiency: Unlike existing nonparametric competitors like SAM+CT (Significance Analysis of Microarrays + Closed Testing), which have exponential time complexity in the number of hypotheses ( $m$ ), the proposed method has linear time complexity ( $O(m)$ ) after sorting. This makes it feasible for large-scale problems ( $m > 100$ ).
Symmetry-Based Estimation: The method exploits the symmetry of test statistics directly, avoiding the need for full permutation groups. It is shown that in specific one-sided settings, using only two transformations (identity and reflection) in a permutation framework yields the same result as the proposed method, but the proposed method is more general and adaptive.
Flexibility: The paper provides two methods:
- Method 1: A direct, high-power procedure for a pre-specified $\gamma$ .
- Method 2: A flexible procedure allowing post-hoc selection of $\gamma$ (based on Hemerik et al., 2024), though with slightly lower power.
Robustness: The methods are valid regardless of the actual parameter values, provided the symmetry assumption holds. They do not require independence between test statistics.

4. Results

4.1 Simulation Studies

FDP Estimation: The proposed estimator $\tilde{FDP}$ consistently provided lower (better) estimates of the FDP compared to the standard SAM method and SAM+CT, particularly when the proportion of true null hypotheses ( $\pi_0$ ) was low (e.g., $\pi_0 = 0.1$ ).
Power: In mFDP control simulations, the proposed method demonstrated superior power compared to:
- The Benjamini-Hochberg (BH) procedure (which controls mean FDP).
- The Romano-Wolf stepdown method (an FDX method).
- The flexible mFDP control method from Hemerik et al. (2024).
- The advantage was most pronounced when the signal was strong and the number of false hypotheses was high.

4.2 Real Data Application

The method was applied to a dataset of residential property sales in Ames, Iowa, to test the coefficients of a linear regression model.

Goal: Test if many covariates have a coefficient significantly greater than $-0.1$ (non-inferiority).
Result: The proposed mFDP method rejected 30 out of 31 hypotheses with a target $\gamma=0.1$ .
Comparison: The Benjamini-Hochberg method rejected only 29. In a reduced sample size scenario ( $n=100$ ), the mFDP method rejected 27 hypotheses, while BH rejected only 16 and Bonferroni rejected 7. This highlights the method's ability to detect signals in smaller samples where traditional methods become overly conservative.

5. Significance and Implications

Paradigm Shift in Multiple Testing: The paper challenges the dominance of FDR control in settings where the "sparsity" assumption (few true signals) is violated. It argues that when researchers expect many discoveries, controlling the median FDP is a more appropriate and powerful metric.
Practical Utility: The availability of the R package mFDP (Hemerik, 2026) allows researchers to easily implement these methods for safety testing, bioequivalence, and other fields requiring non-inferiority or equivalence testing.
Theoretical Rigor: By establishing admissibility and finite-sample validity without independence assumptions, the paper bridges the gap between theoretical nonparametric statistics and practical high-dimensional inference.
Efficiency: The linear computational complexity makes these methods viable for modern "big data" applications where permutation-based methods fail due to computational constraints.

In summary, this paper provides a robust, efficient, and highly powerful framework for multiple testing in scenarios where the researcher expects a majority of hypotheses to be false, offering a significant improvement over existing FDR and FDX control methods in such contexts.