📊 statistics

Calibration without labels in multiple testing

This paper proposes a novel method for calibrating multiple testing results without ground-truth labels by constructing pseudo-labels from $p$ -value spacings to enable stochastic assessment, revealing that the widely used $q$ -value is often severely miscalibrated in real-world psychology and neuroscience literature.

Original authors: Adway S. Wadekar, Jake A. Soloff

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adway S. Wadekar, Jake A. Soloff

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve thousands of tiny mysteries at once. In each mystery, you have a suspect (a hypothesis) and a piece of evidence (a p-value). Your job is to decide which suspects are actually guilty (the null hypothesis is false) and which are innocent (the null hypothesis is true).

The problem is that you have no answers key. In a normal detective story, you eventually find out who the real culprit was. But in this statistical world, the "truth" is hidden forever. You just have the evidence and have to guess.

This paper, by Wadekar and Soloff, introduces a clever trick to solve this mystery without ever seeing the answer key. They propose a way to check if your guesses are "calibrated"—meaning, if you say there is a 10% chance a suspect is guilty, is it actually true that 10% of the time you were wrong?

Here is the breakdown of their solution using everyday analogies:

1. The Problem: Guessing Without a Key

Usually, to check if a weather forecaster is good, you wait to see if it actually rains. If they say "10% chance of rain" and it rains 10% of the time, they are calibrated.

In science, researchers run thousands of tests and get p-values. They often use a standard tool called the q-value to say, "This result is likely real." But because we never know the true answer (we don't know which hypotheses are actually true), we can't check if these q-values are telling the truth. They might be wildly overconfident or underconfident, and we wouldn't know.

2. The Solution: Creating "Fake" Clues (Pseudo-Labels)

The authors' big idea is to create fake clues (which they call "pseudo-labels") that act as a stand-in for the missing truth.

Imagine you are looking at a line of people sorted by how suspicious they look (from most suspicious to least).

The Trick: Instead of asking "Is this person guilty?" (which you can't know), you look at the gap between this person and the next one in line.
The Logic: If the "innocent" people are spread out evenly (like raindrops on a sidewalk), the gaps between them should be uniform. If you see a huge gap between two people, it suggests the person before the gap might be "guilty" (or at least, the pattern has changed).
The Result: By measuring these gaps, the authors create a continuous number for every test that behaves like a probability of guilt. It's not the real truth, but it's a mathematical shadow of the truth that allows them to run standard checks.

3. The Tool: Smoothing the Rough Edges

Once they have these fake clues, they use a technique called Isotonic Calibration.

Think of a bumpy, jagged mountain range. You want to smooth it out into a gentle, steady slope where "more evidence" always equals "higher probability of guilt."

The authors take their jagged data and force it into a smooth, straight line.
They prove that this smoothing process is mathematically identical to three other famous statistical methods (one used by weather forecasters, one used by machine learning, and one used by statisticians studying distributions).
The Payoff: This smoothed line gives you a score (like a "0.05" or "0.10") that you can trust. If the score says 10%, it really means that about 10% of the time, the hypothesis is actually true (a false alarm).

4. The Discovery: The Old Tools Were Broken

The authors tested their new method on a massive collection of real scientific papers from psychology and neuroscience (about 27,000 studies).

They compared their new "smoothed" scores against the old standard (the q-value) and the raw evidence (p-values).

The Result: The old tools were severely miscalibrated. They were like a broken thermometer that said it was 70°F when it was actually freezing.
The New Method: Their new "smoothed" scores lined up perfectly with the truth (as best as we can tell without an answer key).

5. Why This Matters

This paper doesn't just say "we have a new calculator." It changes how we view the goal of science.

Old View: The goal is to count how many mistakes we make in total (like saying "we will have 5% false alarms in this whole batch").
New View: The goal is to give a personalized probability for each specific experiment. "For this specific study, there is a 12% chance the result is a fluke."

Because they created a way to check these probabilities without needing the "answer key," scientists can now look at a single study and say, "I am 88% confident this is real," with a much higher degree of trust than before.

Summary Analogy

Imagine you are grading a stack of 10,000 essays, but you don't have the answer key.

The Old Way: You guess a grade based on a rule of thumb, but you have no idea if your grading scale is fair.
The New Way: You look at the spacing between the essays. If the "bad" essays are usually clustered together and the "good" ones are spread out, you use the gaps between them to create a fake grading scale. You then smooth out your grades so that a "B" always means the same thing.
The Result: You realize your old grading scale was broken, and your new scale gives you a reliable probability that any single essay is actually good, even without seeing the teacher's answer key.

Technical Summary: Calibration without Labels in Multiple Testing

Problem Statement

Large-scale hypothesis testing often aims to provide probability claims about individual hypotheses, such as estimating local false discovery rates (lfdr) via empirical Bayes methods. However, a critical gap exists in interpreting these scores: a reported score $g(p)$ is only meaningful if it is calibrated, meaning that among tests assigned a score $g(p)$ , the null hypothesis is true with probability exactly $g(p)$ .

In standard supervised learning, calibration is assessed using ground-truth labels. In multiple testing, the "labels" (whether the null hypothesis is true or false) are unobserved. Consequently, standard calibration diagnostics cannot be applied directly. While the empirical Bayes tradition recognizes that the distribution of test statistics contains structural information, there has been limited work on formally assessing or enforcing calibration in the absence of ground truth, particularly under model misspecification.

Methodology

The authors reframe multiple testing as a stochastically supervised binary classification problem. They treat the $p$ -values as covariates and the unobserved truth of the null hypothesis as a binary label $Y_i \in \{0, 1\}$ . The goal is to estimate the regression function $lfdr(p) = P(Y=1|p)$.

1. Pseudo-Label Construction

To bridge the gap created by missing labels, the authors construct pseudo-labels derived from the spacings of ordered $p$ -values. Let $p_{(1)} \le \dots \le p_{(m)}$ be the ordered $p$ -values and $\hat{\pi}_{null}$ be an estimate of the proportion of true nulls. The pseudo-label for the $i$ -th observation is defined as:
$\tilde{Y}_i := m \hat{\pi}_{null} (p_{(R_i)} - p_{(R_i-1)})$
where $R_i$ is the rank of $p_i$ .

Theoretical justification relies on the asymptotic properties of order statistic spacings (Pyke, 1965). As $m \to \infty$ , these pseudo-labels satisfy a regression model with multiplicative exponential errors: $\tilde{Y}_i | p_i \approx lfdr(p_i) E_i$ , where $E_i \sim \text{Exp}(1)$ . Crucially, the expectation $E[\tilde{Y}_i | p_i]$ approximates the true $lfdr(p_i)$ , allowing pseudo-labels to serve as drop-in replacements for true labels in standard calibration tools.

2. Isotonic Calibration and Equivalence

The authors propose enforcing monotonicity (smaller $p$ -values should yield smaller error probabilities) via isotonic regression on the pseudo-labels. They demonstrate that three distinct methodological approaches yield the exact same estimator:

Isotonic Regression: Minimizing the squared error between the predictor and pseudo-labels (Pool Adjacent Violators Algorithm).
Maximum Likelihood: Minimizing the negative log-pseudolikelihood derived from the exponential error model (equivalent to minimizing Itakura–Saito distance).
Nonparametric Empirical Bayes: Estimating the marginal density of $p$ -values using the Grenander estimator (the MLE for a non-increasing density) and applying Bayes' rule.

The resulting estimator, denoted $\widehat{lfdr}^\uparrow$ , is a step function that pools adjacent $p$ -values into intervals where the local False Discovery Proportion (FDP) estimate is monotone.

3. Assessment and Diagnostics

The framework allows for the adaptation of supervised calibration diagnostics to multiple testing:

Reliability Diagrams: By binning the pseudo-labels, one can plot the average predicted score against the empirical frequency of "true nulls" (estimated via pseudo-labels).
Brier Regret: The authors define a finite-sample bound on the Brier regret (the excess Brier score relative to the best monotone predictor) for their isotonic estimator.

Key Contributions

Pseudo-Label Construction: The paper introduces a method to generate regression targets from $p$ -value spacings that have the $lfdr$ as their conditional expectation. This unlocks the full toolkit of supervised calibration (e.g., reliability diagrams, isotonic calibration) for multiple testing without ground truth.
Theoretical Unification: The authors prove that isotonic calibration on pseudo-labels, maximum likelihood estimation under an exponential error model, and nonparametric empirical Bayes estimation (via the Grenander estimator) are mathematically equivalent.
Finite-Sample Guarantees: They establish a finite-sample bound on the Brier regret of the isotonic estimator. The bound vanishes as $m \to \infty$ even if the true $lfdr$ is not monotone, provided the empirical CDF converges uniformly. This result holds regardless of whether the monotonicity assumption is strictly true, as the estimator targets the isotonized $lfdr$ (the $L_2$ projection of the true $lfdr$ onto the class of monotone functions).
Empirical Evidence of Miscalibration: Through simulations and real-world data analysis, the authors demonstrate that standard error measures, specifically the q-value, can be severely miscalibrated, sometimes performing worse than raw $p$ -values.

Results

Simulation Studies

The authors compared the Brier regret of three predictors: raw $p$ -values, Storey's $q$ -values, and their isotonic estimator ( $\widehat{lfdr}^\uparrow$ ).

Convergence: The regret of $\widehat{lfdr}^\uparrow$ decreased steadily with the number of tests $m$ , converging at a polynomial rate.
Stagnation: The regret for $p$ -values remained flat (as they are the identity function), and the regret for $q$ -values remained significantly higher than the isotonic estimator in most scenarios.
Robustness: Even in scenarios where the true $lfdr$ was non-monotone (violating the shape constraint), the isotonic estimator converged to the isotonized $lfdr$, maintaining vanishing regret relative to the best monotone predictor.

Real Data Application

The method was applied to a dataset of $\approx 27,000$ $t$ -statistics from cognitive neuroscience and experimental psychology literature (Szucs and Ioannidis, 2017).

Calibration Check: Reliability diagrams showed that both raw $p$ -values and $q$ -values were severely miscalibrated (points deviated significantly from the diagonal $y=x$ ).
Performance: The isotonic estimator $\widehat{lfdr}^\uparrow$ produced scores that lay close to the diagonal, suggesting that the calibrated scores can be interpreted as valid local probability statements for individual studies.

Significance and Claims

The paper argues that calibration should be a primary inferential goal in large-scale hypothesis testing, distinct from mere multiplicity correction. By treating multiple testing as a classification problem with hidden labels, the authors show that meaningful error probabilities can be established even without ground truth.

The significance of the work lies in:

Interpretability: It provides a route to interpret scores as calibrated probabilities even under model misspecification, as calibration is a property of the score itself rather than the generative model.
Methodological Bridge: It successfully imports tools from probabilistic forecasting and supervised learning (specifically isotonic calibration and reliability diagrams) into the domain of multiple testing.
Critical Insight: The empirical finding that popular measures like the $q$ -value can be poorly calibrated challenges the assumption that standard FDR-based measures are inherently reliable for individual hypothesis assessment.

The authors remain modest regarding limitations, noting that their theory relies on $p$ -values being uniformly distributed under the null. They identify the construction of similar pseudo-labels for non-uniform null distributions (e.g., $z$ -scores or high-dimensional statistics) as an open question for future research.