📊 statistics

A semi-supervised framework for diverse multiple hypothesis testing scenarios

The paper introduces RESET, a semi-supervised framework that integrates side information into multiple hypothesis testing via data splitting and an ensemble of classifiers to achieve finite-sample error-rate control while maintaining high power and computational efficiency across diverse scenarios.

Original authors: Jack Freestone, William Stafford Noble, Uri Keich

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jack Freestone, William Stafford Noble, Uri Keich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a massive crime scene where thousands of clues have been left behind. Your job is to find the few real clues that actually point to the criminal, while ignoring the thousands of red herrings that look suspicious but are actually innocent. In the world of science, this is called "multiple hypothesis testing." Scientists often run thousands of experiments at once—like checking if thousands of different genes are linked to a disease or if thousands of protein fragments are present in a blood sample. For each test, they get a number called a "p-value," which is like a "suspicion score." A low score means the clue is very suspicious (likely a real discovery), while a high score means it's probably just noise.

The tricky part is that if you look at thousands of clues, you will inevitably find some that look suspicious just by pure luck. If you aren't careful, you might arrest the wrong people (false discoveries). To stop this, scientists use strict rules to control the "False Discovery Rate" (FDR), which is basically a promise: "We will only say we found a criminal if we are very sure we haven't falsely accused too many innocent people." Recently, scientists have realized that they have extra information—like a witness description or a time of day—that could help them sort the real clues from the fake ones faster. The big question is: how do we use this extra info to find more real clues without breaking the rules and accidentally arresting innocent people?

This is where a new tool called RESET (REScoring via Estimating and Training) comes in. Think of RESET as a super-smart detective assistant that uses a clever trick called "semi-supervised learning." Usually, to teach a computer to spot a criminal, you need a list of known criminals and known innocent people. But in science, you don't know who is who yet—that's the whole mystery! RESET solves this by splitting its own data into two teams: a "Training Team" and an "Estimating Team."

First, RESET takes a bunch of the "decoy" clues (fake clues that we know are innocent because we made them up) and splits them in half. One half goes to the Training Team. The assistant uses this half, along with the extra information (like the witness description), to learn how to spot patterns that separate the real clues from the fake ones. It builds a new, smarter "suspicion score" for every single clue. Then, the Training Team is sent home. The assistant takes the other half of the decoys (the Estimating Team) and the newly re-scored clues to make the final list of discoveries. Because the assistant never used the Estimating Team to learn, it cannot use the data inappropriately. It can use the extra information to find more real clues, but it still has to follow the strict rules to ensure it doesn't falsely accuse anyone.

The paper shows that this method works incredibly well. In computer simulations and real-world data (like analyzing proteins in a lab), RESET was able to find more real discoveries than older methods, often by a significant margin. It was also much faster. For example, when analyzing a large protein dataset, older methods took over a day or even weeks to finish, while RESET finished in less than 20 minutes. The authors tested this in many different scenarios, including when the clues are messy or dependent on each other, and found that RESET consistently kept the error rates under control while finding more "criminals."

The paper also highlights that RESET is flexible. It can handle two different types of scientific problems: those that use "p-values" (the standard suspicion scores) and those that use "competition" (where clues are pitted against each other in a head-to-head match). Most importantly, the authors prove mathematically that RESET guarantees the error rates stay low, even with small amounts of data. They also show that it can control a stricter type of error called "False Discovery Exceedance" (FDX), which ensures that the number of false accusations never spikes above a certain limit, giving scientists even more confidence in their results.

In short, RESET is a new, fast, and reliable way for scientists to use extra information to find more real discoveries without breaking the rules of statistical safety. It's like giving a detective a high-tech scanner that helps them spot the real evidence instantly, while a strict judge ensures they never convict an innocent person. The authors suggest that this tool could replace older, slower methods in many fields, from biology to genetics, helping researchers get better answers in less time.

Technical Summary: A Semi-Supervised Framework for Diverse Multiple Hypothesis Testing Scenarios

Problem Statement

Standard multiple testing procedures typically report a list of discoveries (suspected false null hypotheses) based solely on $p$ -values or test scores, aiming to control error rates such as the False Discovery Rate (FDR) or False Discovery Exceedance (FDX). While recent research has focused on leveraging side information ( $x_i$ ) to improve power by weighting or rescoring hypotheses, existing methods face significant limitations:

Computational Infeasibility: Many state-of-the-art semi-supervised methods (e.g., Adaptive Knockoffs, AdaPT variants) rely on iterative pruning strategies that are computationally prohibitive for large-scale datasets (e.g., peptide detection in proteomics).
Lack of Flexibility: Users must often select specific machine learning models or error control mechanisms (FDR vs. FDX) in advance; selecting a method post-hoc based on results can compromise error control.
Finite-Sample Guarantees: Many methods offering side-information integration only provide asymptotic error control, or lack finite-sample FDX control.
Scenario Compatibility: Few methods seamlessly handle both $p$ -value-based and competition-based (e.g., target-decoy) testing frameworks while maintaining rigorous error control.

Methodology: RESET

The authors propose RESET (REscoring via Estimating and Training), a general semi-supervised framework designed to control finite-sample FDR or FDX in both competition-based and $p$ -value-based settings.

Core Mechanism

RESET operates by splitting the data into training and estimating sets to decouple the learning of a scoring function from the error rate estimation:

Data Transformation: In the $p$ -value setting, $p$ -values are converted into a competition-like format with labels ( $L_i \in \{+1, -1\}$ ) and winning scores ( $W_i$ ) using a mirroring function. In the competition setting, hypotheses naturally possess target/decoy scores and labels.
Data Splitting: The set of "decoy wins" (hypotheses where the null hypothesis appears to win) is randomly split into:
- Training Decoys: Used to train a semi-supervised machine learning model.
- Estimating Decoys: Reserved for estimating the number of false discoveries.
- Pseudo-Targets: The union of all target wins and estimating decoys.
Semi-Supervised Learning (The "Ensemble"): A user-selected semi-supervised model (or an ensemble thereof) is trained to distinguish between the training decoys (negative class) and the pseudo-targets (positive class, which is a mixture of true and false nulls). The model outputs a rescored set of values ( $\tilde{W}$ ) intended to better rank false nulls.
Error Control: The training decoys are discarded. The original labels of the remaining pseudo-targets are revealed. The rescored hypotheses are then processed using standard step-up or step-down procedures:
- FDR Control: Selective SeqStep+ (SSS+).
- FDX Control: FDP-stepdown (FDP-SD).

RESET Ensemble

To address the challenge of model selection, the authors introduce RESET Ensemble. This implementation:

Automatically selects the best-performing classification algorithm from a library (Random Forest, Generalized Additive Models, and various Neural Networks) using cross-validation on the training data.
Utilizes a $K$ -fold scheme where each hypothesis is scored by a model not trained on it, guarding against overfitting.
Operates without requiring the user to pre-specify the optimal model, thereby preserving error control guarantees that might be compromised by data-dependent model selection.

Theoretical Guarantees

The paper establishes that RESET controls finite-sample FDR and FDX under Assumption 1: Conditioned on the scores, side information, and false null labels, the true null labels are independent $\pm 1$ random variables with $P(L_i=1) \leq c_0$ .

Lemma 1 proves that even after the data splitting and machine learning steps, the labels of the true nulls in the estimating set remain conditionally independent, satisfying the requirements for SSS+ and FDP-SD.
The framework is shown to be robust to non-identical distributions of true null labels, a relaxation of assumptions required by prior work.

Key Contributions

Unified Framework: RESET is applicable to both competition-based (e.g., knockoff filters, target-decoy competition) and $p$ -value-based multiple testing scenarios.
Finite-Sample FDX Control: To the best of the authors' knowledge, RESET is the first method to offer finite-sample FDX control in the $p$ -value setting with side information, and provides a flexible approach for FDX in competition settings. (Note: A related method, CLAW, was identified during review but only controls FDR).
Computational Efficiency: By avoiding the iterative label-revelation process of Adaptive Knockoffs and AdaPT, RESET achieves significant speedups (orders of magnitude in some cases).
Automated Model Selection: The "Ensemble" approach removes the burden of manual model selection from the user while maintaining statistical validity.
Theoretical Rigor: The paper provides proofs for finite-sample FDR and FDX control under relaxed independence assumptions, including a novel proof for FDX control with non-identical true null labels.

Results

The authors evaluate RESET across simulations and real-world datasets:

Simulation Studies

Variable Selection (Competition Setting): In simulations involving linear and logistic models with varying dimensions and side information structures, RESET Ensemble consistently demonstrated competitive or superior power compared to Adaptive Knockoffs (AdaKO) and the standard Knockoff Filter (KO). It particularly excelled in high-dimensional settings and when side information was complex.
P-value Setting: Compared against AdaPT, AdaFDR, ZAP, and others, RESET Ensemble achieved comparable or higher power across various side information configurations (geometric, high-dimensional, and dependent data). It showed robustness to local dependencies where some asymptotic methods failed.

Real Data Applications

Peptide Detection (Proteomics):
- Speed: On the HEK293 dataset, RESET Ensemble completed analysis in ~17.7 minutes. In contrast, Adaptive Knockoffs variants were estimated to take days (e.g., AdaKO GAM: ~1.07 days; AdaKO RF: ~846 days).
- Power: On the PRIDE-20 dataset, RESET Ensemble discovered more peptides than both the standard Target-Decoy Competition (TDC) and AdaKO with its default GAM filter. It was comparable to the most powerful AdaKO variant (RF) but vastly faster.
Genomics and Other Domains: Across 10 real datasets (RNA-seq, microbiome, fMRI, eQTL), RESET Ensemble consistently ranked among the top performers in terms of the number of discoveries, often outperforming or matching AdaPT variants and AdaFDR.

Significance and Claims

The paper claims that RESET offers a practical, fast, and statistically rigorous alternative to existing semi-supervised multiple testing methods.

Versatility: It uniquely bridges the gap between $p$ -value and competition-based testing, supporting both FDR and FDX control with finite-sample guarantees.
Scalability: It solves the computational bottleneck of iterative methods like Adaptive Knockoffs, making semi-supervised multiple testing feasible for massive datasets (e.g., proteomics).
User-Friendliness: By automating model selection via the Ensemble approach, it eliminates the need for users to tune complex hyperparameters or select specific algorithms, reducing the risk of suboptimal choices.
Limitations: The authors modestly note that RESET's power depends on the informativeness of the side information; if covariates are uninformative, RESET may not outperform side-information-oblivious baselines. Additionally, the random splitting of decoys introduces variability in the discovery list, though the authors suggest this is marginal relative to data randomness and can be mitigated via e-value derandomization (though this incurs a power loss).

The work positions RESET as a robust tool for modern high-throughput scientific discovery, particularly in fields like proteomics and genomics where large-scale hypothesis testing with auxiliary data is critical.