📊 statistics

Unified Conformalized Multiple Testing with Full Data Efficiency

This paper proposes a unified framework for conformalized multiple testing that maximizes data efficiency by utilizing all available data (null, alternative, and unlabelled) through a full permutation strategy to construct superior scores and calibrate p-values, thereby significantly improving statistical power while rigorously controlling the false discovery rate without requiring extra data splitting.

Original authors: Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren

Published 2026-05-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuyang Huo, Xiaoyang Wu, Changliang Zou, Haojie Ren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to find a few specific suspects (the "non-nulls") hidden in a massive crowd of innocent bystanders (the "nulls"). Your goal is to point out the suspects without falsely accusing too many innocent people. In statistics, this is called multiple testing, and the rule you must follow is to keep your "False Discovery Rate" (FDR) low—meaning you can't accuse too many innocent people just to catch a few real suspects.

For a long time, detectives (statisticians) have had a special tool called Conformalized Testing to help them. It's like a magic magnifying glass that guarantees you won't make too many mistakes, no matter how the crowd is arranged. However, there was a catch: to use this magnifying glass safely, detectives had to throw away a huge chunk of their evidence. They had to split their data into "training" and "testing" piles, leaving a lot of useful information on the table.

This paper, titled "Unified Conformalized Multiple Testing with Full Data Efficiency," proposes a new, smarter way to use the magnifying glass. Here is the breakdown in simple terms:

1. The Old Way: The "Split the Pizza" Problem

Imagine you have a whole pizza (your data) and you want to find the pepperoni slices (the suspects).

Old Method: You cut the pizza in half. You use one half to learn what pepperoni looks like, and you use the other half to check if the slices you found are actually pepperoni.
The Problem: You only get to use half the pizza to learn and half to check. If the pizza is small, you might miss the pepperoni or get confused. Also, different detectives had different ways of cutting the pizza, and it was hard to compare who was doing the best job.

2. The New Way: The "Full Permutation" Party

The authors, Huo, Wu, Zou, and Ren, propose a unified framework called ECOT (Enhanced COnformal Testing). Instead of cutting the pizza, they say: "Let's look at the whole pizza, but we have to play a specific game to stay fair."

The Game (Permutation): Imagine you have a deck of cards representing your data. To check if a specific card is a "suspect," you shuffle the deck in every possible way (permutation) while keeping the rules of the game strict. You ask: "If I shuffle the cards randomly, how often does this card look like a suspect compared to the others?"
The Magic: By using all the data (the suspects you already know, the innocent people you know, and the mystery crowd) to build your "suspect detector," you get a much sharper tool. Because you use the whole deck to play the shuffling game, you don't have to throw away any data.

3. Three Big Wins for the New Method

A. Using Every Clue (Full Data Efficiency)

In the old days, if you had a list of known suspects (positive data) and a list of known innocents (negative data), you often had to leave the known suspects out of the "learning" phase to stay safe.

The New Trick: ECOT lets you use everything. You use the known suspects to teach the detector what to look for, and you use the known innocents to calibrate the "alarm." This makes the detector much smarter and more likely to catch the real suspects without raising false alarms.

B. The "Auto-Pilot" Selector (Adaptive Selection)

Sometimes, you don't know which type of detector works best. Maybe a "Binary Detector" (which looks for differences between two groups) is best for one case, but a "One-Class Detector" (which just looks for things that look weird) is better for another.

The Old Problem: If you try both detectors on the same data to see which is better, you "cheat" by looking at the answer twice, which ruins your safety guarantee.
The New Trick: ECOT has a built-in "Auto-Pilot." It tries all the detectors inside the shuffling game. It picks the best one for each specific person in the crowd without ever "peeking" at the final answer outside the rules. It automatically switches strategies to get the best results while keeping the safety guarantee intact.

C. One Big Rulebook (Unified Framework)

Before this paper, there were many different papers with different names for similar methods (like "AdaDetect," "Integ," "FullND"). It was like having five different rulebooks for the same game.

The New Trick: The authors showed that all these different methods are actually just special versions of their one big "Full Permutation" game. If you follow their rules, you can recreate all the old methods, but you can also build new, better ones that use data more efficiently.

4. What the Experiments Showed

The authors ran thousands of simulations and tested their method on real-world datasets (like spotting credit card fraud or satellite anomalies).

The Result: Their method consistently found more "suspects" (higher power) than the old methods, while still keeping the false accusations (FDR) under control.
The Trade-off: The only downside is that shuffling the deck in every possible way takes a bit more computer time. However, they showed that even with this extra time, the method is fast enough for practical use and much faster than previous "full data" attempts.

Summary Analogy

Think of the old methods as a detective who is only allowed to use a flashlight on half the room to find a thief, while the other half is dark.
The new ECOT method is like a detective who turns on the lights for the entire room to find the thief. To make sure they don't get dizzy and accuse the wrong person, they use a special "shuffling" technique to prove that their findings are solid. They can even switch between different types of flashlights automatically to find the thief faster, all while following a single, strict rulebook that guarantees they won't make too many mistakes.

In short: This paper gives statisticians a way to use all their data to make better decisions, without breaking the safety rules that prevent false accusations.

Technical Summary: Unified Conformalized Multiple Testing with Full Data Efficiency

1. Problem Statement

Conformalized multiple testing provides a model-free framework for controlling the False Discovery Rate (FDR) when identifying test samples with unobserved labels of interest (e.g., outlier detection, sample selection). The standard procedure involves constructing a non-conformity score, calibrating it against a set of null (negative) samples to generate conformal p-values, and applying a multiple testing rule (e.g., Benjamini-Hochberg).

Existing methods face two primary limitations regarding data utilization:

Partial Data Usage: Many approaches split the available null data ( $D_0$ ) into training and calibration sets, discarding valuable information. Others fail to utilize available non-null (positive) labelled data ( $D_1$ ) or unlabelled test data ( $D_u$ ) during score construction.
Fragmented Frameworks: Different methods (e.g., split conformal, integrative conformal, localised conformal) are developed under specific settings and data-utilization schemes. There is no unified theoretical framework that encompasses these methods or provides a systematic principle for designing new ones that fully leverage all available data ( $D_0, D_1, D_u$ ) while rigorously controlling FDR.

The central challenge is to develop a unified approach that maximizes statistical power by using all available data for both score construction and calibration, without violating the exchangeability assumptions required for finite-sample FDR control.

2. Methodology: The ECOT Framework

The authors propose Enhanced COnformal Testing (ECOT), a unified framework based on a full permutation strategy. The core insight is to view conformalized multiple testing through the lens of permutation testing, where the p-value for a test sample $j$ is computed by comparing its score against scores obtained from all possible permutations of the data.

2.1 Core Components

The ECOT procedure consists of three steps:

Score Construction: For each test sample $j \in U$ , an individualized score function $S^{(j)}$ is constructed using all available data ( $D_0, D_1, D_u$ ). A higher score indicates stronger evidence against the null hypothesis.
P-value Computation: A conformal p-value is computed via full permutation:
$p_j = \frac{1}{|\Omega_j|} \sum_{\sigma \in \Omega_j} \mathbb{I}\{S^{(j)}(X_j) \le S^{(j)}_\sigma(X_{\sigma(j)})\}$
Here, $\Omega_j$ is the set of permutations of the calibration set $C \subseteq L_0$ and the test sample $j$ , while fixing indices outside this set. $S^{(j)}_\sigma$ denotes the score function constructed on the permuted dataset.
Testing Procedure: To handle the dependence among p-values induced by data reuse and individualized scores, ECOT employs a conditional calibration step (inspired by Fithian and Lei, 2022). This involves defining modified p-values $\tilde{p}^{(j)}_\ell$ using a permutation-invariant summary (e.g., the median) of the scores, followed by a rejection procedure that ensures finite-sample FDR control.

2.2 Computational Simplifications

Full permutation is computationally intensive ( $O((|C|+1)!)$ ). The framework identifies two symmetry conditions under which the procedure simplifies significantly:

Calibration-Symmetric Scores: If $S^{(j)}$ is invariant under permutations of the calibration set and $j$ , the p-value reduces to a simple rank-based calculation over $C \cup \{j\}$ , eliminating the need for repeated model retraining.
Joint-Symmetric Scores: If a single score function $S$ is used for all samples and is symmetric with respect to all null and test samples, the conditional calibration step becomes unnecessary, and the standard Benjamini-Hochberg (BH) procedure applied to the simplified p-values is sufficient.

2.3 Enhanced Approaches

Leveraging the unified framework, the authors propose two specific enhanced methods:

ECOT-bi (Binary Classification): Utilizes $D_0, D_1$ , and $D_u$ jointly to train a binary classifier. The classifier distinguishes between non-null samples ( $D_1$ ) and the pooled null/unlabelled samples ( $D_0 \cup D_u$ ). This approximates the optimal density ratio without splitting $D_0$ .
ECOT-oc (One-Class Classification): Designed for scenarios where binary classifiers are inefficient (e.g., variance differences). It trains one-class classifiers on $D_0 \cup D_u$ (for the null) and $D_1$ (for the alternative), integrating them to form the score.

2.4 Adaptive Selection

The framework enables Adaptive Approach Selection (ECOT-as). Instead of fixing a single method, the algorithm evaluates multiple candidate approaches (e.g., ECOT-bi, ECOT-oc, FullND) and selects the one yielding the highest power (number of rejections) for each sample. Crucially, the selection step is embedded into the score construction via the full permutation strategy. This treats the selection as part of the score generation, preserving exchangeability and avoiding the "double dipping" problem that typically inflates FDR in post-selection inference.

3. Key Contributions

Unified Framework: The paper introduces ECOT, a general framework that encompasses existing conformalized multiple testing methods (e.g., split conformal, integrative, localised) as special cases under specific symmetry structures.
Full Data Efficiency: The framework provides a principled design strategy to use all available data ( $D_0, D_1, D_u$ ) for score construction while retaining the entire null set for calibration. This is shown to improve statistical power by enhancing score quality and maximizing the calibration set size.
Adaptive Selection with FDR Control: The authors develop a data-driven strategy to automatically select the best conformal testing approach from a set of candidates. By integrating selection into the permutation-based score construction, they achieve finite-sample FDR control without requiring extra data splitting.
Theoretical Guarantees: The paper proves that under standard data exchangeability assumptions, the ECOT procedure controls the FDR at the desired level $\alpha$ in finite samples, regardless of the complexity of the score functions used.

4. Experimental Results

Extensive numerical experiments were conducted on synthetic data and real-world outlier detection datasets (Credit Card, Satellite, Shuttle, CovType, Mammography).

Power Improvement: In scenarios where binary classification is optimal, ECOT-bi and ECOT-as consistently outperformed existing baselines (e.g., AdaDetect, CP-bi, Integ) by achieving higher power while maintaining FDR control. This was attributed to the full utilization of $D_1$ and the avoidance of data splitting.
Robustness to Scenario: In settings where one-class classification is superior (e.g., variance shifts), ECOT-oc significantly outperformed integrative methods by utilizing $D_u$ in the score construction.
Adaptive Selection: ECOT-as successfully tracked the best-performing method across diverse scenarios, achieving power close to the theoretical oracle in many cases.
Real-Data Performance: On real datasets, the ECOT methods (particularly ECOT-bi and ECOT-as) demonstrated superior power compared to baselines, which often suffered from low power due to limited labelled data or inappropriate score constructions.
Computational Cost: While the full permutation approach is computationally heavier than split methods, the adjusted adaptive selection (Algorithm 3) and symmetry reductions keep runtimes within practical limits (approx. 1 minute for moderate sample sizes), significantly faster than full conformal implementations of competing frameworks like OptCS.

5. Significance and Claims

The paper claims that ECOT represents a significant advancement by unifying the fragmented landscape of conformalized multiple testing. Its primary significance lies in:

Systematic Design: Providing a clear, principled path for designing new conformal procedures that maximize data efficiency.
Automatic Adaptation: Solving the problem of selecting the optimal testing strategy without compromising FDR control, a common pitfall in post-selection inference.
Practical Efficiency: Demonstrating that full data utilization leads to tangible gains in power across diverse real-world and synthetic scenarios, making conformal testing more effective for applications like outlier detection and sample selection.

The authors acknowledge that the framework relies on data exchangeability and that full permutation can be computationally intensive, suggesting future work in relaxing assumptions (e.g., non-exchangeable data) and accelerating permutation-based inference.