📊 statistics

Robust Conformalized Selection with Noisy Responses

This paper proposes Robust Conformalized Selection (RCS), a unified framework that ensures valid false discovery rate control and maintains statistical power in candidate selection tasks by addressing the challenge of noisy calibration data through a novel statistical reduction that transforms label contamination into a localized covariate shift problem.

Original authors: Chengyao Yu, Hongxin Wei, Bingyi Jing

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Chengyao Yu, Hongxin Wei, Bingyi Jing

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the head judge at a massive, high-stakes talent show. You have a giant list of thousands of contestants, and you need to pick the top 100 to advance to the finals. To make your job easier, you hire a super-smart AI assistant to scan the crowd and tell you who the winners are. But here's the catch: the AI isn't perfect, and the "scorecards" it used to learn from its training data were scribbled on by a mischievous gremlin. Some of the scores are wrong, some are blurry, and some are just made up.

In the world of data science, this is a common nightmare. Scientists use a technique called "conformalized selection" to pick the best candidates from huge datasets—like finding the right drug molecules or labeling millions of photos. This method is like a safety net; it promises that if you pick a certain number of candidates, you won't accidentally pick too many "fakes" (a concept statisticians call controlling the "False Discovery Rate"). However, this safety net was built on a fragile assumption: that the training data the AI learned from was perfectly clean. If that data is "contaminated" (noisy, wrong, or tampered with), the safety net can snap, letting too many bad candidates through, or it can become so tight that it rejects everyone, leaving you with an empty stage.

This paper, titled "Robust Conformalized Selection with Noisy Responses," tackles the problem of what happens when that safety net is tested against messy, real-world data. The authors, Chengyao Yu, Hongxin Wei, and Bingyi Jing, propose a new, tougher safety net called Robust Conformalized Selection (RCS). Instead of panicking when the data is noisy, RCS treats the noise like a specific type of "shift" in the crowd. It uses a clever statistical trick to adjust for the messiness, essentially saying, "Okay, the scorecards are a bit off, but if we look at the patterns of how they are off, we can still pick the winners reliably." Through simulations and real-world tests, the authors show that while old methods either fail to control the errors or become too cautious to be useful, RCS manages to keep the error rate low while still finding a lot of good candidates. It's a way to make sure your talent show picks the best acts, even if the judges' notes were written in crayon.

The Problem: The "Noisy" Scorecard

Let's dive into why this matters. In many scientific fields, from discovering new medicines to training AI to understand human language, researchers have to sift through massive pools of possibilities. They can't check every single one by hand because it's too expensive or takes too long. So, they rely on machine learning models to predict which ones are good.

To make sure these predictions are trustworthy, scientists use a method called Conformalized Selection. Think of it like a quality control checkpoint. The model looks at a "calibration set" (a group of examples where we know the answers) to learn how to set a threshold. If a new candidate's score is high enough, they get selected. The magic of this method is that it guarantees a specific limit on how many "wrong" picks you'll make (the False Discovery Rate, or FDR).

But there's a huge flaw in the old methods: they assume the calibration set is perfect. In the real world, data is rarely perfect.

Crowdsourcing: When you ask thousands of people on the internet to label photos, some might be tired, some might not understand the task, and some might just guess.
Privacy: Sometimes, to protect people's privacy (like in medical records), the data is intentionally scrambled or "randomized" before anyone sees it.
Lab Errors: In drug discovery, chemical tests can have glitches or variations that make the results slightly wrong.

When you feed this "contaminated" or "noisy" data into the old selection methods, the math breaks. The authors found that the old methods either let too many bad candidates through (failing to control the FDR) or became so paranoid that they rejected almost everyone (losing "power," which is the ability to find the good stuff).

The Solution: The "RCS" Detective

The authors introduce Robust Conformalized Selection (RCS) to fix this. Their key insight is a bit like a detective realizing that the "noise" isn't just random chaos; it follows a pattern.

Imagine you are trying to find the best apples in a barrel, but someone has painted some of the apples a slightly different shade of red. The old method would just look at the color and get confused, either picking too many painted apples or missing the real ones.

RCS, however, looks at the problem differently. It says, "Let's group the apples by the color the AI thinks they are." If the AI thinks an apple is "Red," RCS looks at all the "Red" apples in the training barrel. It then calculates a special "weight" for each apple. This weight answers a specific question: Given that the AI thinks this is a Red apple, how likely is it that the label is actually wrong because of the noise?

By using these weights, RCS translates the messy "label noise" problem into a "covariate shift" problem. In plain English, it's like realizing that the noise isn't random; it's a systematic shift that can be measured and corrected. They use a statistical approach called "empirical-Bayes" to estimate how many false picks they are likely to make, adjusting for the noise in real-time.

What They Found

The authors didn't just guess; they tested this rigorously.

Simulations: They created fake datasets where they knew exactly how much noise was in the data (ranging from 0% to 20% noise). They compared RCS against the old methods (like "PSP" and "cfBH").
- The Result: The old methods either let the error rate skyrocket (failing to control the FDR) or became so conservative they found almost nothing. RCS, on the other hand, kept the error rate right where it was supposed to be (around the target level, like 5% or 10%) while still finding a huge number of correct candidates. In some cases, RCS was significantly more powerful than the old methods, finding many more "winners" without letting in the "losers."
Real-World Tests: They tried RCS on two real datasets:
- CIFAR-10H: A set of 10,000 images where the labels were provided by human workers on Amazon Mechanical Turk (known to be noisy).
- ACS Income Data: A dataset of US income records where they simulated "differential privacy" (intentionally scrambling the data to protect privacy).
- The Result: In both cases, RCS successfully controlled the error rate and found more high-quality candidates than the standard methods. Even when they didn't know the exact nature of the noise (mis-specified models), RCS remained robust and didn't crash.

Why It Matters

This paper doesn't claim to have solved every data problem in the universe. It specifically addresses the gap where existing methods fail because they assume perfect data. The authors show that by acknowledging the noise and adjusting for it mathematically, we can still make reliable, large-scale selections.

They proved that RCS works for two main types of tasks:

Classification: Picking the items that are correctly labeled (like finding the right drug or the right image).
Threshold Selection: Picking items that have a value above a certain line (like finding drugs that bind strongly enough to a target).

The authors emphasize that their method is "robust," meaning it works even if you don't know the exact details of how the data got messed up, as long as you can estimate the general pattern of the noise. They also showed that their method is "optimal," meaning it finds as many good candidates as theoretically possible given the constraints.

In short, if you are a scientist or a data analyst trying to pick the best candidates from a messy, noisy pile of data, RCS offers a new, reliable way to do it without throwing up your hands and saying, "The data is too dirty to use." It turns the "dirty data" problem into a solvable math puzzle, ensuring that your final list of winners is actually worthy of the prize.

Technical Summary: Robust Conformalized Selection with Noisy Responses

Problem Formulation

The paper addresses the critical challenge of conformalized selection when calibration data contains contaminated (noisy) responses. While conformalized selection methods (e.g., Jin and Candès, 2023; Sun and Xia, 2025) have been widely adopted for tasks like reliable automatic labeling and drug discovery, they fundamentally rely on the assumption that calibration data responses are clean and exchangeable with test data. In practice, however, responses are often corrupted due to:

Crowdsourced annotation errors (e.g., Amazon Mechanical Turk).
Differential privacy mechanisms (randomized responses).
Assay variability in scientific domains like drug discovery.

The authors formulate two primary tasks under this noisy setting:

Task 1 (Trustworthy Classification): Selecting test samples where the predicted label $\hat{Y}$ matches the true label $Y$ , while controlling the False Discovery Rate (FDR) for specific groups of classes.
Task 2 (Threshold Selection): Selecting test samples where the unobserved response $Y$ exceeds a certain threshold $c$ , while controlling FDR.

The paper demonstrates that existing methods (specifically cfBH for Task 2 and PSP for Task 1) fail under contamination: they either suffer from inflated FDR (violating the reliability guarantee) or experience severe power loss (becoming overly conservative and failing to select promising candidates).

Methodology: Robust Conformalized Selection (RCS)

The core contribution is Robust Conformalized Selection (RCS), a unified framework designed to maintain valid FDR control and maximize power under general label contamination.

Key Insight: Statistical Reduction

The fundamental innovation of RCS is a statistical reduction that translates the intractable problem of label noise into a localized covariate shift problem.

Instead of treating noise globally, RCS conditions on the predicted class $\hat{Y} = k$ .
It defines localized weights $w_k(x)$ that adjust for the discrepancy between the distribution of clean errors and noisy errors within each predicted class:
$w_k(x) = \frac{P(Y \neq k | X = x)}{P(\tilde{Y} \neq k | X = x)}$
By incorporating these weights, the method transforms the estimation of the False Discovery Proportion (FDP) into a covariate-adjusted empirical-Bayes estimate. This allows the algorithm to correct for the bias introduced by noisy calibration labels without requiring the clean labels to be observed.

Algorithmic Implementation

Uncertainty Scoring: A pre-trained classifier $\hat{f}$ provides predicted labels $\hat{Y}$ and an uncertainty score $U(X)$ (e.g., $1 - \hat{\mu}_{\hat{Y}}(X)$ ).
Group-wise Thresholding: For a target group of classes $A_g$ , RCS estimates the FDP at a threshold $t$ using the weighted calibration data:
$\widehat{FDP}_{A_g}(t) = \frac{\sum_{k \in A_g} \frac{|S^{test}_k|}{|S^{cal}_k|} \sum_{i \in S^{cal}_k} w_k(X_i) \mathbb{I}\{\tilde{Y}_i \neq k, U_i \leq t\}}{1 \vee \sum_{k \in A_g} \sum_{j \in S^{test}_k} \mathbb{I}\{U_{n+j} \leq t\}}$
where $S^{cal}_k$ and $S^{test}_k$ are sets of calibration and test samples predicted as class $k$ .
Threshold Selection: The algorithm selects the largest threshold $t$ such that $\widehat{FDP}_{A_g}(t) \leq \alpha_{A_g}$ .
Handling Specific Models:
- Randomized Response: For this common model, the framework simplifies significantly, removing the need for explicit weight estimation by adjusting the target FDR level directly.
- Task 2 Reduction: Task 2 is reduced to a binary classification instance (selecting $Y > c$ vs. $Y \leq c$ ), allowing the application of the Task 1 RCS framework.

Theoretical Properties

The paper establishes rigorous theoretical guarantees for RCS:

Asymptotic FDR Control: Under mild regularity conditions, RCS controls the group-wise FDR at the target level $\alpha$ as sample sizes $n, m \to \infty$ , even with estimated weights.
Power Optimality: The method achieves optimal asymptotic power (recovery rate) among all valid selection rules, provided the uncertainty score is chosen optimally (e.g., $U^*(X) = 1 - P(Y = \hat{f}(X) | X)$ ).
Robustness: The framework remains valid even when the contamination model (transition matrix $T$ ) is estimated from data or slightly mis-specified, provided the estimates are consistent.

Experimental Results

Extensive experiments on simulated and real-world datasets validate the theoretical claims:

Simulated Classification (Task 1):
- Under randomized response and asymmetric label noise, existing methods (PSP) either lose power drastically or fail to control FDR.
- RCS consistently maintains FDR close to the nominal level (e.g., 0.1) while retaining high power across varying noise levels ( $\epsilon$ ) and sample sizes.
- RCS remains robust even when the contamination model is mis-specified (e.g., assuming a block-structured noise model when the true noise is different) or when the transition matrix is estimated from a small auxiliary dataset.
Simulated Threshold Selection (Task 2):
- In scenarios involving constant status-transition contamination and additive Gaussian noise, the baseline method (BH clip) suffers severe power loss.
- RCS (including versions with estimated weights) significantly outperforms BH clip in power while strictly controlling FDR.
Real-World Applications:
- CIFAR-10H (Image Classification): Using imperfect labels from crowdsourced annotators, RCS selects correctly classified images with higher power than PSP while controlling FDR at 0.05.
- ACS Income Data (Differential Privacy): In a setting simulating label differential privacy, RCS successfully identifies high-income individuals. It maintains stable performance and high power even as the privacy budget $\epsilon_{DP}$ decreases (increasing noise), whereas the baseline (BH clip) degrades significantly.

Significance and Claims

The paper claims to fill a critical gap in the literature by providing the first unified framework for conformalized selection under contaminated calibration data. Its significance lies in:

Practical Applicability: It enables rigorous uncertainty quantification in real-world scenarios where clean labels are expensive or impossible to obtain (e.g., privacy-preserving data, crowdsourcing).
Theoretical Rigor: It moves beyond the "clean data" assumption of prior work, offering provable FDR control and power optimality under general noise models.
Robustness: The method does not require perfect knowledge of the noise mechanism; it functions effectively with estimated or bounded noise parameters.

The authors position RCS not merely as a correction to existing methods but as a necessary evolution for deploying conformalized selection in noisy, real-world environments. They note that while the current work focuses on classification and threshold selection, the underlying principles of RCS could potentially extend to other conformalized selection settings (e.g., multivariate or hierarchical selection) in future research.