📊 statistics

Null-Calibrated Conformal Selection via Target-Membership Scores

This paper proposes Null-Calibrated Conformal Selection (NCCS), a method that utilizes target-membership probabilities instead of conventional prediction-oriented scores to achieve finite-sample valid false discovery rate control, particularly improving selection power for complex targets like variance-driven or multimodal distributions.

Original authors: Seungjin Choi

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seungjin Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a talent scout for a massive music festival. You have a huge list of thousands of unknown bands (the candidates). You need to pick the ones that will fit a very specific vibe for your main stage (the target region).

The problem is: You don't know how good the bands are yet. You have to make your picks based on their demos (the data). If you pick a band that turns out to be terrible, you've wasted money and time (a false discovery). But if you are too scared to pick anyone, you might miss the next big star (low power).

This paper introduces a new way to make these picks, called Null-Calibrated Conformal Selection (NCCS). Here is how it works, broken down into simple concepts:

1. The Old Way vs. The New Way: "Guessing the Score" vs. "Guessing the Fit"

The Old Way (Conventional Scores):
Most existing methods try to predict the exact outcome.

Analogy: Imagine you try to predict the exact decibel level of a band's volume. If your target is "loud bands," you pick the ones with the highest predicted volume.
The Flaw: This works great if "loud" is the only thing that matters. But what if your target is "bands that sound exactly like 1980s pop"? A band might be predicted to be very loud (high volume) but sound nothing like 80s pop. The old method gets confused because it's looking at the wrong thing (volume) instead of the right thing (fit).

The New Way (Target-Membership Scores):
The authors argue you shouldn't predict the outcome; you should predict the probability of fitting the target.

Analogy: Instead of guessing the volume, you ask: "What is the probability this band fits the '80s pop' vibe?"
The Benefit: This is a direct answer to your question. Whether the band is loud, quiet, or weird doesn't matter; only the "fit" matters. The paper calls this the Target-Membership Probability.

When does this matter?

Simple Cases: If your target is just "Loud bands," the old way and the new way give the same results.
Complex Cases: If your target is "Bands that are neither too loud nor too quiet" (an interval), or "Bands that are either very sad or very happy" (multimodal), the old way fails. The new way (predicting the fit) is the only one that gets it right.

2. The "Safety Net": Null-Calibrated Conformal Selection (NCCS)

Once you have a list of bands ranked by how well they "fit," you still need to decide which ones to pick without accidentally picking too many bad ones. This is where the "Calibration" comes in.

The authors propose a specific safety net called NCCS.

The Setup: You have a "Calibration Group" of bands you have already seen and know for a fact do not fit the vibe (the Null group).
The Test: You take a new band from your unknown list. You compare its "fit score" against the scores of the bands you know are bad.
The P-Value: If the new band has a "fit score" that is higher than almost all the "bad" bands, it gets a "green light" (a low p-value). If it looks just like the bad bands, it gets a "red light."

Why is this special?
Most other methods try to guess a "cutoff line" based on the data, which can be risky if the data is weird or if the "bad" bands are rare.

The Paper's Claim: NCCS provides a mathematical guarantee. It promises that even with a small amount of data, you will not pick too many "bad" bands. It's like having a safety net that is mathematically proven to hold, even if you haven't tested it a million times.

3. The Trade-Off: Safety vs. Aggression

The paper is very honest about the pros and cons:

The Aggressive Method (Direct Thresholding): Imagine a scout who says, "I'm going to pick anyone who looks 90% like a hit." This is very powerful (you catch more stars), but if you're wrong about the data, you might accidentally pick a lot of bad bands (violating your safety rules).
The NCCS Method: This scout says, "I will only pick bands that are clearly better than the ones I know are terrible."
- The Result: You might miss a few potential stars (lower power), but you are guaranteed not to pick too many duds (valid False Discovery Rate control).
- When to use it: The paper suggests using NCCS when the "bad" bands are rare or when you absolutely cannot afford to make a mistake.

Summary of the Paper's Core Message

Stop predicting the number; start predicting the fit. If you want to find things that fit a specific category, don't try to guess the exact value. Guess the probability of belonging to that category.
Use the "Bad" examples as a ruler. To decide who to pick, compare your candidates only against the examples you know are not the target.
Safety first. This new method (NCCS) might be slightly more cautious than other methods, but it comes with a strict mathematical promise that you won't make too many mistakes, even with small datasets.

The paper does not claim this method is the "best" at finding the most stars in every situation. Instead, it claims this is the safest and most principled way to do selection when you need to be sure you aren't picking the wrong things, especially when the definition of "good" is complex or tricky.

Technical Summary: Null-Calibrated Conformal Selection via Target-Membership Scores

1. Problem Statement

Conformal selection aims to identify test candidates from a pool whose unknown responses fall within a user-specified target region $I \subseteq \mathcal{Y}$ , while controlling the False Discovery Rate (FDR). The challenge lies in balancing two objectives: minimizing false discoveries (candidates selected where $Y \notin I$ ) and maximizing power (selecting as many true targets as possible).

Existing methodologies typically inherit nonconformity scores from conformal prediction, such as signed residuals, clipped residuals, or conditional quantile scores. These scores are designed for prediction tasks (estimating $Y$ ) rather than selection tasks (determining if $Y \in I$ ). The authors argue that these prediction-oriented scores can be structurally misaligned with the selection objective, particularly when the target region is an interval, variance-driven, multimodal, or defined by multiple conditions. In such cases, a high prediction score (e.g., a large conditional mean) does not necessarily imply a high probability of target membership.

2. Methodology

The Target-Membership Principle
The paper proposes a fundamental shift in score selection: the natural score for conformal selection is the target-membership probability:
$\eta_I(x) = P(Y \in I \mid X = x)$
This score treats the selection problem as a binary classification task where the label is $Z = \mathbb{I}\{Y \in I\}$ . By the Neyman–Pearson lemma, thresholding any monotone transform of $\eta_I(x)$ yields the oracle ranking for maximizing target selection probability at a fixed null selection level.

Score-Target Alignment Characterization
The paper rigorously characterizes when conventional scores suffice and when they fail:

Mean-Monotone Targets: For one-sided targets (e.g., $I = [c, \infty)$ ) where target membership is monotone in the conditional mean, conventional scores (residuals, likelihood ratios) induce the same ranking as $\eta_I(x)$ . In these regimes, the new principle recovers existing practice without offering a power advantage.
Misaligned Targets: For interval targets, variance-driven targets, multimodal distributions, or multi-condition targets, conventional scores often fail to align with $\eta_I(x)$ . For instance, in a variance-driven interval target, the conditional mean may contain no information about membership, whereas $\eta_I(x)$ captures the relevant signal.

Null-Calibrated Conformal Selection (NCCS)
To operationalize the membership score, the paper introduces NCCS, a specific calibration route that converts the ranking into finite-sample valid p-values.

Score Learning: A target-membership score $g(x)$ is learned on a dedicated split ( $D_{sc}$ ) using binary labels $Z_i = \mathbb{I}\{Y_i \in I\}$ . The paper suggests using cost-sensitive logistic regression or gradient-boosted trees.
Null Calibration: Unlike standard conformal selection which compares test scores against all calibration examples, NCCS compares test scores only against confirmed non-target calibration examples ( $C_0 = \{i \in D_{cal} : Y_i \notin I\}$ ).
P-Value Construction: For a test candidate $j$ , the null-calibrated p-value is:
$p_j(g) = \frac{1 + \#\{i \in C_0 : g(X_i) \ge g(X_{N+j})\}}{|C_0| + 1}$
Selection: These p-values are fed into a multiple-testing procedure (e.g., Benjamini–Hochberg or Benjamini–Yekutieli) to select the final set.

3. Key Contributions

Target-Membership Principle: The paper establishes that conformal selection should be viewed as a classification problem for the event $Y \in I$ . It identifies $\eta_I(x)$ as the population oracle ranking score, providing a theoretical basis for score selection that is distinct from prediction-oriented scores.
Alignment Characterization: It precisely delineates the boundary where conventional scores work (mean-monotone targets) versus where they fail (interval, variance-driven, multimodal targets), explaining why membership scoring is the "canonical repair" for the latter.
Finite-Sample Validity via NCCS: The paper isolates a specific calibration route (NCCS) that guarantees finite-sample null p-value validity under null exchangeability. This distinguishes it from direct empirical-FDP thresholding (e.g., Qin et al., 2025), which controls FDR only asymptotically and can be anti-conservative when targets are rare.
Empirical Validation: Experiments confirm the theoretical predictions:
- On mean-monotone targets, membership scores perform similarly to conventional scores.
- On variance-driven interval targets, membership scores substantially outperform mean-score selection (which collapses).
- NCCS maintains strict FDR control in finite samples, whereas direct thresholding can exceed the nominal FDR level, particularly in rare-target regimes.

4. Results and Limitations

Experimental Findings

Mean-Monotone Regime: In an upper-tail target setting, NCCS-BH (using Benjamini-Hochberg) was more conservative than clipped cfBH or direct Neyman-Pearson (Qin-NP) thresholding, resulting in lower power but similar FDR control. This confirms that the principle does not artificially inflate power where conventional methods already work.
Variance-Driven Regime: In an interval target setting driven by conditional variance, mean-score methods (Mean-Null-BH) failed completely (power $\approx 0$ ). NCCS-BH retained substantial power ($0.288$) compared to the oracle ($0.354$), significantly outperforming mean-based approaches.
Rare Targets: When target frequency was low, direct empirical-FDP thresholding (Qin-NP) exhibited inflated FDR (e.g., 0.170 vs. nominal 0.10), while NCCS-BH remained strictly below the nominal level, demonstrating its robustness in finite-sample rare-event scenarios.

Modest Claims and Limitations
The authors explicitly state that NCCS is not presented as a universal power winner.

Power Trade-off: NCCS is often more conservative than direct Neyman-Pearson thresholding because it relies on finite-sample p-value validity rather than asymptotic FDP estimation.
Rare Targets: Learning the membership score itself can be difficult when the target event is rare, potentially requiring distributional or Bayesian models for efficiency.
Exchangeability: The finite-sample validity of NCCS relies on the assumption of null exchangeability. Under covariate shift or label shift, the null calibration set may no longer represent the null test candidates, necessitating weighted calibration (a direction left for future work).
Scope: The paper focuses on the principle and its characterization rather than promoting a single method as superior in all contexts. The contribution is the identification of the correct score for general targets and a calibration route with a specific validity guarantee.

5. Significance

The paper's significance lies in reframing conformal selection as a classification problem, thereby identifying the target-membership probability as the theoretically optimal score. It provides a clear, testable boundary for when this new approach is necessary (non-monotone targets) and when it is redundant. Furthermore, it introduces NCCS as a practical method that prioritizes finite-sample validity over asymptotic power, offering a robust solution for selection problems where controlling false discoveries in small samples or rare-event regimes is critical.