📊 statistics

Conformal C2ST: Turning weak classifiers into strong two-sample tests

This paper introduces Conformal C2ST, a theoretically grounded framework that transforms any trained classifier, even weak or biased ones, into a reliable two-sample test with guaranteed finite-sample Type-I error control and non-trivial power, demonstrating particular effectiveness in validating Neural Posterior Estimation models.

Original authors: Vansh Bansal, Tianyu Chen, James G. Scott

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vansh Bansal, Tianyu Chen, James G. Scott

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if two groups of people are actually from the same town or if they are impostors from a different one. In the world of statistics and machine learning, this is called a two-sample test. You have a "True" group (drawn from distribution $p$ ) and a "Fake" group (drawn from distribution $q$ ), and you need to decide: Are they the same, or are they different?

For a long time, the standard way to solve this was to hire a super-detective (a highly trained AI classifier). This detective's job is to look at the two groups and learn exactly how to tell them apart.

If the detective is perfect, they can spot the difference instantly.
If the groups are actually the same, the detective will be confused and guess randomly.

The Problem:
The old method had a huge flaw: it relied entirely on the detective being a genius. If you hired a weak detective (one who is tired, poorly trained, or just bad at their job), the test would fail.

If the groups were actually different, a weak detective might miss it and say, "They look the same!" (False Negative).
If the groups were the same, a weak detective might get confused and say, "They look different!" (False Positive).

The old rule was: "If you don't have a perfect detective, you can't trust the test."

The Solution: Conformal C2ST
This paper introduces a new method called Conformal C2ST. It's like turning a weak, unreliable detective into a super-reliable judge using a clever trick called Conformal Calibration.

Here is how it works, using a simple analogy:

The "Ranking" Trick

Instead of asking the detective, "Is this person from Town A or Town B?" (a hard Yes/No decision), the new method asks the detective to simply rank the people.

The Setup: You take a group of "True" people (the calibration set) and one "Test" person.
The Ranking: You ask the detective to score everyone. Even if the detective is weak, they might still be able to say, "This True person feels more like a True person than that Test person," or vice versa. They don't need to be 100% right; they just need to be better than random guessing.
The Magic: The method looks at where the Test person's score falls compared to the True people.
- If the Test person is an impostor, their score will usually be very different from the True group.
- The method calculates a "p-value" (a probability score) based on this relative rank.

Why This Changes Everything

The paper proves two amazing things about this new method:

It's Unbreakable (Validity): Even if the detective is terrible, the math guarantees that if the two groups are actually the same, the test will never falsely accuse them of being different more than a tiny, controlled amount (usually 5% of the time). It's like having a judge who never convicts an innocent person, even if the witness is a bit shaky.
It's Still Powerful (Robustness): Even if the detective is weak, as long as they can rank things slightly better than a coin flip, the test can still detect differences.
- The Paper's Analogy: Imagine the detective's decision line is a fence separating two fields.
  - Old Method: If the fence is moved even a little bit (a weak classifier), the test breaks and can't tell the fields apart.
  - New Method: Even if the fence is moved, tilted, or rotated (a bad classifier), the new method looks at the order of the people relative to the fence. It can still see that the groups are different, even if the fence isn't in the perfect spot.

Real-World Application in the Paper

The authors tested this specifically on Neural Posterior Estimation (NPE).

The Scenario: Imagine a scientist using a computer model to guess the location of a hidden object (like a planet or a disease source) based on noisy data. The computer gives a "best guess" distribution.
The Test: How do you know if the computer's guess is accurate? You compare the computer's guesses against the "true" reality.
The Result: The new Conformal C2ST method was able to spot subtle errors in the computer's guesses that the old methods missed. Crucially, it worked even when the "detector" AI used to compare them was weak or poorly trained.

The Takeaway

The paper's main message is simple: You don't need a perfect AI to check if your model is good.

In the past, if your AI was weak, you couldn't trust your validation tests. Now, with Conformal C2ST, you can take a weak, imperfect, or even slightly broken classifier, run it through this "ranking and calibration" process, and get a reliable, mathematically guaranteed answer about whether your model is working correctly. It turns a "weak signal" into a "strong test."

Technical Summary: Conformal C2ST

Problem Statement

The paper addresses the two-sample testing problem, a fundamental task in statistics and machine learning aimed at determining whether two sets of samples, drawn from underlying distributions $p$ and $q$ , are identically distributed ( $p=q$ ). A primary application is evaluating generative models and Neural Posterior Estimation (NPE) models, where $p$ represents the true data-generating process (or true posterior) and $q$ is a learned approximation.

The standard approach, the Classifier Two-Sample Test (C2ST), trains a classifier to distinguish between samples from $p$ and $q$ . If the classifier achieves high accuracy, the distributions are deemed different. However, the paper identifies a critical limitation: the reliability and power of C2ST hinge on the availability of a near-Bayes-optimal classifier. In practice, classifiers are often weak, biased, overfit, or poorly calibrated due to limited data or model capacity. This creates an ambiguity: if a model "passes" a C2ST, it is unclear whether the distributions are truly identical or if the classifier simply lacks the power to detect the difference. Existing theoretical guarantees for C2ST often assume access to an optimal or asymptotically consistent classifier, a condition rarely met in real-world scenarios like NPE.

Methodology

The authors propose the Conformal C2ST, a framework that converts the scores of any trained classifier—regardless of its weakness or bias—into exact, finite-sample $p$ -values. The method builds upon the conformal prediction framework of Hu & Lei (2024) but adapts it for the specific regime of evaluating accessible generative models where marginal distributions may be equal.

The core mechanism involves conformal calibration rather than hard thresholding:

Scoring: A classifier $f$ is trained to distinguish $p$ from $q$ , producing a score $s(x)$ (e.g., log-odds) for any input $x$ .
Conformal $p$ -value: For a test point $\tilde{X} \sim q$ , a conformal $p$ -value $U$ is computed by comparing its score against a set of calibration scores drawn from $p$ . The $p$ -value represents the relative rank of the test point's score among the calibration set.
Aggregation: To test the global hypothesis $H_0: p=q$ , this procedure is repeated for multiple independent test points from $q$ . The resulting collection of $p$ -values is aggregated (e.g., using a Kolmogorov–Smirnov test) to assess deviation from uniformity.

The paper introduces two specific variants:

Conformal Uniform Test: Uses a fresh calibration set for each test point. This offers exact finite-sample validity and maximizes power but requires additional sampling from $p$ .
Conformal Multiple Test: Uses a single shared calibration set for all test points. This matches the computational budget of the standard C2ST (one sample from $p$ and one from $q$ per pair) while retaining robustness.

Key Contributions

Theoretical Validity: The authors establish that the Conformal C2ST provides exact finite-sample Type-I error control under the null hypothesis ( $p=q$ ) for any deterministic scoring function. This validity holds regardless of the classifier's quality, effectively decoupling error control from discriminative accuracy.
Robustness to Weak Classifiers: The paper proves that the test maintains non-trivial power under the alternative ( $p \neq q$ ) even when the classifier is weak or misspecified. Theoretical results show that the power degrades gently (as $O(\epsilon^{2/3})$ ) with the estimation error of the density ratio, provided the classifier preserves a reasonable ranking signal (measured by AUC).
Ranking over Accuracy: The methodology shifts the focus from classification accuracy (thresholding at 0.5) to ranking quality. The test relies on the relative ordering of scores (AUC), which is more robust to miscalibration and weak decision boundaries than absolute probability estimates.
Practical Variants: The introduction of the "Multiple" test allows for robust two-sample testing without increasing the simulation budget compared to standard C2ST, making it feasible for expensive simulators.

Experimental Results

The authors evaluate the Conformal C2ST against baselines including standard C2ST, Simulation-Based Calibration (SBC), Test of Accuracy with Random Points (TARP), Discriminative Calibration (DC), and kernel-based tests (MMDFuse) across several benchmarks:

Controlled Perturbations: In experiments with perturbed Gaussian posteriors (mean shifts, covariance scaling, heavy tails), the Conformal C2ST variants consistently outperformed standard C2ST and other baselines in detecting subtle discrepancies. Notably, DC failed to control Type-I error in these settings.
Classifier Degradation: The authors systematically degraded classifier quality by interpolating trained parameters with random initialization. Results showed that standard C2ST power collapsed rapidly as the classifier weakened. In contrast, Conformal C2ST maintained high power, demonstrating resilience to biased or poorly trained classifiers.
High-Dimensional Images: On CIFAR-10 with various corruptions (blur, swirl, noise), the Conformal C2ST achieved state-of-the-art power and remained robust when the underlying classifier was weak.
Real-World Scientific Inverse Problem: In a gravitational lensing benchmark, the Conformal C2ST detected biases at lower levels than competitors and remained valid under classifier degradation, whereas other methods (like DC) were invalid due to inflated Type-I error.

Significance and Claims

The paper claims that the Conformal C2ST establishes a practical, theoretically grounded diagnostic tool for model validation. Its primary significance lies in resolving the "weak classifier" ambiguity: a model passing a Conformal C2ST test is a more trustworthy indication of distributional equality than passing a standard C2ST, because the former's validity does not rely on the classifier being near-optimal.

The authors emphasize that a near-Bayes-optimal classifier is not strictly required for reliable model validation; a weak but informative ranking suffices. This makes the method particularly valuable for NPE and generative model evaluation where obtaining perfect classifiers is often infeasible. The work is modest in scope, focusing on the validation of the learned posterior $q$ against the true posterior $p$ (the $M$ -closed setting) rather than the broader $M$ -open problem of validating the simulator itself against real-world data. The authors also note limitations, such as the global nature of the test (it does not localize mismatches) and the computational cost of the Uniform variant when sampling from $p$ is expensive.