📊 statistics

Reliable AUC Evaluation for Positive-Unlabeled Classifiers: Calibrated Confidence Intervals under an Unknown Class Prior

This paper proposes a method to derive calibrated, two-sided confidence intervals for the true Area Under the Curve (AUC) in Positive-Unlabeled learning by exactly recovering the target AUC from observable metrics and propagating the uncertainty of the estimated positive fraction, thereby addressing the bias and lack of reliability in current performance evaluations.

Original authors: Vincent Looten

Published 2026-09-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vincent Looten

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of machine learning, computers are often taught to recognize patterns by showing them examples of what they are looking for and examples of what they are not. Imagine a doctor trying to teach an algorithm to spot a specific type of tumor. The computer needs to see clear pictures of tumors and clear pictures of healthy tissue to learn the difference. But in many real-world situations, getting those clear examples of "healthy" is difficult. Often, researchers only have a list of confirmed positive cases and a large, messy pile of unlabeled data that contains a mix of both the positive cases and the negative ones, with no way to tell them apart. This is known as positive-unlabeled learning. The goal is to build a system that can still rank the good cases higher than the bad ones, even when the bad cases are hidden inside the crowd. The standard way to measure how well a system does this is by calculating a score that represents its ability to distinguish between the two groups. However, when the negative group is hidden and mixed in, the standard score becomes misleading. It tells you how well the system ranks against the messy pile, not against the true negatives, and it usually presents this number as a single, exact point without any indication of how much that number might be wrong due to chance.

A researcher named Vincent Looten has tackled this problem by developing a new way to measure performance that accounts for the hidden mix and provides a reliable range of confidence. The core of the work is a mathematical correction that peels away the contamination of the unlabeled data to reveal the true performance against the negative cases. The researcher found that you cannot simply look at the messy pile and guess the answer; you must first estimate how much of that pile is actually the positive case you are looking for. Once you have that estimate, you can use a specific formula to adjust the performance score. But the researcher went further, realizing that simply adjusting the number is not enough. Because the estimate of the mix is itself uncertain, that uncertainty must be carried through the calculation. The study shows that if you ignore this uncertainty, your final score will be wrong. By carefully tracking how the uncertainty in the mix estimate affects the final score, the researcher derived a method to produce a calibrated interval. This interval acts like a safety net, telling the user the range within which the true performance almost certainly lies, rather than just giving a single, potentially misleading number.

The study reveals that this method works beautifully when the positive and negative cases are distinct enough to be separated. In these clear situations, the new method produces a two-sided range that captures the true performance almost exactly as often as a standard statistical rule would expect. However, the researcher also discovered a hard limit to this approach. When the positive and negative cases are so similar that they blur together, the mix becomes impossible to pin down with certainty. In this specific scenario, the two-sided range breaks down because the math simply cannot support it. Instead of forcing a false range, the method switches to a one-sided bound. This bound provides a guaranteed minimum floor for performance, admitting that while the exact score is unknown, the system is certainly at least this good. This switch is not a failure of the method but a feature of it, ensuring that the report remains honest even when the data is too ambiguous to support a precise estimate.

To test these ideas, the researcher applied the method to real-world data, specifically using medical records for breast cancer where the true labels were known but treated as hidden to simulate the problem. The results were stark. A traditional approach that ignores the hidden mix and reports a single number failed completely, never capturing the true performance in the tests. Another approach that tried to correct the number but used a simple estimate for the mix also struggled, especially when the data did not follow a perfect bell-curve shape. Only the new method, which combined the correction with a robust way of estimating the mix and a careful calculation of the uncertainty, succeeded. It consistently produced intervals that contained the true performance, provided the positive and negative cases were not too similar. The study confirms that the key to getting a reliable answer is not just the algorithm doing the ranking, but the quality of the estimate used to describe the hidden mix. If that estimate is biased or inaccurate, the final performance score will be wrong, no matter how sophisticated the ranking system is.

The work also clarifies when a researcher should trust a two-sided range and when they should settle for a one-sided floor. The transition point depends on how distinct the two groups are relative to the amount of data available. If the groups are well-separated, the full range is valid. If they are very close, the method correctly identifies that a precise two-sided range is impossible and offers the safer, one-sided guarantee instead. This distinction is crucial for practitioners who need to know not just how well a system works, but how sure they can be about that number. The researcher packaged this entire process into a tool that can wrap around any existing scoring system, allowing users to input their data and receive a decontaminated score along with a confidence interval or a safety floor. This tool does not require the data to follow a specific mathematical shape, making it useful for the messy, real-world datasets that often defy simple models.

Ultimately, this research shifts the focus from simply building better classifiers to understanding how to measure them correctly when the data is incomplete. It demonstrates that in the absence of clear negative examples, the most critical piece of information is the estimate of how many positive cases are hiding in the unlabeled pile. The study proves that with the right mathematical adjustments, it is possible to recover a true measure of performance and attach a reliable margin of error to it. However, it also sets a clear boundary: when the signal is too weak to separate the groups, the method refuses to guess, offering instead a conservative lower limit. This honesty about the limits of what can be known is perhaps the most valuable finding of all, ensuring that decisions based on these scores are made with a clear understanding of the underlying uncertainty.

Technical Summary: Reliable AUC Evaluation for Positive-Unlabeled Classifiers

Problem Statement
In Positive-Unlabeled (PU) learning, classifiers are trained on a labeled positive set and an unlabeled set containing a mixture of positives and negatives in an unknown proportion ( $\pi$ ). A critical gap exists in evaluating these classifiers:

Bias: The Area Under the ROC Curve (AUC) measured against the unlabeled sample ( $AUC(P, U)$ ) is a biased estimator of the true target AUC against genuine negatives ( $\theta = AUC(P, N)$ ). The gap depends on the unknown positive fraction $\pi$ .
Lack of Uncertainty Quantification: Existing PU methods report performance as point estimates conditional on a supplied $\pi$ , without calibrated confidence intervals. Consequently, it is impossible to distinguish real performance differences between methods from sampling noise.
Identifiability Issues: When positive and negative score distributions are close, $\pi$ and the negative distribution are not separately identifiable, raising the question of whether a two-sided confidence interval for $\theta$ can ever exist.

Methodology
The paper proposes a framework to recover the target AUC ( $\theta$ ) and construct calibrated confidence intervals by propagating the uncertainty of an estimated proportion through an exact decontamination identity.

Decontamination Identity: The author derives an exact identity relating the observable AUC ( $\mu_A$ ) to the target AUC ( $\theta$ ) and the positive fraction ( $\pi$ ):
$\theta = g(\mu_A, \pi) = \frac{\mu_A - \pi/2}{1 - \pi}$
This identity reveals that the transformation is affine in the observable AUC but non-linear (curved) in the proportion $\pi$ .
Second-Order Propagation: Unlike standard first-order (delta method) analyses, the author performs a second-order propagation of uncertainty. They demonstrate that the curvature of the function $g$ with respect to $\pi$ introduces a bias term of order $Var(\delta_\pi)$ and a specific variance structure. This curvature term is essential for achieving nominal coverage, particularly in the "proportional regime" where dimension $p$ and sample size $n$ grow together ( $p/n \to \gamma$ ).
Closed-Form Variance and Inputs: Under a bi-Gaussian isotropic model, the author derives closed-form expressions for:
- The achieved separation ( $d^*$ ) using resolvent laws from random matrix theory (Marchenko-Pastur).
- The variances of the Mann-Whitney statistic against contaminated samples.
- The variance of the proportion estimator, which includes a "coupling" term because both the AUC and proportion estimates rely on the same unlabeled sample.
- The final calibrated variance for $\hat{\theta}$ , which includes terms for the AUC channel, the proportion channel, and the coupling channel.
Boundary of Validity (Singular Learning Theory): The paper characterizes the domain where a two-sided interval is valid using the Real Log Canonical Threshold (RLCT) of the underlying mixture model.
- Regular Regime: When the separation $d^*$ is large relative to $n_U^{-1/2}$ , the proportion is estimable at the standard $\sqrt{n}$ rate, and a two-sided interval exists.
- Singular Regime: As the distributions merge ( $d^* \lesssim n_U^{-1/2}$ ), the proportion becomes non-identifiable at the standard rate. Here, the two-sided interval loses calibration. The honest report in this regime is a one-sided bound: a lower bound derived from the contaminated AUC (which is always $\le \theta$ ) and an upper bound derived from distribution-free proportion bounds (Patra and Sen).
Implementation: A Python routine (pu_auc_inference) is provided. It offers two paths:
- Closed-form: Fast, assuming the bi-Gaussian model holds and standardization is known or supplied.
- Bootstrap: Distribution-free, resampling the entire pipeline to capture uncertainty in standardization and proportion estimation, recommended for real-world data where the bi-Gaussian assumption may fail.

Key Results

Necessity of Curvature: Experiments show that omitting the second-order curvature term leads to severe undercoverage (e.g., 0.83–0.90 instead of 0.95).
Decontamination is Mandatory: Intervals centered on the contaminated AUC ( $\mu_A$ ) never cover the true $\theta$ , regardless of the contamination level.
Role of the Proportion Estimator: The validity of the interval is "binding" on the quality of the proportion estimator.
- On real, non-Gaussian data, a moment-based estimator (designed for bi-Gaussian data) introduces significant bias, causing the interval to fail.
- A distribution-free proportion estimator restores coverage to near-nominal levels at low-to-moderate contamination.
Threshold Confirmation: Numerical experiments confirm the theoretical crossover point ( $d^* \sim n_U^{-1/2}$ ) where the interval transitions from two-sided to one-sided, consistent with the RLCT of $1/2$ .

Significance and Claims
The paper claims to fill a specific gap in PU literature: while extensive work exists on training classifiers and estimating proportions, no prior work has provided a calibrated two-sided confidence interval for the AUC value in PU settings.

Feasibility: The author demonstrates that reporting $\theta$ with a calibrated interval is both needed and feasible, provided the proportion estimator is robust.
Limitations: The method does not produce a new classifier; it evaluates existing scores. The interval's validity is strictly conditional on the proportion estimator's performance. If the proportion is weakly identified (near the singularity), the method correctly degrades to a one-sided bound rather than providing a false sense of precision.
Practical Utility: The provided routine allows practitioners to wrap any scoring model (e.g., uPU, nnPU, tree-based methods) to obtain a decontaminated AUC estimate with a stated uncertainty range and a validity flag indicating whether the regular or singular regime applies.

The work bridges the gap between mixture estimation theory (singular learning) and AUC inference, establishing that the "honest" report in PU learning must account for the uncertainty of the class prior and the geometric singularity of the mixture model.

Technical Summary: Reliable AUC Evaluation for Positive-Unlabeled Classifiers

More like this