🤖 machine learning

Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks

This paper identifies and validates a specific "leak fingerprint" in near-OOD benchmarks caused by training data contamination, demonstrating that such leaks lead to misleadingly low unsupervised detection scores while remaining perfectly decodable by supervised models, and proposes a corrected protocol to ensure benchmark integrity.

Original authors: Vishnu Bindu Balachandran

Published 2026-07-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Vishnu Bindu Balachandran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to teach a robot how to spot a fake. You show the robot a thousand photos of real cats and tell it, "These are the real ones." Then, you show it a photo of a dog and ask, "Is this a cat?" If the robot says, "No, that's a dog," it's doing its job. This field of science is called Out-of-Distribution (OOD) detection. It's about teaching computers to recognize when something they see is totally different from what they learned in school.

But here is the tricky part: sometimes the "fake" thing isn't a dog; it's a very strange-looking cat, or a photo of a cat taken in a weird light. This is called a "near-OOD" problem. It's like asking the robot to spot a cat that looks suspiciously like a tiger. To test if the robot is smart, scientists create special tests where they hide one type of animal from the robot's training and see if it can spot it later. The whole system relies on a simple rule: the robot must never have seen the "test" animal before, or the test is broken. If the robot has already seen the test animal, it won't be able to tell if it's a "fake" or just a familiar friend.

Now, imagine you are auditing a robot that claims to be a master detective. You run a test, and the robot fails spectacularly. It doesn't just get a few questions wrong; it gets them backwards. It confidently tells you that the "fake" animal is actually a real one, and the real ones are fakes. In fact, it performs so badly that it's worse than if it had just guessed randomly. Most people would assume the robot is broken or the test was too hard. But in this paper, the author, Vishnu Bindu Balachandran, discovers that the robot isn't broken at all. The test itself was rigged.

The author found a "leak" in how the test was built. The scientists who designed the test accidentally included the "fake" animal in the robot's training school. So, when the robot saw the test animal, it thought, "Hey, I know this guy! He's one of my friends!" and gave it a passing grade. Because the robot was supposed to flag this animal as a stranger, it got a terrible score. The author calls this a "leak," like a secret message that got into the wrong room.

To prove this, the author did something clever. They took the exact same test, but this time, they made sure the robot had never seen that specific animal before. They wiped the robot's memory of that animal and re-ran the test. Suddenly, the robot's score jumped from a disastrous 0.326 (where 0.5 is a random guess) to a brilliant 0.911. The robot wasn't stupid; the test was cheating.

The paper then introduces a "fingerprint" to catch this kind of cheating in the future. It's a simple check: if a test is rigged, a smart computer can easily tell the difference between the "fake" and "real" groups if it looks at the labels (supervised), but a dumb computer (unsupervised) will get confused and think the "fake" group is actually the "real" one. If you see this specific pattern—where the smart computer says "Easy!" and the dumb computer says "Wait, what?"—you know there's a leak.

The author tested this fingerprint on dozens of other famous robot tests. They found that almost all of them were clean and fair. Only one specific test, involving a very tricky pair of datasets, triggered the alarm, and that was because it was genuinely hard, not because it was leaked. This means the standard way scientists build these tests is usually safe, but the specific document test the author started with had a hidden mistake.

Finally, the author re-ran the original robot test using the corrected, leak-free method. They found something surprising: even when the test was fair, the robot's special "perturbation" method (which tries to spot fakes by poking the robot and watching how it reacts) didn't actually work better than a simple, old-fashioned math trick. The robot could "read" the difference if it was allowed to cheat and look at the answers, but it couldn't "detect" the difference on its own. The paper concludes that the robot's fancy new method isn't a magic bullet, and that the real victory here is fixing the broken test and giving everyone a tool to make sure their tests aren't rigged in the future.

Technical Summary: "Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks"

Problem Statement
The paper addresses a critical flaw in the evaluation of Out-of-Distribution (OOD) detection, specifically within "near-OOD" settings where held-out classes are semantically similar to training classes. The authors identify a specific type of benchmark contamination where a class designated as "OOD" is inadvertently included in the model's training set and, consequently, inside the in-distribution (ID) fit set used to train the unsupervised detector.

This construction error leads to a catastrophic inversion of evaluation metrics. Because the "OOD" class is actually a trained class, its examples lie deep within the ID feature bulk. An unsupervised detector, which scores points based on distance from the ID bulk, correctly identifies these examples as in-distribution. Consequently, the detector is penalized for not flagging them as anomalies, resulting in AUROC scores well below the 0.5 chance level (e.g., 0.326). The authors argue that such "below-chance" performance is a strong signal of structural benchmark failure rather than a genuine method failure.

Methodology
The paper employs a multi-stage approach involving empirical auditing, diagnostic derivation, and controlled validation:

Empirical Discovery: The authors audited a perturbation-response OOD detector (PRS) on a standard document benchmark (Tobacco). They observed an AUROC of 0.326. Upon investigation, they found the "OOD" class (Memo) was a trained class whose training data remained in the one-class ID fit set.
Correction Protocol: They implemented a "leave-class-out" protocol, removing the designated OOD class from the training data and re-fitting the model. This correction raised the AUROC to 0.911, confirming the leak was the cause of the failure.
Leak Fingerprint Derivation: The authors derived a diagnostic test (the "leak fingerprint") based on the contradiction between supervised and unsupervised performance:
- Supervised Decodability: A trained class is perfectly linearly separable from the rest of the ID data. A supervised reader (OracleLDA) achieves near-perfect AUROC ( $\approx 1.0$ ).
- Unsupervised Collapse: Because the class is inside the ID fit set, unsupervised detectors score it as in-distribution, resulting in AUROC scores well below the clean baseline (operationally defined as $< 0.65$ , often below chance).
- The Fingerprint Rule: A split is flagged as contaminated if OracleLDA ≥ 0.95 AND best-unsupervised-AUROC < 0.65.
Controlled Validation: The fingerprint was tested on a battery of 52 settings across four fine-tuned backbones (ResNet-50 and ViT-B/16 on CIFAR-10 and CIFAR-100). This included 20 deliberately leaked settings (positives), 20 matched clean controls where the class was excluded from training (negatives), and 12 novel-source controls.
In-the-Wild Audit: The fingerprint was applied to 24 standard near/far OOD benchmark pairs (including OpenOOD standards) and 12 text/document settings to verify if standard cross-dataset construction introduces this specific leak.

Key Results

Leak Identification: In the initial document benchmark, the leak caused an AUROC of 0.326. After applying the leave-class-out correction, the score improved to 0.911.
Fingerprint Performance: On the controlled 52-setting battery, the fingerprint achieved:
- Sensitivity: 18/20 (0.90) on deliberately leaked settings.
- Specificity: 31/32 (0.97) on clean controls.
- Matched Controls: Perfect specificity (20/20) on matched controls where the only variable was the exclusion of the class from the fit set.
- The single false positive occurred on an intrinsically hard, clean near-OOD pair (CIFAR-100 $\to$ CIFAR-10), which the authors note is a known edge case where unsupervised detection naturally collapses without a leak.
In-the-Wild Findings: The fingerprint fired on exactly one of 24 standard benchmark pairs (the intrinsically hard CIFAR-100 $\to$ CIFAR-10 pair, correctly identified as a scope limit, not a leak). It fired on zero far-OOD pairs. This confirms that standard cross-dataset construction does not introduce this specific fit-set leak.
Re-evaluation of Perturbation Detectors: After correcting the benchmarks, the authors re-evaluated the perturbation-response detector (PRS). They found that while the OOD signal is decodable by a supervised reader (AUROC 0.87–1.00), it is not detectable by unsupervised methods. The PRS detector did not meaningfully improve over a plain Mahalanobis-on-embeddings baseline and was often outperformed by an off-the-shelf Isolation Forest.

Significance and Claims
The paper claims three primary contributions, modestly framing them as corrections and diagnostics rather than new detection methods:

Identification of a Benchmark Leak: The authors demonstrate that designating a semantically similar trained class as "OOD" without removing it from the training/fit set creates a "leak" that inverts evaluation scores. This is a construction discipline lapse, not an inherent limitation of OOD detection.
A Validated Diagnostic Tool: They provide a "leak fingerprint" that can audit existing benchmarks for this specific contamination without requiring backbone re-training. The tool effectively distinguishes between a genuine data leak and a genuinely hard, clean near-OOD problem.
A Corrected Protocol and Re-evaluation: By enforcing a strict "leave-class-out" protocol, the authors reveal that perturbation-based signals are decodable but not detectable by unsupervised means. This suggests that the perceived superiority of such methods in previous literature may have been obscured by the very leaks this paper identifies.

The paper explicitly states it does not propose a new OOD method. Instead, it offers a "corrected protocol" and a "validated detector for the contamination," emphasizing that the perturbation detector (PRS) is no better than a plain baseline once the benchmark is cleaned. The work also includes a retraction of an earlier headline correlation found to be an artifact of circular measurement, reinforcing the paper's focus on rigorous auditing over methodological novelty.

Technical Summary: "Decodable but Not Detectable: A Leakage Fingerprint for Near-OOD Benchmarks"

More like this