🤖 machine learning

Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

This paper demonstrates that within-dataset class-split evaluation for anomaly detection can become ill-posed and produce unstable or inverted scores when held-out anomaly classes overlap with normal data in representation space, proposing a training-free diagnostic called "neighborhood class leakage" to predict such failures across various datasets and feature spaces.

Original authors: Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, Olivier Salvado

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alejandro Ascarate, Leo Lebrat, Rodrigo Santa Cruz, Clinton Fookes, Olivier Salvado

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: "Testing the Test"

Imagine you are a teacher trying to test how well a student can spot a "fake" apple. You have a basket of 10 types of real apples (Red Delicious, Granny Smith, Gala, etc.).

The Standard Test (Class-Split):
To make the test, you decide that 9 types of apples are "Normal" and the 10th type is the "Anomaly" (the fake one). You train the student only on the 9 normal types. Then, you show them a mix of the 9 normal types and the 10th type, asking them to point out the fakes.

The paper argues that this test is broken in many cases. Sometimes, the "fake" apple (the 10th type) looks so much like the real ones that the student gets confused. Worse, depending on which apple you pick as the "fake," the student might start pointing at the real apples as the fakes, or they might just guess randomly.

The Core Problem: The "Overlapping Crowd"

The authors discovered that in many image datasets (like CIFAR-10 or Imagenette), the "anomaly" class isn't actually far away from the "normal" classes in the computer's mind.

The Metaphor: Imagine a crowded party.
- Normal Group: A mix of people wearing red, blue, and green shirts.
- Anomaly Group: People wearing yellow shirts.
- The Problem: In this specific party, the people in yellow shirts are standing right in the middle of the people in blue and green shirts. They are mixed in so well that you can't tell them apart just by looking at who is standing next to whom.

When the computer tries to find the "odd one out," it gets confused.

The Collapse: The computer's score for "how weird this is" drops to chance level (50/50 guessing).
The Inversion: Sometimes, the computer gets it backwards. It thinks the "weird" yellow shirts are actually the "normal" ones, and the "normal" blue shirts are the weird ones.

The "Direction" Confusion

The most dangerous part of this problem is Direction Instability.

Scenario A: If you pick "Yellow" as the anomaly, the computer learns: "High score = Weird."
Scenario B: If you pick "Purple" as the anomaly, the computer learns: "Low score = Weird."

If you don't know which class is the anomaly ahead of time (which is the case in real life), the computer has no consistent rule. It's like a compass that points North when you are in New York, but points South when you are in London. You can't trust it to guide you anywhere.

The New Tool: "Neighborhood Leakage"

The authors invented a simple, free way to check if a test is broken before you even run the anomaly detector. They call it Neighborhood Leakage.

The Analogy: Imagine you are looking at a person in a crowd. You look at their 10 closest neighbors.
- Low Leakage: All 10 neighbors are wearing the same color shirt as the person. The groups are clean and separated. The test is likely valid.
- High Leakage: The person is wearing a red shirt, but 8 of their 10 closest neighbors are wearing blue, green, or yellow. The groups are mixed up.

The paper shows that if you have High Leakage, your test results will be unstable, inverted, or random. It's a "red flag" that says, "Stop! The geometry of this data is too messy for this specific test to work."

What They Found

They tested this on three famous image datasets:

Fashion-MNIST (Simple): The clothes are distinct. Leakage is low. The test works okay.
CIFAR-10 & Imagenette (Complex): These have cars, animals, and birds. The "anomaly" classes often overlap heavily with the "normal" classes.
- Result: High leakage.
- Consequence: The tests showed massive instability. Sometimes the "anomaly" was ranked as the most normal thing in the room.

The Takeaway

The paper concludes that we shouldn't blindly trust "Class-Split" tests (where you hide one category to act as the anomaly) as proof that an AI is good at finding anomalies.

Old View: "If the AI gets a high score, it's smart at finding anomalies."
New View: "If the AI gets a high score, it might just be good at exploiting a specific quirk of that one test. We need to check the 'Leakage' first. If the groups are mixed up, the test is a 'stress test' of the data's shape, not a proof of the AI's skill."

In short: Before you trust a test that says "This AI can find the odd one out," check if the "odd one out" is actually hiding in plain sight among the others. If it is, the test results are meaningless.

Technical Summary: Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

1. Problem Statement

Within-dataset class-split evaluation is a standard protocol in anomaly detection (AD), where specific semantic classes are designated as "normal" and others as "anomalous" to simulate out-of-distribution (OOD) detection. While attractive for avoiding external data sources, this paper argues that the protocol can become ill-posed in a precise operational sense when the chosen "anomaly" class overlaps significantly with the "normal" mixture in the representation space.

The core failure mode identified is class-dependent score-direction instability. When the held-out anomaly class lies within or near the high-density regions of the normal mixture, anomaly scores may collapse toward chance (AUROC $\approx$ 0.5) or, more critically, invert (AUROC < 0.5). In such regimes, the direction of "anomalousness" (whether a high or low score indicates an anomaly) depends on the specific, unknown anomaly class. Consequently, benchmark conclusions become fragile, potentially rewarding methods that exploit dataset-specific quirks rather than demonstrating robust, unconditional OOD detection capabilities.

2. Methodology

The authors propose a framework to diagnose and quantify this instability without requiring additional training.

Diagnostic: Neighborhood Class Leakage
The paper introduces a simple, training-free metric called neighborhood class leakage ( $L_k$ ) to quantify the overlap between class-conditional manifolds in a fixed representation space $r(x)$ .

Definition: For a sample $x_i$ , the leakage $\ell_k(i)$ is the fraction of its $k$ -nearest neighbors (in representation space) that belong to a different semantic class.
Aggregation: The dataset-level index $L_k$ is the average leakage across all samples.
Rationale: High leakage indicates that local geometry does not align with semantic class structure. If an anomaly class occupies regions densely populated by normal samples, standard AD scores (based on distance-to-normality or local density) will fail to separate them consistently.

Instability Metrics
To characterize the benchmark's reliability, the authors define several statistics based on the "K-1 vs. 1" protocol (iterating through each class as the held-out anomaly):

AUROC Dispersion: Variance of AUROC scores across different held-out classes.
Near-Random Rate ( $\rho_{rnd}$ ): The frequency of AUROC scores falling near 0.5.
Inversion Rate ( $\rho_{inv}$ ): The frequency of AUROC scores falling below 0.5.
Direction Instability Rate ( $\rho_{dir}$ ): A measure of how often the preferred score direction (positive or negative deviation from chance) flips depending on the anomaly class.

Experimental Setup
The study evaluates three datasets of increasing complexity:

Fashion-MNIST: Used as a negative control (low overlap).
CIFAR-10: Intermediate complexity.
Imagenette: High complexity.

These are tested in two representation spaces:

Pixel Space: Vectorized raw inputs.
VAE Latent Space: Encoded features from a Variational Autoencoder trained only on the normal pool.

Three standard unconditional AD scorers are applied: k-Nearest Neighbors (kNN), Isolation Forest, and Local Outlier Factor (LOF).

3. Key Results

The empirical results demonstrate a strong correlation between high neighborhood leakage and benchmark instability:

Fashion-MNIST: Exhibits low leakage ( $L_k \approx 0.24$ ) and stable behavior, with low inversion and direction-instability rates.
CIFAR-10 and Imagenette: Exhibit high leakage ( $L_k > 0.76$ ) in both pixel and latent spaces. These datasets show substantial inversion rates (up to 50%) and direction-instability rates (up to 63%).
Consistency: The instability persists across different scoring families (kNN, Isolation Forest, LOF) and representation spaces, suggesting the failure is inherent to the geometric overlap of the data rather than a specific detector's flaw.
Per-Class Variance: In CIFAR-10 pixel space, specific anomaly classes yield AUROC scores significantly below chance, while others yield scores well above chance, despite using the same scoring convention.

4. Contributions

The paper makes three primary contributions:

Formalization of a Failure Mode: It identifies and formalizes class-dependent score-direction instability as a practical failure mode of within-dataset class-split evaluation, driven by normal-anomaly overlap.
Training-Free Diagnostic: It proposes the neighborhood leakage index ( $L_k$ ) as a simple, pre-evaluation check to predict when a class-split benchmark is likely to yield unreliable or inverted rankings.
Empirical Validation: It validates the phenomenon across a controlled matrix of datasets, representations, and scorers, demonstrating that high leakage predicts instability regardless of the specific AD method used.

5. Significance and Claims

The paper argues that class-split benchmarks should not be treated as unconditional evidence of anomaly detection ability. Instead, they should be viewed as geometry-dependent stress tests.

Rejection of "Re-inversion": The authors refute the notion that a "better" AD method can simply "re-invert" rankings to fix low AUROC. If the direction of anomalousness depends on the unknown anomaly type, a method that achieves high AUROC on a specific split may be exploiting idiosyncratic correlations between class labels and geometry rather than learning a class-agnostic notion of atypicality.
Interpretation of Benchmarks: The leakage index serves as a precondition check. Low leakage supports the interpretation that AUROC measures separation from the normal mixture. High leakage, coupled with high inversion or direction-instability rates, indicates the benchmark is probing overlapping geometry rather than a consistent semantic anomaly.
Recommendation: The authors suggest that future class-split benchmarks should report leakage and direction-instability diagnostics before their results are used to claim unconditional-OOD detection capabilities.