💻 computer science

Don't Collapse Your Features: Why CenterLoss Hurts OOD Detection and Multi-Scale Mahalanobis Wins

This paper introduces GOEN, a simple and efficient OOD detection pipeline that outperforms state-of-the-art baselines by leveraging multi-scale features and Mahalanobis distance while demonstrating that CenterLoss, despite improving classification accuracy, significantly degrades OOD detection by distorting the feature geometry required for reliable epistemic uncertainty estimation.

Original authors: Rahul D Ray

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rahul D Ray

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize ten specific animals: a cat, a dog, a horse, a bird, and so on. You show it thousands of pictures until it becomes an expert at sorting them. But what happens when you show the robot a picture of a toaster, a cloud, or a completely made-up creature?

This is the problem of Out-of-Distribution (OOD) detection. In the real world, we need our AI to say, "I don't know what this is," rather than confidently guessing it's a cat when it's actually a toaster. If the AI can't tell the difference, it might make dangerous mistakes.

This paper introduces a new method called GOEN (Geometry-Optimised Epistemic Network) to help AI recognize when it's seeing something new. The authors discovered a surprising secret: trying to make the AI's internal "knowledge map" too perfect actually makes it worse at spotting the unknown.

Here is the breakdown of their findings using simple analogies:

1. The Trap of "Perfect" Clumps (The CenterLoss Problem)

Usually, when training an AI, researchers try to make the features of each animal group (cats, dogs, etc.) clump together tightly. They use a tool called CenterLoss to squeeze these groups into neat, compact balls. The logic is: "If all the cats are in one tight ball and all the dogs are in another, the AI will be very accurate."

The Surprise: The authors found that while this makes the AI better at identifying cats and dogs, it makes it worse at spotting toasters.

The Analogy: Imagine a library where you force every book about "History" into a tiny, single shelf, and every book about "Science" into another tiny shelf. Now, if someone brings in a book about "History of Science," it doesn't fit neatly into either shelf. Because the shelves are so squeezed together, the book ends up right in the middle, looking like it belongs to both. The AI gets confused and thinks, "Oh, this must be a History book," even though it's something new.
The Result: By stopping the "squeezing" (removing CenterLoss), the groups of knowledge spread out a bit more. This creates more space between the groups, making it much easier for the AI to see when something falls into the empty space (the "unknown" zone).

2. Looking at the Big Picture and the Details (Multi-Scale Features)

To spot the unknown, the AI needs to look at things in two ways at once:

The Texture (Layer 2): Looking at the brushstrokes, colors, and grain. This helps it notice if a picture looks like a street sign (SVHN) instead of a cartoon animal.
The Meaning (Layer 4): Looking at the overall shape and concept. This helps it realize a "fox" is not a "dog."

The Analogy: Imagine trying to identify a stranger. If you only look at their face (meaning), you might miss that they are wearing a costume. If you only look at their shoes (texture), you might miss who they are. GOEN looks at both simultaneously. The paper shows that using only the "face" or only the "shoes" makes the AI less effective at spotting imposters.

3. The "Real World" Test (Calibration)

After the AI learns the animals, the researchers give it a special "final exam" to learn how to say "I don't know."

They show it Gaussian Noise (static on a TV screen). This is easy to spot as "weird."
Crucially, they also show it real, tricky examples (like house numbers from a different dataset). These look somewhat like the animals but are actually different.
The Analogy: If you only teach a security guard to spot a rock (easy), they might miss a person wearing a rock costume (hard). By training the guard on the "rock costume" (the hard examples), they learn to be suspicious of things that look familiar but feel wrong.

4. The Winning Formula

The authors combined these ideas into a pipeline:

Don't squeeze the groups too tight. Let the "cat" and "dog" groups have some breathing room.
Look at details and big pictures together.
Train the "I don't know" button using real, tricky examples, not just random noise.

The Results

When they tested this new method (GOEN) against other famous AI safety methods:

GOEN correctly identified unknown inputs 94.8% of the time.
The next best method (Deep Ensembles) only got 88.3%.
Other methods like KNN and ODIN scored even lower.

The Big Takeaway

The paper's main message is a warning to AI researchers: Just because an AI is great at sorting known things doesn't mean it's good at knowing its limits.

In fact, trying to make the AI's internal knowledge too perfect (too compact) can blind it to the unknown. By letting the knowledge groups breathe and training the AI with real-world tricky examples, we can build systems that are safer and more honest about what they don't know.

The entire system is also very fast, training in under 20 minutes on a single computer, making it a practical tool for real-world safety.

Technical Summary: Don't Collapse Your Features: Why CenterLoss Hurts OOD Detection and Multi-Scale Mahalanobis Wins

Problem Statement
The safe deployment of machine learning systems in open-world environments requires robust Out-of-Distribution (OOD) detection. Current methods often rely on feature representations optimized solely for classification accuracy, neglecting the distinct requirements of epistemic uncertainty. A prevailing assumption in the field is that improving classification geometry (e.g., making feature clusters more compact) automatically leads to better uncertainty estimation. This paper challenges that assumption, investigating why popular regularizers like CenterLoss, which enforce feature compactness, may actually degrade OOD detection performance.

Methodology: The GOEN Pipeline
The authors introduce the Geometry-Optimised Epistemic Network (GOEN), a three-stage pipeline designed to optimize feature geometry specifically for OOD detection rather than just classification. The pipeline consists of:

Multi-Scale Feature Extraction: A ResNet-18 backbone is trained using standard cross-entropy loss. Crucially, the model extracts features from two intermediate layers: Layer 2 (128-dimensional, capturing low-level texture and domain shifts) and Layer 4 (512-dimensional, capturing high-level semantic content). These are concatenated to form a 640-dimensional vector.
L2 Normalization and Mahalanobis Fitting: The extracted features are L2-normalized to lie on a unit sphere. This step mitigates "feature collapse," where a single principal component dominates variance, ensuring a well-conditioned covariance matrix. Class-conditional means ( $\mu_c$ ) and a tied covariance matrix ( $\Sigma$ ) are fitted on the training data. The OOD score is derived from the Mahalanobis distance of a test point to the nearest class centroid.
Calibration Head: A lightweight Multi-Layer Perceptron (MLP) is trained to map three uncertainty cues to a calibrated probability:
- Log-Mahalanobis distance.
- Maximum cosine similarity to class means.
- Predictive entropy from the classification head.
- Training Data: The head is trained on a mixture of In-Distribution (ID) data and "hard" OOD examples (a subset of SVHN) alongside synthetic Gaussian noise.

Key Findings and Contributions
The paper's primary contribution is a counter-intuitive finding revealed through systematic ablation: CenterLoss significantly harms OOD detection.

The CenterLoss Paradox: While CenterLoss improves classification accuracy by forcing features into tighter clusters, it reduces the average OOD AUROC from 0.9483 (without CenterLoss) to 0.9366 (with CenterLoss). The authors hypothesize that over-compacting feature clusters compresses the inter-class margins, making it difficult to distinguish OOD points that lie between clusters. Furthermore, tight clusters can lead to ill-conditioned covariance matrices, degrading the effectiveness of the Mahalanobis distance.
Multi-Scale Necessity: Ablation studies show that using only semantic features (Layer 4) drops performance significantly (Avg AUROC 0.9270 vs. 0.9483). Combining texture (Layer 2) and semantic features is essential for detecting both domain shifts (e.g., SVHN) and semantic shifts (e.g., CIFAR-100).
Real Hard OOD Calibration: Training the calibration head with real domain-shift examples (SVHN) rather than just synthetic noise is critical. Removing SVHN from the calibration mix reduced performance on the SVHN test set by nearly 2.5% AUROC, demonstrating that synthetic noise alone cannot teach the model to recognize complex domain shifts.

Experimental Results
Evaluated on CIFAR-10 (ID) against SVHN, filtered CIFAR-100, and synthetic Gaussian noise (OOD), the best GOEN variant (GOEN-NoCenterLoss) achieved:

Average OOD AUROC: 0.9483, surpassing all baselines.
Comparison: It outperformed Deep Ensembles (0.8827), KNN (0.8967), ODIN (0.8870), and standard Mahalanobis (0.6790).
Efficiency: The entire pipeline trains in under 20 minutes on a single GPU.
Stability: Seeding experiments confirmed the method's stability, with low variance in ID accuracy (0.9344 ± 0.0013) and OOD AUROC across five random seeds.

Significance and Claims
The paper argues that classification geometry and epistemic geometry are not aligned. Optimizing for classification accuracy (via compactness regularizers) can actively harm a model's ability to recognize when it is encountering novel data.

The authors position GOEN as a practical blueprint for building AI systems that "reliably recognise their own limitations." By decoupling the objectives of classification and epistemic uncertainty, and by leveraging multi-scale features with L2-normalized Mahalanobis distance, GOEN provides a simple, efficient, and state-of-the-art solution for OOD detection without the computational overhead of deep ensembles or complex variational inference. The work serves as a cautionary tale against the automatic assumption that techniques improving classification will inherently improve uncertainty estimation.