🤖 machine learning

MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse

This paper introduces MahaVar, a post-hoc out-of-distribution detection method that leverages the theoretically grounded observation that in-distribution samples exhibit high class-wise Mahalanobis distance variance due to Neural Collapse geometry, achieving state-of-the-art performance on standard benchmarks.

Original authors: Donghwan Kim, Hyunsoo Yoon

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Donghwan Kim, Hyunsoo Yoon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart security guard (a neural network) who has spent years learning to recognize specific types of people: "Doctors," "Firefighters," and "Chefs." This guard is great at his job when he sees someone wearing a white coat, a helmet, or a toque. But what happens when a random tourist walks in wearing a bright pink clown suit?

In the world of AI, this tourist is called an Out-of-Distribution (OOD) sample. The problem is that the guard might get confused and confidently say, "That's definitely a Chef!" because the clown suit has some colors the guard has seen before. This is dangerous in real life (like a self-driving car mistaking a plastic bag for a rock).

This paper, MahaVar, introduces a new way for the security guard to spot these "clowns" before they cause trouble. Here is how it works, broken down simply:

The Old Way: "How Close Are You?"

Previously, the guard used a method called Mahalanobis Distance. Imagine the guard has a map with specific "zones" for Doctors, Firefighters, and Chefs.

When a new person arrives, the guard measures how far they are from the center of the "Doctor" zone, the "Firefighter" zone, etc.
If the person is very close to one of these zones (say, the Doctor zone), the guard assumes they are a Doctor.
If they are far from all zones, the guard assumes they are an outsider.

The Flaw: Sometimes, a confused tourist might be equally far from all zones, or just slightly closer to one by accident. The old method only looked at the single closest zone and ignored the rest.

The New Discovery: The "Sharp Mountain" vs. The "Flat Hill"

The authors of this paper noticed something interesting about how the distances look when you line them up from closest to farthest.

For a real Doctor (In-Distribution): The distances look like a sharp mountain peak. The distance to the "Doctor" zone is tiny (the peak), but the distance to the "Firefighter" and "Chef" zones is huge. There is a massive gap between the closest one and the rest.
- Analogy: Imagine a mountain climber standing right at the summit. They are very close to the top, but very far from the base camp and every other peak. The difference in height is huge.
For a confused tourist (Out-of-Distribution): The distances look like a flat hill. The tourist isn't close to the "Doctor" zone, nor the "Firefighter" zone. They are just sort of "in the middle" of nowhere. The distances to all the zones are roughly the same and moderately large.
- Analogy: Imagine someone standing on a flat plateau. They are the same distance from the Doctor's peak, the Firefighter's peak, and the Chef's peak. There is no sharp peak; it's just a flat, boring landscape.

The "Variance" Secret

The paper calls this difference Variance.

High Variance (The Mountain): One distance is tiny, the rest are huge. The numbers jump around a lot. This means "I am definitely a Doctor."
Low Variance (The Flat Hill): All distances are similar. The numbers are boring and flat. This means "I don't belong to any specific group."

The authors realized that by measuring how much these distances "jump around" (the variance), they could tell the difference between a real ID sample and a fake OOD sample much better than just looking at the closest one.

The Solution: MahaVar

They built a new tool called MahaVar.

It still checks how close the person is to their nearest group (like the old method).
But, it also adds a "Variance Score." It asks: "Is the distance to the nearest group sharply different from the others?"
If the answer is "Yes" (High Variance), it's likely a real ID sample.
If the answer is "No" (Low Variance), it's likely an OOD sample.

Why It Works (The "Neural Collapse" Theory)

The paper explains why this happens using a concept called Neural Collapse.
Imagine the AI has been training for so long that the "Doctor" zone has shrunk into a tiny, tight dot, and the "Firefighter" and "Chef" zones have moved far away to form a perfect geometric shape (like a star).

Real Doctors get pulled right into that tiny "Doctor" dot.
Tourists, who don't belong, get stuck floating in the empty space between the dots.
Because of this geometry, real samples naturally create that "sharp mountain" pattern, while fake samples create the "flat hill" pattern.

The Results

The authors tested this on famous image datasets (like CIFAR and ImageNet).

The Result: MahaVar consistently caught more "clowns" (OOD samples) and made fewer mistakes than previous methods.
The Catch: It works best when the AI has been trained well and has enough "room" in its brain (dimensions) to form those perfect geometric shapes. On some very complex models, it still works great, but the "sharp mountain" isn't quite as sharp.

Summary

Think of MahaVar as a security guard who doesn't just ask, "Are you close to a known group?" but also asks, "Is your relationship to that group unique compared to everyone else?" If you are uniquely close to one group and far from all others, you're probably a real member. If you are sort-of-close to everyone, you're probably an imposter. This simple addition of "checking the spread" makes the system much more reliable.

Technical Summary: MahaVar: OOD Detection via Class-wise Mahalanobis Distance Variance under Neural Collapse

Problem Statement
Deploying deep neural networks in real-world environments presents a critical challenge: models trained under a closed-world assumption often encounter test samples that fall outside their training distribution (Out-of-Distribution or OOD). These samples can lead to overconfident and potentially catastrophic predictions. While Out-of-Distribution (OOD) detection is essential for safety-critical domains like autonomous driving and medical imaging, existing post-hoc methods (which require no retraining or access to OOD data) often rely on limited signals. Specifically, Mahalanobis distance-based detectors, a strong baseline, typically focus solely on the minimum distance to the nearest class-conditional Gaussian distribution, potentially ignoring the structural geometry of distances across all classes.

Methodology: MahaVar
The authors propose MahaVar, a simple yet effective post-hoc OOD detector that augments the standard Mahalanobis distance with a class-wise distance variance term. The method is grounded in the following logic:

Empirical Observation: The authors observe that for In-Distribution (ID) samples, the sorted class-wise Mahalanobis distances exhibit a "pronounced sharp minimum structure." The distance to the nearest class is significantly smaller than distances to all other classes, which remain large and relatively uniform. In contrast, OOD samples exhibit a less pronounced minimum, with distances to all classes being comparatively large and less variable. This structural asymmetry implies that ID samples have a higher variance in class-wise distances compared to OOD samples.
Theoretical Grounding (Neural Collapse): The paper provides a theoretical analysis based on Neural Collapse, a geometric phenomenon where penultimate features of each class collapse toward their class mean, and class means arrange themselves into a Simplex Equiangular Tight Frame (ETF). Under relaxed Neural Collapse assumptions (specifically within-class compactness and inter-class separation), the authors prove that ID samples structurally exhibit strictly higher class-wise distance variance than OOD samples. Theorem 4.3 and Corollary 4.4 establish sufficient conditions under which this variance separation is guaranteed, particularly when OOD samples maintain sufficient angular separation from class means or do not lie within the subspace spanned by the class means.
Scoring Function: The proposed OOD score $S(x)$ combines the negative minimum Mahalanobis distance with the class-wise variance of these distances:
$S(x) = -\min_{c} d_{Maha,c}(\hat{\phi}(x)) + \alpha \cdot \text{Var}_c[d_{Maha,c}(\hat{\phi}(x))]$
Where $\hat{\phi}(x)$ represents L2-normalized penultimate features, and $\alpha$ is a hyperparameter controlling the weight of the variance term.

Key Contributions
The paper makes three primary contributions:

Empirical Discovery: Identification of a previously unexploited structural asymmetry: ID samples exhibit a sharp minimum in class-wise Mahalanobis distances (high variance), while OOD samples exhibit a flatter distribution (low variance).
Theoretical Justification: A principled analysis linking this observation to Neural Collapse geometry, proving that under specific geometric conditions, ID samples strictly dominate OOD samples in terms of class-wise distance variance.
Algorithmic Proposal: The introduction of MahaVar, a post-hoc detector that integrates this variance term into the Mahalanobis++ framework.

Experimental Results
The authors evaluated MahaVar following the OpenOOD v1.5 benchmark protocol across three In-Distribution (ID) settings: CIFAR-10, CIFAR-100, and ImageNet, using both Near-OOD and Far-OOD evaluation settings.

Performance: MahaVar achieves State-of-the-Art (SOTA) performance on CIFAR-100 and ImageNet among all evaluated methods.
Comparison to Baselines: It consistently outperforms all Mahalanobis-based baselines (including Mahalanobis and Mahalanobis++) across all three benchmarks in terms of AUROC and FPR@95.
Robustness: On the large-scale ImageNet benchmark, averaged over three backbone architectures (ResNet-50, Swin-B, and ViT-B), MahaVar achieves the best average AUROC and FPR@95. Notably, while other activation-shaping methods (like ASH-S and SCALE) degrade significantly on Transformer-based backbones (Swin-B, ViT-B), MahaVar maintains stable and superior performance.
Ablation Studies: Experiments confirm that the variance term provides meaningful complementary information even without L2 normalization, though the combination of L2 normalization and variance yields the best results. The method is also robust to the choice of distance metric (L1, L2, Mahalanobis), with Mahalanobis performing best due to its handling of covariance structure.

Significance and Claims
The paper claims that the full class-wise distance structure contains richer discriminative information than the nearest-class distance alone. By leveraging the geometric properties of Neural Collapse, MahaVar offers a theoretically grounded and empirically robust method for OOD detection. The authors emphasize that this approach is particularly valuable for real-world deployment where retraining models is impractical.

However, the authors maintain a modest tone regarding limitations:

On CIFAR-10, KNN-based methods still achieve a slightly higher AUROC, suggesting non-parametric approaches may be superior in low-class-count settings.
The theoretical framework relies on Neural Collapse assumptions, which assume well-trained networks near the terminal phase of training. The authors note that relaxing these assumptions to cover more general feature geometries is a direction for future work.
The practical implementation omits the global mean centering step used in the theoretical proofs, as centering was found to degrade performance on ResNet-50 due to non-negative feature geometry induced by ReLU activations.