A Geometry-Based View of Mahalanobis OOD Detection

This paper reveals that the reliability of Mahalanobis-based out-of-distribution detection is highly dependent on the geometric properties of the feature space, specifically within-class spectral structure and local intrinsic dimensionality, and proposes a radially scaled $\ell_2$ normalization method that dynamically adjusts feature radii to optimize detection performance based on these geometric signals.

The Gist

The Big Picture: The "Security Guard" Problem

Imagine you have a very smart security guard (an AI model) who has spent years studying photos of cats and dogs. Their job is to identify if a new photo is a cat or a dog.

But what happens if someone hands the guard a photo of a toaster or a cloud?

• The Problem: The guard might get confused. They might say, "That's definitely a cat!" with 99% confidence, even though it's a toaster. This is dangerous in real life (e.g., a self-driving car thinking a plastic bag is a pedestrian).
• The Goal: We need a way to tell the guard, "Hey, stop! That's not a cat or a dog. That's something weird. Don't guess." This is called Out-of-Distribution (OOD) Detection.

The Old Tool: The "Mahalanobis Ruler"

For a long time, the best tool for this job was called the Mahalanobis Distance. Think of this as a special ruler that measures how far away a new photo is from the "center" of the cats and dogs the guard knows.

• How it works: If the photo is close to the center of the "cat cloud," it's a cat. If it's far away, it's weird.
• The Catch: The paper found that this ruler is unreliable. Sometimes it works perfectly; other times, it fails miserably.
• Why? It turns out the ruler's accuracy depends entirely on how the guard sees the world. If the guard was trained on a specific type of data, the "shape" of their mental map changes. A ruler that works on a flat map might fail on a mountainous one.

The Discovery: It's All About "Shape"

The authors realized that the secret to making the ruler work isn't changing the ruler itself, but understanding the geometry (the shape) of the data.

They looked at two main features of the data's shape:

1. The "Cluster Tightness" (Spectral Structure): How tightly are the cats and dogs huddled together? Are they in a tight ball, or are they scattered loosely?
2. The "Local Dimension" (Intrinsic Dimensionality): How many directions can a cat wiggle? Is a cat just a 2D drawing, or does it have 3D depth?

The Analogy:
Imagine the "Cat" data is a balloon.

• If the balloon is tight and smooth (low dimension, tight cluster), the ruler works great.
• If the balloon is wrinkly, stretched, and full of holes (high dimension, loose cluster), the ruler gets confused.

The paper found a simple formula (a "summary") that combines these two shape features. If you know the shape of the data, you can predict whether the ruler will work or fail.

The Solution: The "Radial Squeeze" (The Magic Knob)

Since the shape of the data changes depending on how the AI was trained, the authors invented a magic knob to fix the shape after the AI is trained.

They call this Radially Scaled Normalization.

The Analogy:
Imagine the data points are people standing in a room.

• Some people are standing very close to the center (small radius).
• Some are standing far away near the walls (large radius).
• The "ruler" gets confused because the room is messy.

The authors introduced a knob (called $\beta$) that acts like a shrink-wrap machine:

• Turn the knob one way: It pulls everyone who is far away closer to the center, and pushes everyone who is too close slightly out. It smooths out the room.
• Turn it the other way: It does the opposite.

By adjusting this knob, they can reshape the room so that the "ruler" (the Mahalanobis detector) works perfectly, without needing to retrain the AI or see any "toaster" examples.

The Best Part: Tuning Without Seeing the Enemy

Usually, to tune a security system, you need to show it examples of the enemy (toasters) to see what works. But in the real world, you don't know what the "toasters" will look like.

The authors found a clever trick:

• You can look at the shape of the "Cat" room (the training data) alone.
• By measuring the "tightness" and "wiggle-room" of the cats, you can mathematically predict exactly where to set the magic knob ($\beta$).
• This allows you to tune the system to be super-accurate at spotting weird stuff, without ever seeing a single piece of weird stuff.

Summary in One Sentence

The paper shows that AI security guards fail because the "shape" of their knowledge changes, but we can fix this by using a simple mathematical "shrink-wrap" tool to reshape the data, making the security guard much better at spotting weird, dangerous inputs.

Technical Summary

1. Problem Statement

Out-of-Distribution (OOD) detection is critical for the reliable deployment of vision models, yet Mahalanobis-based detectors, despite being strong baselines, exhibit highly inconsistent performance across modern pretrained representations (e.g., Vision Transformers, CLIP, EVA).

• The Core Issue: The same quadratic detector can succeed on one model and fail on another. Performance shifts drastically based on pretraining data, fine-tuning regimes, and feature normalization.
• The Gap: It remains unclear which intrinsic properties of the in-distribution (ID) feature space determine whether a Mahalanobis detector will succeed or fail. Furthermore, existing methods often rely on fixed normalization (e.g., projecting to the unit sphere) without a principled way to adapt to specific model geometries.

2. Methodology

The authors approach the problem through the lens of representation geometry, conducting a large-scale study across diverse foundation models and Mahalanobis variants.

A. Geometric Analysis of Representations

The authors analyze the internal geometry of feature spaces to identify predictors of OOD performance. They focus on two complementary families of metrics:

1. Manifold Metrics: Specifically Local Intrinsic Dimensionality (LID), which measures the local degrees of freedom in the feature neighborhood.
2. Spectral Metrics: Derived from the eigenvalue spectra of covariance matrices (Global Covariance $C$, Within-class Scatter $S_w$, Between-class Scatter $S_b$).

They identify a two-term ID summary that consistently tracks Mahalanobis OOD behavior:
$$\text{Proxy} = m \cdot |s|$$
Where:

• $m$: Local Intrinsic Dimensionality (LID).
• $|s|$: The magnitude of the within-class spectral slope (decay rate of eigenvalues in $S_w$).

Key Insight: There is a compensatory trade-off. If the local manifold is simple (low LID), reliable detection requires very compact clusters (steep spectral decay). If the manifold is rich (high LID), OOD samples can deviate in many directions, requiring less compact clusters. The product $m \cdot |s|$ captures this balance.

B. Radially Scaled Normalization ($\phi_\beta$)

Motivated by the geometric view, the authors introduce a post-hoc control mechanism to reshape the feature space without altering the backbone or detector form. They propose radially scaled $\ell_2$ normalization:
$$\phi_\beta(z) = \frac{z}{\|z\|^\beta}$$

• $\beta = 0$: Original features (identity).
• $\beta = 1$: Standard unit-sphere normalization (Mahalanobis++).
• $\beta > 1$: Contracts large norms, pushing features toward the unit sphere.
• $\beta < 1$ (or negative): Expands norms, pushing features away from the sphere.

This transformation preserves feature directions (angular class structure) while modifying radii (norms), thereby changing the geometry presented to the quadratic detector.

C. ID-Only $\beta$ Selection

Since the optimal $\beta$ varies by model and dataset, the authors propose a selection rule that requires no OOD data:

1. Compute the geometric proxy $P(\beta) = m(\beta) \cdot |s(\beta)|$ over a grid of $\beta$ values using only ID data.
2. Select the $\hat{\beta}$ corresponding to the most pronounced interior turning point (maximum or minimum) of the proxy curve.
3. Apply this $\hat{\beta}$ to the features before computing the Mahalanobis score.

3. Key Contributions

1. Large-Scale Benchmark: A comprehensive evaluation of Mahalanobis-style detectors (MD, RMD, MMD) across diverse SSL and foundation models (ViT, BEiT, EVA, CLIP), revealing that performance is highly representation-dependent and not solely correlated with classification accuracy.
2. Geometric Predictor: The identification of the $m \cdot |s|$ product (LID $\times$ spectral slope) as a robust, detector-invariant summary that predicts OOD performance. This links detector reliability to measurable geometric properties of the ID space.
3. Radial Scaling Mechanism: The introduction of $\phi_\beta(z)$, a continuous geometric control knob that adjusts feature radii.
4. Practical Selection Rule: An ID-only procedure to select $\beta$ based on the geometric proxy, which approaches "oracle" performance (tuned with OOD data) without needing access to OOD samples.

4. Results

• Performance Gains: The Radially Scaled (RS) variants (RS-MD and RS-RMD) consistently outperform fixed baselines ($\beta=0$ and $\beta=1$) across diverse models and OOD benchmarks (NINCO, iNaturalist, SSB-Hard, etc.).
  • On average, RS-MD achieves a lower FPR@95 than standard MD and MD++ (unit sphere).
  • The proxy-selected $\hat{\beta}$ significantly reduces the error gap compared to the oracle optimal $\beta$.
• Geometry-Performance Correlation:
  • RMD correlates strongly with within-class geometry ($S_w$), emphasizing the need for compact clusters.
  • MMD correlates with global geometry ($C$), depending on the overall manifold shape.
  • The study confirms that large per-direction separation does not guarantee low FPR; the interaction between the quadratic weighting and the eigen-spectrum is crucial.
• Stability Analysis: The authors provide a theoretical "Unified Stability Lens," decomposing the instability of Mahalanobis scores into a size channel (norms) and a stretch channel (alignment with the whitening geometry). They show that the proxy $m \cdot |s|$ effectively tracks the instability trajectory along the $\beta$ parameter.

5. Significance and Impact

• Reliability in Deployment: The paper provides a practical, post-hoc tool to improve the reliability of vision models in safety-critical applications (e.g., medical imaging, autonomous driving) by reducing false positives without retraining.
• Theoretical Insight: It shifts the understanding of OOD detection from a purely statistical problem to a geometric one, demonstrating that the success of Mahalanobis detectors depends on the interplay between local dimensionality and spectral decay.
• Generalizability: The proposed method is model-agnostic and works across various architectures (ViT, CNNs, CLIP) and training regimes (pretrained, fine-tuned, linear probing), offering a unified solution to the variability of OOD detection performance.
• Efficiency: The selection of $\beta$ requires only ID data and simple geometric computations, making it computationally cheap and easy to integrate into existing pipelines.

In summary, the paper argues that normalization is a geometric control mechanism. By dynamically adjusting the radial geometry of features via $\beta$, one can align the feature space with the assumptions of the Mahalanobis detector, significantly improving OOD detection robustness across modern foundation models.