📊 statistics

A Bayesian Nonparametric Perspective on Mahalanobis Distance for Out of Distribution Detection

This paper establishes a formal connection between Bayesian nonparametric models and the relative Mahalanobis distance score for out-of-distribution detection, proposing generalized hierarchical mixture models that demonstrate superior performance, particularly when training classes exhibit varying covariance structures or limited data.

Original authors: Randolph W. Linderman (Electrical and Computer Engineering Department, Duke University, Durham, NC, USA), Noah Cowan (Statistics Department, Stanford University, Stanford, CA, USA), Yiran Chen (Electr

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Randolph W. Linderman (Electrical and Computer Engineering Department, Duke University, Durham, NC, USA), Noah Cowan (Statistics Department, Stanford University, Stanford, CA, USA), Yiran Chen (Electrical and Computer Engineering Department, Duke University, Durham, NC, USA), Scott W. Linderman (Statistics Department, Stanford University, Stanford, CA, USA, The Wu Tsai Neurosciences Institute, Stanford University, Stanford, CA, USA)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a security guard at a very exclusive art gallery. You have spent years studying the paintings in the gallery (your training data). You know exactly how a "real" painting looks, feels, and behaves.

One day, a stranger walks in holding a strange object. Is it a masterpiece from a new artist you haven't seen yet (an Out-of-Distribution or OOD sample), or is it just a weirdly shaped trash can (an outlier)? Your job is to spot the trash can immediately and sound the alarm.

This paper is about building a smarter security guard using a specific type of math called Bayesian Nonparametrics. Here is how the authors explain their approach, using simple analogies.

The Old Way: The "One-Size-Fits-All" Ruler

For a long time, security guards used a simple tool called the Mahalanobis Distance (specifically, a version called RMDS).

Think of this like a ruler that assumes every single painting in the gallery has the exact same shape and texture.

If a painting is slightly stretched, the ruler says, "That's weird!"
If a painting is slightly crumpled, the ruler says, "That's weird!"

The problem is that in the real world, not all paintings are the same. Some are wide and flat; others are tall and thin. Some are painted on rough canvas; others on smooth silk. If you force a "one-size-fits-all" ruler to measure them all, you might miss the trash can because it looks like a "normal" wide painting, or you might falsely alarm because a tall painting looks "abnormal" to your flat ruler.

The New Idea: The "Flexible, Shared Wisdom" Approach

The authors propose a new security guard based on Dirichlet Process Mixture Models (DPMMs).

Instead of assuming every painting is the same, this new guard understands that:

There are many types of paintings: The gallery might have 100 different styles (clusters).
New paintings might appear: There is always a chance a new style exists that the guard has never seen before.
They share wisdom: Even though every painting style is different, they can still learn from each other.

The "Shared Wisdom" Analogy (Hierarchical Priors)

Imagine the gallery has 100 different art departments.

The Old Way: Each department tries to measure its own paintings using its own ruler. If a department only has 5 paintings, their ruler is shaky and unreliable.
The New Way (Hierarchical): All departments share a "Master Ruler" that they can borrow from. If a department has very few paintings, they lean heavily on the Master Ruler. If a department has thousands of paintings, they trust their own measurements more.

This is what the authors call Hierarchical Priors. It allows the system to be flexible (admitting different shapes) but also stable (not getting confused when data is scarce).

The Big Discovery: The Old and New are Cousins

The paper makes a surprising discovery. The authors proved mathematically that the old "One-Size-Fits-All" ruler (RMDS) is actually just a special, simplified version of their new "Flexible" system.

Think of it like this: The old ruler is like a black-and-white photo of the scene. The new system is a high-definition color video. The authors showed that if you squint at the video and turn off the color, it looks exactly like the black-and-white photo. This means the old method wasn't "wrong"; it was just a simplified version of a more powerful idea.

The New Tools: Three Types of Flexible Guards

Because they realized the old method was just a simplified version, the authors built three new, more powerful guards to handle different situations:

The Full-Body Guard (Full Covariance): This guard looks at every detail of the painting's shape and texture simultaneously. It's very accurate but needs a lot of memory and time. It works best when you have a huge gallery with many paintings per style.
The Quick-Scan Guard (Diagonal Covariance): This guard looks at the width and height separately, ignoring how they might interact. It's faster and needs less memory. It works well when the gallery is huge but you don't have enough time to look at every detail.
The "Linked" Quick-Scan Guard (Coupled Diagonal): This is the authors' novel invention. It's like the Quick-Scan guard, but it notices that if the width of a painting is "weird," the height is probably "weird" too. It links these two ideas together. This turned out to be the best performer in many tests, especially when the gallery had many styles but not many examples of each.

What Did They Find?

The authors tested these new guards on a famous benchmark called OpenOOD (a giant test of different art galleries and trash cans).

When the gallery is small and crowded: The new "Linked" guard was the best at spotting the trash cans.
When the gallery has very few examples of each style: The new guards were much better than the old ruler because the old ruler got confused by the lack of data.
When the styles are very different from each other: The new guards adapted better because they didn't force everything into the same "box."

The Bottom Line

The paper argues that we shouldn't just rely on simple, rigid rulers (like the old RMDS) to detect weird data. Instead, we should use Bayesian Nonparametric models that act like a flexible, shared-wisdom system.

By treating the problem as "figuring out if this new item belongs to a known group or a brand new group," and by letting different groups share statistical strength, we can build security systems that are much better at spotting the trash cans in the art gallery, especially when the gallery is messy or the examples are scarce.

In short: The paper takes a complex, flexible math framework, shows it explains why our current simple tools work, and then uses that framework to build better, more adaptable tools that outperform the old ones in tricky situations.

Technical Summary: A Bayesian Nonparametric Perspective on Mahalanobis Distance for Out of Distribution Detection

Problem Statement

Machine learning systems, particularly those built on foundation models trained on massive datasets, inevitably encounter data that deviates from their training distributions. Detecting these out-of-distribution (OOD) samples is critical for safety and reliability. While Bayesian nonparametric methods offer a coherent probabilistic framework for estimating the probability that a data point belongs to a novel cluster, they have largely been eschewed in favor of simpler distance-based metrics. A primary barrier to adopting Bayesian approaches is the computational and statistical challenge of estimating high-dimensional covariance matrices for feature embeddings, especially when the number of training samples per class is small relative to the embedding dimension.

Methodology

Theoretical Connection: RMDS and DPMMs

The authors establish a formal theoretical relationship between the widely used Relative Mahalanobis Distance Score (RMDS) and Bayesian nonparametric inference. Specifically, they demonstrate that the RMDS is a close approximation of the outlier probability derived from a Gaussian Dirichlet Process Mixture Model (DPMM) with tied covariance matrices (where all classes share a single covariance matrix).

Proposition 4.1: Shows that the inlier probability of a general DPMM can be expressed as a logistic function of a score $\tilde{C}(x)$ , which is analogous to the relative Mahalanobis score.
Proposition 4.2: Proves that under specific regimes (sub-Gaussian data, equal cluster sizes, and specific conditions on the relative covariance matrix), the DPMM score from a tied-covariance model is almost perfectly correlated with the RMDS. This connects the heuristic RMDS approach to rigorous Bayesian inference.

Hierarchical Gaussian DPMMs

Recognizing that the tied-covariance assumption (and thus RMDS) may not hold when classes exhibit significantly different covariance structures, the authors propose Hierarchical Gaussian DPMMs. These models relax the tied-covariance assumption by inferring separate covariance matrices for each cluster while sharing statistical strength across classes via a hierarchical prior.

The paper introduces three specific model variants:

Full Covariance Model: Uses a conjugate Normal-Inverse Wishart (NIW) prior. This allows for full covariance matrices but risks overfitting in high dimensions.
Diagonal Covariance Model: Assumes variances are independent across dimensions, using a scaled inverse chi-squared prior. This reduces parameter count but assumes independence between dimensions.
Coupled Diagonal Covariance Model: A novel approach that captures correlations between dimensions by introducing a class-specific scale factor $\gamma_k$ . This factor scales the variances for all dimensions of a specific class, allowing the model to capture systematic variations in variance magnitude across classes while maintaining computational efficiency.

Estimation

The authors develop efficient Expectation-Maximization (EM) algorithms to estimate the hyperparameters of these hierarchical models.

For the Full Covariance and Diagonal models, the M-steps for hyperparameters (concentration $\nu_0$ and precision scaling $\kappa_0$ ) are computed in closed form or via a generalized Newton method.
For the Coupled Diagonal model, the posterior over the scale factor $\gamma_k$ does not have a closed form. The authors approximate this using numerical quadrature over a dense grid, leveraging the fact that $\gamma_k$ is one-dimensional.

Key Results

Synthetic Experiments

Simulations on 2D data confirmed that hierarchical models outperform tied-covariance models (and RMDS) when:

Covariance matrices differ significantly across clusters (low $\nu_0$ ).
The number of data points per class ( $N_k$ ) is small relative to the dimensionality ( $D$ ).
In regimes where $N_k/D$ is small, independent RMDS (using per-class covariances without hierarchical priors) fails due to overfitting, whereas the hierarchical DPMMs maintain robust performance.

OpenOOD Benchmark Evaluation

The models were evaluated on the OpenOOD benchmark using CIFAR-10, CIFAR-100, and ImageNet-1K datasets with various OOD datasets (Near and Far).

ImageNet-1K: The Coupled Diagonal model achieved the best performance on Near OOD detection, outperforming the standard RMDS and tied DPMM. The Full Covariance model performed poorly, likely due to overfitting in the high-dimensional (768D) space.
CIFAR-10: The Full Covariance hierarchical model yielded the highest performance on both Near and Far OOD tasks, suggesting that for lower-dimensional embeddings (512D) and smaller datasets, the full covariance structure is learnable and beneficial.
Dimensionality Analysis: As feature dimensionality increased, the Full Covariance model's performance plateaued or degraded, while the Diagonal and Coupled Diagonal models maintained superior performance, particularly on Near OOD benchmarks.

Significance and Claims

The paper claims that Bayesian nonparametric methods with hierarchical priors provide a well-grounded and practically competitive approach to OOD detection. The primary contributions are:

Theoretical Unification: Providing a formal link between the heuristic RMDS and Bayesian inference under a Gaussian DPMM, validating RMDS as a specific limit of a more general probabilistic framework.
Generalization: Proposing hierarchical models that generalize RMDS by relaxing the equal covariance assumption, thereby adapting to scenarios where class covariances differ.
Practical Utility: Demonstrating that these models can improve OOD detection, particularly in regimes characterized by:
- Heterogeneous covariance structures across classes.
- Limited data per class relative to feature dimensionality.
- The need for fine-grained distinction between in-distribution and "Near OOD" samples.

The authors note that while the full covariance model can overfit in high dimensions, the Coupled Diagonal model offers a robust compromise, capturing inter-class variance correlations without the parameter explosion of full covariance estimation. They conclude that casting OOD detection as inference in a generative model allows for principled modeling choices that can lead to performance improvements over existing distance-based baselines.