MaRS: Robust Out-of-Distribution Detection via Mahalanobis Residual Scoring

Technical Summary: MaRS – Robust Out-of-Distribution Detection via Mahalanobis Residual Scoring

Problem Statement
Foundation models provide highly descriptive representations for medical images, yet their reliability degrades under distribution shifts caused by changes in patient populations, imaging devices, or acquisition conditions. While Out-of-Distribution (OOD) detection is essential for safe clinical deployment, existing methods face specific limitations. Confidence-based methods often suffer from fragility due to class imbalance and miscalibration. Distance-based methods (e.g., Deep kNN, Mahalanobis) can degrade in high-dimensional embeddings where nearest-neighbor distances become less informative and covariance estimates are difficult to regularize. Reconstruction-based methods, which learn an in-distribution (ID) manifold and flag samples with large reconstruction errors, have seen limited adoption due to inconsistent performance. The authors posit that this inconsistency does not stem from poor reconstruction quality, but rather from how reconstruction errors are scored. Standard approaches utilize uniform $L_2$ residual norms, which collapse the anisotropic structure of residuals, thereby suppressing informative deviations that occur in low-variance directions.

Methodology: MaRS
The authors introduce MaRS (Mahalanobis Residual Scoring), a label-free, fully post-hoc OOD detector designed to operate on frozen backbone features. The method consists of three primary components:

1. Manifold Projection via Autoencoder: Given a frozen backbone $\Phi$ producing features $z$, a lightweight autoencoder $(E, D)$ is trained on ID features to minimize the reconstruction loss $\|z - \hat{z}\|_2^2$. Geometrically, this implements a projection $\Pi_{M_{ID}}(z)$ onto the ID manifold. The reconstruction residual is defined as $r(x) = z - \hat{z}$.
2. Residual Space Analysis: The authors observe that while ID residuals are small and reflect reconstruction imperfections, OOD residuals contain systematic components reflecting deviation from the learned manifold. Crucially, the distribution of these residuals is highly anisotropic.
3. Mahalanobis Scoring: Instead of a uniform $L_2$ norm, MaRS computes the OOD score using the Mahalanobis distance in the residual space:
   $$S_{MaRS}(x) = r(x)^\top \Sigma^{-1} r(x)$$
   where $\Sigma$ is the covariance matrix of the ID residuals. Through eigendecomposition $\Sigma = U\Lambda U^\top$, the score is expressed as a weighted sum of squared projections onto principal components:
   $$S_{MaRS}(x) = \sum_{i=1}^{D} \frac{(u_i^\top r(x))^2}{\lambda_i}$$
   This formulation amplifies deviations along low-variance (stable) directions (where $\lambda_i$ is small) and downweights high-variance directions (where $\lambda_i$ is large). Unlike Mahalanobis methods in embedding space that rely on class-conditional means, MaRS is label-free and relies on a single global covariance of the residuals, effectively suppressing class-semantic variation to isolate distributional drift.

Key Contributions

• MaRS Framework: A plug-and-play, label-free OOD detector that applies Mahalanobis distance to autoencoder reconstruction residuals. It operates directly on frozen backbone features, ensuring computational efficiency and adaptability across modalities without requiring backbone fine-tuning.
• Identification of Residual Anisotropy: The paper identifies residual anisotropy as a critical, previously overlooked signal for OOD detection. It demonstrates that variance-aware Mahalanobis scoring in residual space substantially outperforms uniform $L_2$ aggregation.
• Robustness to Architecture and Scale: The authors show that MaRS remains robust across different backbone architectures (supervised ViT and self-supervised DINOv3) and model scales (Small and Base). Furthermore, the method benefits from operating on pre-normalization features, where the intrinsic anisotropic variance structure is preserved, unlike latent-space baselines that require normalization.

Experimental Results
Extensive experiments were conducted across three imaging modalities (histopathology, chest X-ray, dermatoscopy), multiple types of distribution shifts (covariate, near-OOD, far-OOD, semantic), and different model families.

• Performance: MaRS consistently outperformed established confidence-based (MSP, ODIN, ViM), distance-based (Deep kNN, OCSVM, Mahalanobis++), and reconstruction-based (Residual, AE) baselines.
• Statistical Significance: MaRS significantly outperformed Mahalanobis++ (the strongest latent-space baseline) across 3 runs and 5 dataset splits in terms of AUROC and FPR@95 (paired Wilcoxon signed-rank test, $p < 0.05$).
• Ablation Findings: Spectral analysis of the residual covariance revealed that over 90% of the variance is concentrated in fewer than one-third of the principal components. OOD deviations were found to concentrate in the remaining low-variance directions. The ratio of OOD to ID residual energy was 1.64$\times$ larger in the low-variance subspace, confirming that Mahalanobis scoring effectively isolates these informative signals.
• Normalization Sensitivity: While normalization improves performance for standard latent-space baselines, it degrades MaRS by flattening the variance spectrum and suppressing the low-variance directions where OOD signals are most pronounced.

Significance
The paper claims that a central limitation of existing reconstruction-based OOD detectors is the implicit assumption of isotropic reconstruction errors. By replacing uniform $L_2$ aggregation with covariance-aware Mahalanobis scoring, MaRS addresses this limitation. The authors assert that this approach provides a simple yet powerful principle for post-hoc OOD detection on frozen representations, offering a robust, label-free solution that generalizes well across medical imaging modalities and model scales without relying on class labels or predefined subspace assumptions. The work suggests that future improvements in medical OOD detection may depend on better modeling of residual space anisotropy rather than solely on improving reconstruction quality or backbone representations.