💻 computer science

Unified Multi-Layer Subspace Modeling for Cross-Domain OOD Detection

The paper introduces PRISM, a model-agnostic post-hoc method that unifies multi-layer features into a single hierarchical embedding to detect out-of-distribution inputs by combining class-conditional Mahalanobis distance and residual energy, achieving state-of-the-art cross-domain performance with a single default configuration and minimal inference overhead.

Original authors: Gerhard Krumpl, Henning Avenhaus, Horst Possegger

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gerhard Krumpl, Henning Avenhaus, Horst Possegger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence has become remarkably good at recognizing patterns. When shown thousands of photographs of cats, dogs, or cars, these systems learn to identify those specific objects with high accuracy. However, a fundamental flaw remains: these systems are often overconfident. If you show a trained cat-recognizer an image of a toaster, the system might not simply say, "I don't know." Instead, it might confidently declare, "That is a very strange cat." This happens because the system was only trained on cats; it has no concept of what lies outside its training. In high-stakes fields like medical imaging or industrial safety, such a mistake is dangerous. A system that confidently misdiagnoses a patient or misses a crack in a machine part because it assumes the anomaly is just a weird version of a known object can lead to silent, catastrophic failures. The goal of researchers in this field is to build a safety mechanism that can reliably spot when an input belongs to a category the system has never seen before.

For years, scientists have tried to solve this by looking at the internal workings of these neural networks. These networks process images through many layers, much like peeling an onion. The early layers detect simple things like edges and colors, while the deeper layers recognize complex shapes and objects. Most existing methods for spotting unknown inputs rely on just one of these layers. Some look only at the final decision the network makes, while others examine the features just before that final step. The problem is that no single layer is perfect for every situation. Sometimes the early layers hold the best clues that an image is strange; other times, the deep layers are more telling. Because the "best" layer changes depending on the type of image and the specific problem, methods that pick just one layer often fail when moved to a new environment. Other approaches try to combine the signals from multiple layers, but they usually require a calibration step where they are tuned using examples of the very unknown data they are supposed to detect. This creates a paradox: to build a detector for the unknown, you first need to see examples of the unknown, which defeats the purpose of a general safety tool.

In a new study, researchers propose a method called PRISM that avoids these pitfalls by treating the entire network as a single, unified system rather than a collection of separate layers. Instead of picking one layer to trust or averaging the scores from several layers, PRISM gathers information from the shallow, middle, and deep parts of the network all at once. It stitches these different views together into a single, comprehensive representation of the image. The researchers then use a statistical technique to map out the shape of the "normal" data within this combined space. They create a model of what typical, known images look like when viewed through all these layers simultaneously. When a new image arrives, the system checks two things. First, it measures how far the image sits from the center of the known data within that combined space. Second, it checks if the image has any features that point in a direction the system has never seen before, essentially measuring how much of the image cannot be explained by the known patterns.

The researchers tested this approach across a wide variety of real-world scenarios, including natural photographs, medical scans like MRIs and endoscopic videos, and industrial inspection tasks. They compared PRISM against twenty-two other state-of-the-art methods. The results showed that PRISM consistently outperformed the others, achieving the highest average accuracy across all these different domains without needing any special tuning for each one. Crucially, it did this using only data from the known, "in-distribution" examples, requiring no examples of the unknown to calibrate its settings. While other methods performed well in one specific area, such as medical imaging, they often struggled when applied to industrial photos or natural scenes. PRISM, by contrast, maintained strong performance everywhere. The study also found that the method adds almost no extra time to the computer's processing speed, making it practical for real-time use.

The core finding of this work is that the most reliable way to detect the unknown is not to look for a single "best" layer or to rely on pre-tuned combinations, but to fuse the entire depth of the network into a single, coherent view. By modeling the geometry of the known data across all layers at once, the system becomes robust to the specific quirks of different datasets. The researchers demonstrated that this approach eliminates the need for the unstable calibration steps that plague current methods. They showed that when you stop trying to guess which part of the network is most important and instead let the whole network speak at once, you get a detector that is far more consistent and reliable. This suggests that for safety-critical applications, the path forward lies in unified, multi-layer modeling that respects the full complexity of the data, rather than simplifying the problem by looking at just one slice of the network.

Technical Summary: PRISM for Cross-Domain OOD Detection

Problem Statement

Out-of-Distribution (OOD) detection is a critical challenge for deploying neural networks in real-world safety-critical systems. While models often achieve high accuracy on in-distribution (ID) data, they can produce overconfident, incorrect predictions on inputs that deviate from the training distribution. Existing post-hoc OOD detection methods typically rely on a single representation level (e.g., logits or penultimate features) or aggregate scores from multiple layers using depth selection or OOD-calibrated weighting.

The authors identify a fundamental limitation in these approaches: the most informative representation level for OOD detection varies significantly across different OOD types and application domains (natural images, medical imaging, industrial inspection). Consequently, fixed-layer choices or aggregation strategies calibrated on specific OOD validation sets are brittle and fail to generalize across domains. Furthermore, many existing multi-layer methods require OOD-dependent calibration or synthetic data to learn layer weights, introducing complexity and reducing transferability to scenarios where representative OOD data is unavailable.

Methodology: PRISM

The paper proposes PRISM (Projected Representation with Intermediate-layer Subspace Modeling), a model-agnostic, post-hoc OOD detection method designed for cross-domain robustness. PRISM adheres to a "fuse-then-model" principle, avoiding layer-wise score aggregation in favor of creating a unified feature representation before statistical modeling.

The method consists of three primary steps:

Unified Multi-Layer Feature Construction:
Instead of selecting a single layer, PRISM extracts feature maps from multiple intermediate layers (from shallow to deep) across the network. For Convolutional Neural Networks (CNNs), spatial feature maps are aggregated via global average pooling; for Vision Transformers (ViTs), token sequences are averaged. Each resulting vector is $\ell_2$ -normalized to prevent layers with larger activation magnitudes from dominating the joint representation. These normalized vectors are concatenated to form a single, high-dimensional multi-layer feature vector $\mathbf{z}(\mathbf{x})$ .
Subspace Modeling of ID Data:
PRISM assumes that ID data lies on a low-dimensional manifold within this fused feature space. Using only ID training data, it applies Principal Component Analysis (PCA) to estimate a principal subspace $\mathbf{W}$ that captures the dominant modes of ID variability. At inference, a feature vector is decomposed into:
- Projection ( $\mathbf{u}$ ): The component aligned with the principal subspace, capturing semantic patterns consistent with ID data.
- Residual ( $\mathbf{r}$ ): The orthogonal component representing features unsupported by the learned ID manifold.
OOD Scoring:
PRISM combines two complementary signals into a single score without learned weights or OOD calibration:
- Projected Mahalanobis Distance ( $s_{\text{proj}}$ ): A class-conditional Mahalanobis distance computed in the projected subspace. This detects deviations from class-conditional structures within the learned manifold.
- Residual Energy ( $s_{\text{res}}$ ): The squared $\ell_2$ norm of the residual vector. This captures energy in directions orthogonal to the ID manifold, detecting novel or unsupported feature directions.
The final OOD score is the negative product of these two terms: $s_{\text{PRISM}}(\mathbf{x}) = -s_{\text{proj}}(\mathbf{x}) \cdot s_{\text{res}}(\mathbf{x})$ . This multiplicative combination emphasizes samples that deviate from the ID structure both within and outside the principal subspace. All statistics (means, covariance, PCA parameters) are estimated exclusively from ID data.

Key Contributions

Identification of Aggregation Sensitivity: The authors demonstrate that layer-wise aggregation and OOD-based calibration are highly sensitive to the chosen validation OOD set, leading to unstable cross-domain behavior. They show that optimal layer weights vary significantly across different medical and natural image shifts.
Unified Multi-Layer Subspace Modeling: PRISM introduces a principle that eliminates the need for layer selection, score aggregation heuristics, and OOD-dependent calibration. By fusing representations prior to modeling, it retains the robustness benefits of statistical feature modeling while capturing complementary information from shallow to deep layers.
Comprehensive Cross-Domain Evaluation: The method is evaluated across natural, medical, and industrial benchmarks (OpenOOD, OpenMIBOOD, ICONIC-444) using a single default configuration. PRISM achieves state-of-the-art mean AUROC and the best average rank across datasets without requiring OOD-based calibration.
Efficiency: The method incurs minimal inference overhead (less than 1.0% for batch sizes $\geq 16$ ) compared to baselines, making it practical for real-world deployment.

Results

Extensive experiments across 22 state-of-the-art post-hoc methods on diverse benchmarks yield the following findings:

Overall Performance: PRISM achieves the highest mean AUROC (93.92%) across all benchmarks, outperforming the second-best method (MDS++, 89.13%) by 4.79 points.
Domain Robustness: Unlike competing methods that show pronounced domain-dependent variability (e.g., MDSEns performs well on medical data but poorly on ImageNet-1k; GRAM excels in industrial settings but degrades in medical tasks), PRISM maintains consistent high performance across natural, medical, and industrial domains.
OOD Category Robustness: PRISM maintains high AUROC from near-OOD to extreme-OOD categories, indicating robustness to increasing distributional shifts.
Architectural Generalization: PRISM performs consistently well across different backbone architectures, including ResNets, Swin Transformers, and ViTs, without architecture-specific tuning.
Ablation Studies: The paper confirms that both the projected Mahalanobis term and the residual energy term are necessary for optimal performance, as they capture different types of geometric deviations. The unified multi-layer representation is shown to be more effective than single-layer baselines or simple score aggregation.

Significance and Claims

The paper claims that PRISM addresses the instability of current OOD detection methods by prioritizing transferable, ID-only deployment. By fusing shallow-to-deep representations and modeling the unified ID geometry in a low-dimensional subspace, PRISM avoids the pitfalls of benchmark-specific depth preferences and OOD-dependent calibration.

The authors emphasize that while no single post-hoc detector is universally best across all domains and OOD types, PRISM provides a robust, "default" solution that integrates complementary information across representation depths. This approach reduces sensitivity to the specific OOD definitions or validation sets used during development, offering a more reliable mechanism for identifying distributional shifts in diverse real-world applications where OOD data may be scarce or unknown. The method's ability to achieve state-of-the-art performance with a fixed configuration across natural, medical, and industrial domains underscores the value of unified multi-layer subspace modeling for cross-domain robustness.