DAVIS: OOD Detection via Dominant Activations and Variance for Increased Separation

1. Problem Statement

The safe deployment of Machine Learning (ML) models in real-world scenarios requires robust Out-of-Distribution (OOD) detection. Models must recognize inputs that fall outside their training distribution (e.g., novel contexts or unknown classes) and signal uncertainty rather than making confident, incorrect predictions.

The Core Limitation:
Most existing post-hoc OOD detection methods operate on feature vectors derived from Global Average Pooling (GAP) applied to the penultimate layer of a neural network.

• Information Loss: GAP is a lossy operation that summarizes each channel's spatial activation map into a single scalar (the mean).
• Discarded Signals: This process permanently discards valuable distributional statistics, specifically the peak intensity (dominant activation) and the variance (spread) of the activation maps.
• Structural Mismatch: While GAP promotes spatial invariance beneficial for classification, it suppresses the very statistical cues (peaks and variance) that are highly discriminative for distinguishing between In-Distribution (ID) and OOD samples.

2. Methodology: DAVIS

The authors propose DAVIS (Dominant Activations and Variance for Increased Separation), a simple, post-hoc, plug-and-play module that enriches feature representations before downstream scoring.

Key Mechanism:
Instead of relying solely on the mean activation ($\mu$) produced by GAP, DAVIS extracts richer statistics from the pre-pooling activation maps ($g(x) \in \mathbb{R}^{n \times k \times k}$):

1. Mean ($\mu$): The standard channel-wise average.
2. Dominant Activation ($m$): The maximum activation value in each channel.
3. Standard Deviation ($\sigma$): The channel-wise standard deviation.

Feature Formulations:
DAVIS proposes two ways to construct the enriched feature vector $h_{DAVIS}(x)$:

1. DAVIS($\mu, \sigma$): Augments the mean with scaled standard deviation.
   $$h_{DAVIS}(x) = \mu(x) + \gamma \sigma(x)$$
   Where $\gamma$ is a hyperparameter.
2. DAVIS($m$): Replaces the mean feature vector entirely with the dominant (maximum) activation.
   $$h_{DAVIS}(x) = m(x)$$

Integration:

• Model-Agnostic: DAVIS does not require retraining. It modifies the penultimate layer representation before it is passed to the final classification layer.
• Compatibility: It seamlessly integrates with existing OOD scoring functions, including:
  • Logit-based (MSP, ODIN)
  • Energy-based (Energy score)
  • Gradient-based (GradNorm)
  • Distance-based (kNN)
• Workflow: The enriched vector $h_{DAVIS}(x)$ is passed through the original fully connected layer to produce modified logits, which are then used to compute the OOD score.

3. Theoretical Analysis

The paper provides a statistical analysis proving why DAVIS improves separation:

• Observation 1: For well-trained models, ID samples consistently exhibit higher magnitudes in mean, maximum, and standard deviation statistics compared to OOD samples.
• Separation Gap: The authors define a separation gap ($\Delta$) between ID and OOD distributions. They prove mathematically that incorporating standard deviation ($\Delta_{\mu,\sigma}$) or using the maximum ($\Delta_m$) yields a larger separation gap than using the mean alone ($\Delta_\mu$).
• Logit Propagation: Under the assumption that the classifier's weight matrix has non-negative row sums (a property often observed empirically), the increased feature separation directly translates to increased separation in the output logit space, leading to better OOD scoring.

4. Key Contributions

1. Identification of GAP Limitations: The paper identifies the information loss from Global Average Pooling as a critical, overlooked weakness in post-hoc OOD detection.
2. DAVIS Framework: Introduction of a simple, complementary module that enriches feature representations using pre-pooling statistics (max and variance).
3. Extensive Empirical Validation: Evaluation across diverse architectures (ResNet, DenseNet, EfficientNet, Swin-B, ConvNeXt-B) and datasets (CIFAR-10/100, ImageNet-1K).
4. Robustness to Modern Architectures: Demonstration that DAVIS is particularly effective on modern models (e.g., Swin Transformers, ConvNeXt) where traditional baselines often fail due to differences in normalization (LayerNorm vs. BatchNorm) and activation functions (GeLU/SiLU vs. ReLU).

5. Experimental Results

The authors evaluated DAVIS on standard benchmarks (OpenOOD) and near-OOD scenarios.

Performance Highlights:

• ImageNet Benchmarks:
  • Swin-B: DAVIS($\mu, \sigma$) + KNN reduced FPR95 by 25.09% compared to the previous best (GEN).
  • ConvNeXt-B: DAVIS($\mu, \sigma$) + KNN reduced FPR95 by 16.53% compared to CARef.
  • ResNet-50: DAVIS($\mu, \sigma$) + SCALE reduced FPR95 by 7.96% compared to SCALE.
• CIFAR Benchmarks:
  • On CIFAR-10, DAVIS + DICE reduced FPR95 by 52.08% (ResNet-18) and 45.12% (ResNet-34) compared to strong baselines.
  • On CIFAR-100, significant improvements were observed across all tested architectures.
• Near-OOD Detection: DAVIS demonstrated robustness in semantically challenging near-OOD settings (e.g., SSB-Hard, NINCO), reducing FPR95 by 25.52% on Swin-B compared to ReAct.
• Comparison with SOTA: DAVIS consistently outperformed or matched state-of-the-art methods like ReAct, DICE, ASH, SCALE, and CADRef, often achieving the best results when combined with energy-based or kNN scoring.

6. Significance and Impact

• Paradigm Shift: The work challenges the standard practice of relying solely on GAP-derived mean features for OOD detection, advocating for the retention of spatial distribution statistics.
• Practical Utility: As a post-hoc method, DAVIS requires no retraining and negligible computational overhead (less than 0.1% increase in GFLOPs), making it highly deployable in production systems.
• Modern Model Compatibility: It addresses a specific gap where modern deep learning architectures (using LayerNorm and non-ReLU activations) degrade the performance of traditional OOD detectors. DAVIS leverages fundamental statistical cues that remain robust across these architectural shifts.
• Generalizability: The method works effectively across a wide spectrum of model families, from traditional CNNs to Vision Transformers and hybrid architectures.

In conclusion, DAVIS provides a principled, simple, and highly effective solution to the information loss problem in OOD detection, significantly improving the reliability of ML models in open-world settings.