Better than Average: Spatially-Aware Aggregation of Segmentation Uncertainty Improves Downstream Performance

1. Problem Statement

In safety-critical domains like biomedical imaging and autonomous driving, Uncertainty Quantification (UQ) is essential for detecting unreliable segmentation predictions. While UQ methods generate pixel-wise uncertainty scores, these must be aggregated into a single image-level scalar score to support downstream tasks such as:

• Out-of-Distribution (OoD) Detection: Identifying inputs that differ from the training distribution.
• Failure Detection (FD): Flagging images where the segmentation is likely to contain errors requiring manual review.

The Core Gap: The standard default for aggregation is the Global Average (AVG) across all pixels. However, the authors argue that AVG is suboptimal because it ignores the spatial structure of uncertainty. Critical failure modes (e.g., ambiguous boundaries, unseen object classes) often manifest as localized uncertainty patterns that simple averaging dilutes or obscures. Existing alternatives (patch-based, threshold-based, class-based) lack systematic comparison, leading to inconsistent best practices.

2. Methodology

The paper proposes a comprehensive framework to analyze, improve, and unify uncertainty aggregation strategies.

A. Analysis of Common Strategies (Intensity-Based)

The authors formally analyze the properties of existing aggregation strategies (AggSs), identifying three key pitfalls:

1. Lack of Spatial Sensitivity: Methods like AVG assign identical scores to distinct spatial patterns (e.g., uniform low uncertainty vs. compact high uncertainty), failing to detect localized risks.
2. Lack of Proportion Invariance: Methods like Above-Quantile Average (AQA) produce different scores if the background (low uncertainty) is cropped, even if the high-uncertainty regions remain unchanged.
3. Non-Monotonicity: Methods like Above-Threshold Average (ATA) can yield lower scores when pixel-wise uncertainty increases globally, violating intuitive risk assessment.

B. Novel Spatial Aggregation Strategies

To address the lack of spatial awareness, the authors introduce three new strategies based on established measures for grayscale image structure. They calculate a Spatial Mass Ratio (SMR), which represents the fraction of total uncertainty mass concentrated in regions with high local spatial structure:

• MOR (Moran's I): Measures spatial autocorrelation. High SMR indicates uncertainty is clustered (e.g., around an object), while low SMR indicates noise-like randomness.
• EDS (Edge Density Score): Measures the density of edges. High SMR indicates uncertainty is concentrated along boundaries.
• ENT (Shannon Entropy): Measures local heterogeneity. High SMR indicates uncertainty is in highly variable regions.

C. Meta-Aggregation (GMM-All)

Recognizing that no single strategy performs best across all datasets, the authors propose a Meta-Aggregator:

• Concept: Treat the scores from multiple individual AggSs (both intensity-based and spatial) as a feature vector.
• Implementation: Fit a Gaussian Mixture Model (GMM) on the feature vectors of known In-Distribution (iD) data.
• Scoring: The meta-score is the Negative Log-Likelihood (NLL) of a new sample's feature vector under the fitted GMM. High NLL indicates the sample deviates from the iD distribution (i.e., it is OoD or a failure).
• Variants:
  • GMM-Int: Uses only intensity-based features.
  • GMM-Spa: Uses only spatial features.
  • GMM-All: Combines both (the proposed robust default).

3. Key Contributions

1. Systematic Benchmarking: The first extensive evaluation of AggSs across 10 diverse datasets (pathology, CT scans, microscopy, autonomous driving, agriculture) covering semantic, instance, and panoptic segmentation tasks.
2. Spatially-Aware Aggregators: Introduction of MOR, EDS, and ENT as effective aggregators that capture structural uncertainty patterns ignored by pixel-wise averaging.
3. Meta-Aggregation Strategy (GMM-All): A robust, all-purpose method that combines intensity and spatial features, demonstrating superior stability across diverse scenarios compared to single-strategy baselines.
4. Open-Source Library: Release of aggrigator, a Python library for uncertainty aggregation.

4. Experimental Results

The evaluation focused on AUROC (for OoD detection) and Excess-Area Under the Risk-Coverage Curve (E-AURC) (for Failure Detection).

• Global Average (AVG) is Suboptimal: AVG performed poorly in 6 out of 10 OoD scenarios, often approaching random guessing. It failed to capture localized risks.
• Spatial Strategies Excel: Aggregators leveraging spatial structure (MOR, EDS, ENT) consistently outperformed intensity-only methods in detecting OoD samples and failures, particularly when uncertainty was localized to boundaries or specific clusters.
• Task and Dataset Dependence: No single AggS was universally dominant. For example:
  • Prediction-based methods (Balanced Class Average - BCA, Imbalanced Class Average - ICA) performed well due to proportion invariance.
  • Threshold-based methods (ATA) performed surprisingly well in Failure Detection for specific datasets (e.g., WORM-Nem) where errors concentrated at boundaries, despite poor OoD performance.
• Superiority of GMM-All:
  • GMM-All achieved robust performance across almost all datasets and tasks, often outperforming the best individual AggSs.
  • It successfully handled cases where individual features failed to separate iD from OoD (e.g., in the CAR-CS dataset, EDS was the dominant feature, but GMM-All integrated it with others for stability).
  • SHAP Analysis confirmed that GMM-All effectively utilizes different features depending on the dataset (e.g., EDS for synthetic-to-real shifts, prediction-based ratios for class imbalance).

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the "default" use of Global Average for uncertainty aggregation, demonstrating that spatial context is critical for reliable downstream decision-making.
• Practical Guidance: It provides a clear hierarchy of strategies:
  1. If dataset characteristics are known, select a specific spatial or prediction-aware AggS.
  2. If dataset characteristics are unknown or variable, GMM-All is the recommended robust default.
• Impact: By improving the reliability of uncertainty scores, this work enhances the safety of automated systems in medicine and autonomous driving, ensuring that risky samples are correctly flagged for human intervention.

The authors conclude that while intensity-based aggregation is simple, spatially-aware aggregation preserves critical structural cues, and meta-aggregation offers the most reliable path forward for general-purpose deployment.