Technical Summary: Adaptive Resolution Label Aggregation (ARLA)

1. Problem Statement

Reliable ground truth labels are fundamental to supervised deep learning, particularly for segmentation and image-to-image tasks. However, creating such labels is often expensive and difficult, frequently resulting in datasets that are inconsistent, noisy, ambiguous at class boundaries, or incomplete due to factors like cloud cover in remote sensing.

While significant research exists on training models to be robust against weak or noisy labels, a critical gap remains in evaluating models on such unreliable data. Existing benchmarks often rely on test sets with inherent errors (e.g., Northcutt et al. [19] found 6% errors in ImageNet validation), which can lead to incorrect model selection and misleading performance metrics. Current solutions for evaluation—such as ignoring boundary pixels, using soft labels, or aggregating multiple annotators—often fail to exploit the information within unreliable labels or require impossible manual cleaning. Furthermore, no existing method specifically addresses the evaluation of segmentation models on unreliable labels where the error source is unknown or unquantifiable.

2. Methodology: Adaptive Resolution Label Aggregation (ARLA)

The authors propose Adaptive Resolution Label Aggregation (ARLA), a post-hoc evaluation method that dynamically adapts the resolution of both the ground truth label and the model prediction at inference time. ARLA does not require re-training the model.

Core Mechanism

The method operates by partitioning the spatial domain of the label and prediction into an $s \times s$ grid of contiguous subpatches. For each subpatch, the method aggregates the pixel values based on a sensitivity threshold $\tau$:

1. Partitioning: The label $L$ (and prediction $P$) is divided into subpatches $S_{a,b}$.
2. Aggregation Logic:
   • For binary segmentation, the proportion of positive class (1) elements within a subpatch is calculated.
   • If the proportion $\ge \tau$ (sensitivity threshold), the entire subpatch is set to 1.
   • Otherwise, it is set to 0.
   • For multi-class segmentation, the most frequent class in the subpatch is selected, or priority classes are assigned if specific thresholds are met.
3. Metric Calculation: Standard evaluation metrics (e.g., Precision, Recall, F1) are computed on the aggregated label $L'$ and aggregated prediction $P'$.

Parameterization

• Subpatch Size ($s \times s$): Controls the resolution of aggregation. A $1 \times 1$ grid converts the task to binary classification, while $s$ matching the original resolution preserves the original evaluation.
• Sensitivity ($\tau$): Determines how much "noise" or inconsistency is tolerated within a patch before it is considered a positive class.

The method is designed to minimize label noise and inconsistencies (e.g., small erroneous blobs, boundary ambiguities) while preserving the true signal of the model's performance.

3. Key Contributions

• Novel Evaluation Framework: ARLA is presented as the first method specifically designed to evaluate segmentation models on unreliable, noisy, or inconsistent labels without re-training.
• Signal Extraction: It extracts a clearer signal of model performance by exploiting the information encapsulated in labels while minimizing the impact of label errors.
• Flexibility: The method allows for adjustable parameters to adapt the aggregated resolution to the precision of the label or the level of noise, enabling evaluation across a spectrum of granularities.
• Baseline Comparisons: The paper establishes ARLA against two baseline approaches:
  1. Buffer Method: Ignoring pixels within a tolerance buffer around class boundaries.
  2. Soft Label Method: Blending classes at boundaries using fuzzy metrics.

4. Results and Applications

The paper validates ARLA using a real-world flood prediction dataset from the Copernicus Emergency Management Service, characterized by inconsistent forest labeling, cloud cover obscuring flood waters, and ambiguous boundaries.

Case Studies

• Inconsistent Forest Labeling: In scenarios where annotators inconsistently included/excluded flooded forests, ARLA (with appropriate subpatching) correctly identified the model's performance by aggregating the inconsistent regions, whereas standard evaluation reported high errors.
• Cloud Cover Errors: When annotators excluded flood waters hidden by clouds (creating false negatives in the label), a model that correctly predicted these areas was penalized in standard evaluation. ARLA recognized the model's correct prediction by aggregating the cloud-covered regions, significantly improving the reported precision.
• Baseline Performance: In comparative tests (Table 1), ARLA outperformed the Buffer and Soft Label methods. The Buffer method was found to be misleading as it excluded large portions of the label (including valid model errors), while Soft Labels struggled with large discrepancies. ARLA successfully handled large label errors without obscuring true model weaknesses.

Flood Event Analysis (Emilia-Romagna, 2023)

Applied to a severe flood event with heavy cloud cover and noisy automated labels:

• Original Evaluation: The model showed low precision (0.52) due to "over-prediction" in cloud-covered areas where the label was missing.
• ARLA Evaluation: Using a coarse resolution ($1 \times 1$ subpatches, $\tau=2.5\%$), the precision increased to 0.80. The method revealed that the model was actually robust to the cloud cover but struggled with recall in other areas (recall remained 0.63).
• Insight: ARLA shifted the conclusion from "the model over-predicts" to "the model under-predicts in specific areas," providing a more accurate behavioral analysis.

5. Significance and Claims

The authors claim that ARLA offers a reliable approach to evaluate models on unreliable labels. Its significance lies in:

• Transparency: It allows researchers to report model performance across a range of resolutions and sensitivities (e.g., F1 scores varying between 0.5 and 0.7 depending on aggregation), rather than relying on a single, potentially misleading metric derived from noisy ground truth.
• Practicality: By operating at inference time, it avoids the computational cost of re-training.
• Domain Relevance: It is particularly effective for remote sensing and flood prediction, where perfect labels are often impossible to derive due to environmental factors (clouds, resolution limits).

The paper concludes that while ARLA is a powerful new tool, future work is needed to systematically analyze the effects of parameter combinations and develop standardized methods for selecting the optimal resolution based on label error size. The authors emphasize that ARLA parameters should be viewed as tools to evaluate performance under different conditions, not as hyperparameters to be optimized solely to maximize scores.