Contrastive Mask Fidelity: Reference-Free Auditing of Ground-Truth Masks in Remote Sensing Semantic Segmentation

Technical Summary: Contrastive Mask Fidelity (CMF)

Problem Statement: The Evaluation Paradox
Semantic segmentation in remote sensing faces a fundamental "evaluation paradox." Models are trained and evaluated against human-drawn masks, which are conventionally treated as infallible ground truth. However, remote sensing annotations are frequently coarse, incomplete, misaligned, or over-smoothed due to the complexity of aerial and satellite imagery (e.g., small objects, heterogeneous land cover). Consequently, standard overlap metrics like mIoU quantify agreement with potentially flawed labels rather than faithfulness to the visual evidence. A visually superior mask may receive a lower score if it deviates from an imperfect annotation. Existing label-quality methods often rely on models trained on the same noisy label distribution, inheriting the very biases they aim to detect. There is a lack of a reference-free adjudicator capable of comparing competing masks (human vs. model) directly against the image evidence without designating either as the absolute truth.

Methodology: The GeoVeritas Pipeline
The authors propose GeoVeritas, a three-stage, training-free auditing system centered on Contrastive Mask Fidelity (CMF).

1. Candidate Generation (Stage I):
   The system utilizes Seg-Probe, a training-free open-vocabulary segmentation probe built upon the frozen SegEarth-OV3 backbone (which uses a frozen SAM 3 encoder). Seg-Probe is not the primary contribution but serves as a robust source of candidate masks to challenge human annotations. It employs three training-free components to generate high-quality candidates:

   • Synonym Evidence Pooling (SEP): Aggregates evidence from multiple synonymous prompts using a log-sum-exp soft-OR to reduce reliance on specific class names.
   • Multi-Scale Evidence (MSE): Extracts and aligns evidence from multiple image scales.
   • Test-Time Augmentation (TTA): Averages evidence from original and horizontally flipped views.

2. Contrastive Mask Fidelity Scoring (Stage II):
   CMF evaluates the fidelity of a binary mask $M_c$ for class $c$ against an image $x$ using a frozen, external vision-language judge (RemoteCLIP). The process involves creating two counterfactual views of the image:

   • Keep View ($x^k$): Reveals only the region defined by the mask $M_c$, while blurring the rest.
   • Erase View ($x^e$): Hides the region $M_c$ (blurring it) while revealing the rest of the image.

   The judge computes the probability $P(c|r)$ for each view using a temperature-scaled softmax over class texts. The Fidelity Score $F(M_c)$ is defined as the difference:
   $$F(M_c) = P(c | x^k_c) - P(c | x^e_c)$$
   This contrastive design ensures the score reflects both purity (the kept region contains the class) and completeness (no class evidence remains outside the mask). Crucially, the softmax normalization and the opposing nature of the two terms reduce sensitivity to trivial mask area inflation.

3. Arbitration and Label Correction (Stage III):
   For each image-class pair, the system compares the fidelity score of the candidate mask ($M^p_c$) against the human annotation ($M^a_c$). If the difference $\Delta F > 0$, the candidate is deemed more faithful. A conservative arbitration rule replaces the human label with the candidate mask only if the margin $\Delta F$ exceeds a threshold (default $\delta=0$), otherwise retaining the human label to avoid over-correction.

Key Contributions

• CMF Metric: A reference-free, training-free metric that transforms mask disagreement into an image-conditioned keep-erase contrast, utilizing a frozen external vision-language judge to assess semantic mask fidelity.
• Large-Scale Audit: An audit of 10,731 image-class pairs across ten remote sensing benchmarks (including LoveDA, Potsdam, Vaihingen, etc.). The study reveals systematic, class-dependent annotation distortions: man-made classes (buildings, roads, cars) favor model candidates in 62–85% of cases, while ambiguous land cover often favors human annotations.
• Expert Validation: On a blinded three-annotator consensus set, CMF matches expert judgment on 81% of pairs, outperforming keep-only scoring, model confidence, and a trained label-quality baseline.
• Actionable Supervision: Training a lightweight segmenter on labels arbitrated by CMF yields improved cross-domain transfer performance compared to models trained on raw annotations or pure pseudo-labels, isolating the benefit of the arbitration process.

Results

• Seg-Probe Performance: Seg-Probe outperforms prior training-free baselines on 9 of 10 datasets, achieving an average mIoU of 56.3% (vs. 55.4% for SegEarth-OV3), validating its role as a strong candidate generator.
• Corruption Sensitivity: CMF was calibrated on controlled mask corruptions and shown to respond correctly to known errors while maintaining stability against mask area variations.
• Cross-Domain Transfer: Models trained on CMF-arbitrated labels demonstrated superior transferability to unseen target domains compared to those trained on raw annotations or matched replacement controls, confirming that the gains stem from the selection of higher-fidelity labels rather than simple pseudo-label distillation.

Significance
The paper positions CMF not as a method to replace human annotation entirely, but as a scalable tool for auditing ground truth. By treating the reference mask as a hypothesis rather than an infallible truth, CMF provides a mechanism to identify and correct systematic annotation errors, particularly for well-delineated man-made objects. This approach addresses the evaluation paradox by shifting the focus from agreement with a label to faithfulness to the image evidence, offering a pathway to more reliable supervision in remote sensing semantic segmentation.