Benchmarking the Alignment of Data-Quality Metrics, Human Judgment and Land-Cover Segmentation Performance for Earth Observation

Technical Summary: Benchmarking the Alignment of Data-Quality Metrics, Human Judgment, and Land-Cover Segmentation Performance for Earth Observation

Problem Statement
The training of deep learning models for Earth observation (EO) is often constrained by the volume and quality of available datasets. While synthetic data augmentation via deep generative models (e.g., GANs, Diffusion Models) offers a solution, assessing the actual utility of this synthetic data remains a challenge. Standard evaluation practices rely on distribution-level distance metrics (e.g., Fréchet Inception Distance [FID], Kernel Inception Distance [KID], Inception Score [IS]) computed in the latent space of networks pre-trained on generic datasets like ImageNet.

The core problem identified is that these metrics, optimized for natural images with object-centric compositions, may not transfer effectively to the specialized domain of EO. EO imagery is characterized by top-down viewpoints, arbitrary orientations, and distributed semantic information. Consequently, standard metrics may fail to reflect downstream task utility, diverge from human perception under imperceptible perturbations, or be misled by structural memorization that suppresses high-frequency details crucial for segmentation tasks. There is a lack of systematic investigation into how automatic metrics, human perception, and downstream performance align in this specific domain.

Methodology
The authors conducted a comprehensive three-way alignment study using real and synthetic remote sensing datasets covering diverse geographic regions (Türkiye, Europe, South Korea, and California-Nevada). The study integrated three distinct evaluation pillars:

1. Perturbation Experiments: Seven quantitative metrics (FID, KID, IS, LPIPS, SSIM, PSNR) were evaluated against six specific image transformations: Gaussian noise, perspective warping, rotation, flipping, resized cropping, and a combined transformation. This tested the robustness of metrics against semantics-preserving and semantics-altering changes.
2. Human Perception Study: A four-stage questionnaire involving 88 participants assessed:
   • Augmentation Alignment: Ability to identify identical scenes under geometric transformations.
   • Downstream Utility: Ability to match Earth observation images with their corresponding land-cover segmentation masks (testing interpretability of real vs. synthetic pairs).
   • Conditional Generation Preference: Preference among outputs from different generative architectures (Stable Diffusion, U-Net GAN, StyleGAN3) given a conditioning mask.
   • Realism Assessment: Likert-scale rating (1–5) of image realism across real and synthetic datasets.
3. Downstream Task Benchmark: Semantic segmentation models (including UNet, DeepLabV3+, SegFormer, etc.) were trained on varying mixtures of real and synthetic data. Performance was measured via F1 scores to determine if synthetic data improved baselines despite potentially poor fidelity scores.

Key Contributions

• Vulnerability of Generative Metrics: The study demonstrates that widely used latent-space distribution metrics are highly sensitive to minor, semantics-preserving perturbations (e.g., rotation), which drastically alter scores without affecting human recognition.
• Human Perception Baseline: A four-stage study quantifying human recognition accuracy, visual preference, and cognitive load, providing a human reference to evaluate the validity of automated quality metrics.
• Three-Way Alignment Analysis: A quantitative analysis revealing significant discrepancies among human perception, automated metrics, and downstream model performance, exposing failure modes in standard automated evaluation.
• Land-Cover Segmentation Benchmark: Evidence that visual fidelity (as measured by FID) does not reliably predict downstream performance. Synthetic data with poor automatic scores can improve real-data baselines when combined, while high-fidelity metrics do not guarantee utility.

Results

• Metric Instability: Automatic metrics showed high volatility under perturbations. For instance, rotation increased the FID of the ARAS-Test dataset from 2.09 to 36.51, while human accuracy only dropped from 97.13% to 94.05%. Conversely, noise and combined perturbations caused the most significant degradation in human scene perception but were also the most detrimental to metric scores.
• Perception vs. Metrics: Synthetic datasets often received lower FID scores than real datasets but were perceived as equally or more realistic by humans. For example, the synthetic ARAS-SGAN3 dataset (FID ~16.85) was rated more realistic (µ=3.43) than real datasets like BELDE (µ=3.32) and BELDE-CA-NV (µ=3.09), despite the latter having significantly better FID scores in some comparisons.
• Decoupling of Utility: A stark misalignment was found between FID scores and downstream utility. Augmenting real data with CUGAN-generated synthetic data degraded the FID score from 2.09 to 21.27 but improved the downstream segmentation F1 score from 76.5% to 77.6%. Similarly, human preference for realism (e.g., BELDE-K vs. BELDE-CA-NV) did not correlate with the segmentation model's performance on those datasets.
• Model Preferences: In conditional generation tasks, Stable Diffusion models trained on ARAS400k were most preferred by humans (39%), yet they received the lowest standalone realism scores (2.51), indicating that fidelity to a conditioning mask and perceived realism are distinct properties.

Significance and Claims
The paper argues that automatic quality evaluation of synthetic datasets for Earth observation cannot rely solely on metrics rooted in ImageNet-pretrained feature spaces. The authors claim that:

1. Metrics are Unreliable Proxies: Standard metrics like FID are unreliable indicators for geospatial data because they are hyper-sensitive to low-level structural changes (like rotation) that are irrelevant to human observers and downstream tasks.
2. Utility is Decoupled from Fidelity: High visual fidelity (low FID) does not guarantee high downstream utility, and synthetic data with poor fidelity scores can still enhance model performance when mixed with real data.
3. Need for Holistic Evaluation: A reliable evaluation framework must jointly consider quantitative metrics, human perception, and downstream task performance. No single axis is sufficient, particularly given the domain shift between natural images and Earth observation imagery.

The authors conclude that optimizing generative models purely for the minimization of automatic metrics risks prioritizing "visual cleanliness" over genuine data utility, and future work should aim to develop metrics that are robust to perturbations, aligned with human perception, and predictive of downstream task performance.