Clustering Guided Domain-Specific Pretrained Foundation Model Very High-Resolution Arctic Remote Sensing

Technical Summary: Clustering Guided Domain-Specific Pretrained Foundation Model for Very High-Resolution Arctic Remote Sensing

Problem Statement
Despite the unprecedented availability of Earth observation (EO) data, vast archives remain underutilized for scientific synthesis. A central challenge in Earth informatics is transforming large, heterogeneous remote sensing datasets into science-ready products. While recent advances in self-supervised learning and foundation models (e.g., Masked Autoencoders or MAEs) have shown promise in computer vision, their direct application to remote sensing is non-trivial. Remote sensing imagery differs fundamentally from everyday images; "geo-objects" are emergent properties of scale, spatial context, and temporal processes rather than discrete visual instances. Furthermore, existing Remote Sensing Foundation Models (RSFMs), such as Prithvi-EO-2.0, are primarily trained on medium-to-coarse resolution data or mixed-resolution datasets. This raises the question of whether a single generalized model can adequately represent diverse Earth system processes, or if domain adaptation is necessary to capture environment-specific patterns, particularly in the Arctic where rapid climate-driven changes and fine-scale permafrost features (e.g., ice-wedge polygons, retrogressive thaw slumps) require very high spatial resolution (VHSR) analysis.

Methodology
The study proposes a novel framework for developing an Arctic-focused RSFM by combining diversity-aware regional-scale image curation with MAE self-supervised pretraining of a Vision Transformer (ViT) encoder. The methodology consists of five key components:

1. Data Curation and Sampling:

   • Source: The authors utilized 267 TB of Vantor VHSR multispectral satellite imagery (approx. 0.5 m resolution) spanning over 5 million km² of the Arctic permafrost tundra ecoregion.
   • Strategy: To avoid oversampling visually repetitive or low-information areas, a two-stage clustering workflow was employed.
     • Stage 1 (Scene-level): Approximately 32,000 candidate scenes were characterized using spectral descriptors (per-band mean/SD) and acquisition metadata. Affinity-propagation clustering was used to select representative scene exemplars without pre-specifying the number of clusters.
     • Stage 2 (Chip-level): Selected scenes were partitioned into non-overlapping 1024x1024 pixel chips. These chips were further characterized by spectral statistics and Gabor-based texture features. A second affinity-propagation clustering step selected representative chip exemplars within each scene.
   • Outcome: This process curated approximately 3 million representative training patches, ensuring broad scene diversity while reducing redundancy.

2. Pretraining Architecture:

   • Model: A ViT-Large encoder was pretrained using a domain-adapted MAE objective.
   • Loss Function: The reconstruction objective combined a covariance-aware Mahalanobis term and a Spectral Angle Mapper (SAM) term to capture both multivariate spectral deviation and spectral-shape consistency, better reflecting remote sensing spectral behavior than standard pixel-wise loss.
   • Training: The model was trained on 16 compute nodes (64 GPUs total) for 800 epochs. Checkpoints were evaluated on downstream validation tasks to select the final weights, rather than relying solely on reconstruction loss.

3. Downstream Evaluation:

   • The pretrained encoder was integrated into the ViTDetLoc framework (a transformer-based detection and segmentation architecture augmented with location embeddings).
   • Datasets: The model was fine-tuned and evaluated on four hand-labeled Arctic feature-mapping datasets:
     • Human-built infrastructure (buildings, roads, tanks).
     • Ice-wedge polygons (IWP).
     • Retrogressive thaw slumps (RTS).
     • Tundra capillary networks (TCNs).
   • Baselines: Performance was compared against an ImageNet-initialized ViT-Large baseline and the general-purpose Earth foundation model, Prithvi-EO-2.0.

Key Contributions

• Domain-Specific Curation: The paper introduces a scalable, diversity-aware sampling strategy that selects training data based on actual image characteristics (spectral, texture, metadata) rather than coarse land-cover labels, addressing the fine-scale variability inherent in VHSR Arctic imagery.
• Arctic-Specific Foundation Model: The development of an RSFM pretrained specifically on curated Arctic VHSR imagery, demonstrating that domain-adapted self-supervised pretraining yields more transferable representations for fine-scale Arctic mapping than general-purpose models.
• Validation on Sparse Features: The study validates the model on challenging, sparse, and visually subtle Arctic features (e.g., capillary networks and thaw slumps) where standard models often struggle.

Results

• Curation Analysis: While random sampling produced subsets with lower Wasserstein distances to the full archive (indicating closer marginal distribution matching), the clustering-based approach resulted in greater dispersion in descriptor space. This confirmed that the curation strategy successfully prioritized diversity and reduced redundancy, exposing the model to a broader range of visual and acquisition conditions.
• Downstream Performance:
  • The Arctic MAE-pretrained model consistently outperformed the ImageNet-initialized baseline across all four tasks, achieving foreground mean F1 score improvements of approximately 5–8 percentage points.
  • Specific foreground mean F1 scores for the Arctic MAE were: 0.87 (Infrastructure), 0.72 (IWP), 0.93 (RTS), and 0.87 (TCNs).
  • Crucially, the proposed model outperformed Prithvi-EO-2.0 in all downstream comparisons, achieving at least a 15% improvement. The authors attribute this to the resolution mismatch (Prithvi uses 30m data vs. 0.5m Vantor) and the domain shift, suggesting that cross-resolution transfer from general models is insufficient for specialized, fine-scale Arctic tasks.

Significance and Claims
The paper claims that optimizing the pretraining data distribution at a regional scale, while keeping the architecture and MAE objective fixed, can produce a reusable, domain-specific encoder for multiple VHSR remote sensing applications. The authors argue that for tasks involving fine-scale texture, local morphology, and specific surface conditions (such as Arctic permafrost dynamics), domain-specific self-supervised pretraining on curated imagery provides more transferable representations than relying on general-purpose Earth observation foundation models. This work demonstrates a scalable path to transforming large, unlabeled Arctic image archives into foundation models that support downstream detection and segmentation tasks even with limited labeled data.