How do Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks?

Technical Summary: How Self-Supervised Remote Sensing Vision Models Transfer to Downstream Tasks

Problem Statement
Self-supervised geospatial foundation models (GeoFMs) have emerged to learn transferable representations from large-scale unlabeled satellite imagery. However, their downstream behavior remains difficult to characterize. Unlike natural image understanding, remote sensing (RS) data presents unique challenges: it lacks discrete object boundaries, relies heavily on spectral, temporal, and geographic context, and frequently requires dense pixel-level prediction rather than scene-level classification. Recent benchmarks indicate that GeoFMs do not consistently outperform supervised baselines, and no single model design dominates across diverse downstream tasks. Furthermore, it is unclear how different self-supervised learning (SSL) objectives preserve specific types of information (e.g., low-level spectral vs. high-level semantic) and how this information is organized across network depth. Consequently, standard downstream adaptation strategies, such as heavy multi-scale decoders, may be misaligned with how GeoFMs structure their internal representations.

Methodology
The authors conduct a comprehensive evaluation of six representative GeoFMs spanning three pretraining families: joint-embedding (MoCo, DINO v1), reconstruction (MAE, Prithvi v1), and multimodal/hybrid (CROMA, TerraMind). These models vary in architecture scale (ViT-S vs. ViT-B), parameter count (42M to 2B), and pretraining data (SSL4EO, NASA HLS, TerraMesh).

The study employs three primary analytical approaches:

1. Downstream Transfer Evaluation: The models are evaluated across diverse tasks including image-level classification (EuroSAT), regression (NeuCo-Bench for low-level and semantic tasks), and pixel-level segmentation (PANGAEA-Bench, PASTIS, Sen1Floods11). Evaluations consider varying label availability (1-shot, 10%, 100%) and adaptation pipelines (frozen encoder vs. fine-tuning).
2. Depthwise Representation Analysis: Using layerwise linear probes, the authors assess which transformer blocks contain task-relevant information. This distinguishes between low-level tasks (e.g., biomass estimation) and high-level semantic tasks.
3. Adaptation and Fine-Tuning Analysis: The study compares different decoder architectures (heavy multi-scale UPerNet, lightweight multi-scale, and single-level upsamplers) and quantifies representation shifts during fine-tuning using Centered Kernel Alignment (CKA).

Key Contributions

• Sensitivity of Model Rankings: The authors demonstrate that GeoFM rankings are highly sensitive to task type, label availability, and adaptation settings. A model performing well on one task (e.g., semantic regression) may underperform on another (e.g., dense segmentation), even when pretrained on the same dataset.
• Depthwise Information Organization: Through layerwise probing, the study reveals that task-relevant information is often more accessible in intermediate transformer blocks than in final-layer embeddings. Reconstruction-based models (MAE, Prithvi) tend to retain low-level information deeper in the network, while joint-embedding models (MoCo, DINO) show a sharper transition from low-level to semantic phases. TerraMind exhibits progressive improvement across all depths.
• Impact of Adaptation Strategies: Downstream adaptation choices, particularly decoder design and fine-tuning, significantly influence performance. The authors find that lighter, single-level decoders often match or outperform heavy multi-scale decoders (like UPerNet), suggesting standard dense-prediction heads may not align with GeoFM information organization.
• Fine-Tuning Dynamics: CKA analysis shows that fine-tuning does not rewrite GeoFMs uniformly. The strongest representation changes are often localized to specific sub-layers, particularly the first linear layer of the MLP within ViT blocks, rather than being distributed evenly across the network.

Results

• Task Dependency: On regression and classification tasks, model rankings shift based on label availability and distance metrics. For instance, joint-embedding models excel at high-level semantic tasks (EuroSAT) but struggle with low-level regression, whereas reconstruction models show the inverse trend.
• Decoder Efficiency: In segmentation case studies (PASTIS and Sen1Floods11), lightweight decoders with fewer parameters frequently outperform the standard UPerNet. For example, a single-level upsampler using intermediate layers (Block 5 or 7) often yields better results than using final-layer features with a heavy decoder.
• Fine-Tuning Effects: Fine-tuning provides substantial gains for complex, multi-class tasks (PASTIS) but offers minimal or negative returns for simpler binary tasks (Sen1Floods11). Notably, a supervised ImageNet baseline (ViT-B/16 adapted to 13 channels) becomes competitive with GeoFMs when fine-tuned, even in low-label regimes.
• Scaling vs. Robustness: While TerraMind (trained on the largest dataset with 2B parameters) demonstrates robustness across most tasks, smaller general-purpose models and the ImageNet baseline remain competitive on specific datasets, indicating that scaling alone does not guarantee superior transfer performance for all geospatial tasks.

Significance and Claims
The paper argues that the transfer behavior of GeoFMs is not intrinsic to the model alone but is a function of the interplay between the pretraining objective, the downstream task structure, and the adaptation strategy. The authors claim that current evaluation pipelines often fail to account for how information is organized across network depth, leading to suboptimal model selection and adaptation.

The study motivates a shift away from "one-size-fits-all" adaptation strategies. Specifically, it suggests that:

1. Representation-Aware Evaluation: Future benchmarks should consider intermediate layer representations rather than relying solely on final embeddings.
2. Tailored Adaptation: Decoder designs should be aligned with the depthwise profiles of specific GeoFMs, potentially favoring lighter, single-level decoders or adaptive layer selection over heavy multi-scale fusion.
3. Targeted Fine-Tuning: Since fine-tuning induces localized changes (e.g., in MLP layers), future work could explore parameter-efficient fine-tuning methods that target these specific sub-modules rather than full network updates.

Ultimately, the authors conclude that understanding the internal mechanisms of GeoFMs is crucial for developing more accurate and efficient Earth observation systems, as current large-scale models do not universally outperform smaller or supervised baselines without carefully designed downstream transfer schemes.