Technical Summary: The View From Space: Navigating Instrumentation Differences with EOFMs

Problem Statement

Earth Observation Foundation Models (EOFMs) have become prevalent tools for processing massive volumes of remotely sensed data, often utilizing pre-trained outputs as embeddings for tasks like similarity search and content-specific queries. However, a critical gap exists in understanding how diverse sensor architectures impact the internal representations of these models.

While traditional computer vision operates in a relatively uniform space (consistent RGB bands and standard quantization), earth observation data is highly non-uniform. Sensors possess diverse architectures, observation conditions, and complex processing stacks (radiometric/geometric calibration, atmospheric correction, cloud masking). Existing work often trains models on single modalities and benchmarks them by simply matching spectral bands across different sensors (e.g., assuming a model trained on Landsat-8 should generalize seamlessly to Harmonized Landsat and Sentinel (HLS) or Sentinel-2). It remains unclear whether minor differences in instrument readings and sensor architectures result in significant variances in the embedding spaces of EOFMs, potentially undermining their utility in downstream tasks like few-shot learning or similarity search.

Methodology

To evaluate the sensitivity of EOFM representation spaces to sensor architecture, the authors constructed a novel dataset and employed an unsupervised analysis approach with frozen model weights.

Dataset Construction:

• Location: 600 randomly sampled points from Indiana, USA.
• Resolution: 30-meter spatial resolution, 224x224 pixel 2-month mosaic composites.
• Modalities: Four optical datasets (HLS, Landsat-8, Landsat-9, Sentinel-2) and one SAR dataset (Sentinel-1, VV/VH polarizations).
• Preprocessing: Optical data was masked for >20% cloud occlusion using QA pixels. The USDA Crop Data Layer was aligned as a semantic reference.

Models Evaluated:

• Prithvi: A model pretrained solely on HLS data with a fixed six-band input. It represents the archetype of models that require matching spectral bands by wavelength and discarding extras.
• DOFA: A model utilizing a dynamic network embedding layer that generates convolution weights based on the central wavelengths of inputs. It was trained on diverse modalities (aerial RGB to satellite) and can accept any band configuration, characterizing inputs solely by central wavelength.

Evaluation Techniques:

1. Embedding Space Visualization: Used t-SNE to reduce the 768-dimensional latent space to 2D, visualizing clusters based on source modality. The goal was to determine if data clusters by sensor type rather than semantic features (e.g., land cover).
2. Local Neighborhood Analysis:
   • CLS Tokens: Measured the overlap of nearest neighbors (5, 10, 20) between modalities to assess global image similarity.
   • Patch Embeddings: Compared 10-Nearest Neighbors of image patches using cosine similarity. Semantic similarity was proxied by the percentage of neighbors sharing the same USDA Crop Data Layer class.
   • Modality Classification: Trained simple classifiers (Random Forest and 5-NN) to predict the input modality of a patch embedding, testing how easily the embedding space can be partitioned by sensor source.

Key Results

The study demonstrates that EOFM representation spaces are highly sensitive to sensor architecture, leading to significant distributional shifts even when spectral bands are nominally matched.

• Visual Clustering: t-SNE plots (Figures 1 & 2) revealed clear, distinct clusters based on the source modality for both Prithvi and DOFA. This indicates that in the high-dimensional space, data is distributed principally by sensor source rather than semantic content. While DOFA showed slightly greater resilience to modality changes compared to Prithvi, distinct separations remained in most pairwise cases.
• Low Neighborhood Overlap: In local neighborhood analysis of CLS tokens, the average percentage of matching neighbors between different modalities was rarely above 30%. For instance, between Landsat-8 and Sentinel-2, only ~18-20% of neighbors matched. This suggests that in similarity search contexts, the input modality becomes the primary determinant of "similar" candidates, potentially overriding semantic similarity.
• Semantic Drift: When analyzing patch embeddings, the percentage of neighbors sharing the same crop class varied by up to 4% depending on the modality. This confirms that modality changes alter not just the neighborhood structure but also the semantic similarity of local image patches.
• High Modality Predictability: Simple classifiers could predict the source modality of a patch embedding with high accuracy (90.7% for Prithvi and 88.7% for DOFA) using a Random Forest. This implies that the embedding spaces are easily partitioned by sensor source, making it difficult to achieve modality-invariant representations.

Significance and Claims

The paper claims that the current design and usage of EOFMs face significant pitfalls due to a lack of consideration for sensor architecture diversity.

• Limitations of Current Practices: The authors argue that simply matching spectral bands by wavelength and training on single modalities is insufficient. These approaches fail to account for the subtler underlying distribution shifts caused by different sensor technologies and processing stacks.
• Implications for Users: Users of EOFMs must carefully align their data collection regimes with the pretraining data of the chosen model. Relying on embeddings from a model trained on one sensor for data from another without rigorous validation risks significant performance degradation in downstream tasks.
• Future Directions: The findings motivate the development of models that explicitly consider diverse sensor modalities and the complexities of spectral capture technology. The authors advocate for the creation of highly multimodal, multi-input stream datasets (like the one constructed in this study) to rigorously test the cross-modality capabilities of new models.

The paper concludes that understanding these instrumentation differences is vital for the community to move forward with robust remote-sensing science, ensuring that EOFMs deliver reliable impact across the diverse landscape of earth observation data.