Multi-modal, multi-scale representation learning for satellite imagery analysis just needs a good ALiBi

1. Problem Statement

The volume of satellite imagery generated by governmental and commercial constellations is outpacing human analysis capabilities. While representation learning offers a solution for automatic feature extraction, current models face significant limitations:

• Single-Dimension Focus: Existing foundation models typically learn stable representations across only one dimension: Ground Sample Distance (GSD/scale), image modality (e.g., SAR vs. Optical), or temporal captures. Few models successfully integrate both multi-scale and multi-modal learning simultaneously.
• Data Scarcity: Labeled data is expensive and scarce, necessitating self-supervised approaches.
• Alignment Challenges: Aligning images from different sensors (e.g., Sentinel-1 SAR, Sentinel-2 Optical, NAIP) and different resolutions (low-res vs. high-res) that cover the same physical area is complex for standard Transformer architectures, which often struggle to extrapolate to varying sequence lengths and resolutions.

2. Methodology

The authors propose Scale-ALiBi, a novel architecture that extends the 2D-ALiBi/X-ALiBi attention mechanism to handle multi-modal and multi-scale data natively.

A. Scale-ALiBi Attention Mechanism

The core innovation is a modification to the Transformer attention matrix to incorporate GSD information directly into the attention bias.

• Standard ALiBi: Uses a linear bias based on the distance between tokens to allow extrapolation to longer sequences.
• Scale-ALiBi Extension: The bias term $g(i, j)$ is redefined as:
  $$g(i, j) = \text{distance}(i, j) \cdot \text{GSD}$$
  Where $\text{distance}(i, j)$ is the Euclidean distance between patches $i$ and $j$, and GSD is the Ground Sample Distance of the source image.
• Functionality: This allows the model to compare tokens from images of different resolutions (e.g., a 256x256 Sentinel-2 patch vs. a 512x512 NAIP patch) by scaling the attention penalty based on the physical resolution of the source. This enables the model to handle variable token counts representing the same physical area.

B. Model Architecture (Triple-Contrastive & Reconstructive)

The model utilizes a Triple-Contrastive and Masked Autoencoder (MAE) architecture (see Figure 1 in the paper):

1. Encoders: Three Vision Transformer (ViT) encoders are trained:
   • $E_{radar}$: Low-resolution SAR (Sentinel-1).
   • $E_{lores}$: Low-resolution Optical (Sentinel-2).
   • $E_{hires}$: High-resolution Optical (NAIP, 2x resolution, producing 4x the tokens).
2. Contrastive Learning: An extended InfoNCE loss aligns representations from all three modalities ($M$) for the same physical location. The loss function "squeezes" representations of the same scene together and "pushes" them away from other scenes.
3. Cross-Encoding:
   • First, $E_{lores}$ and $E_{radar}$ tokens are cross-encoded to create a joint radar-optical representation.
   • Second, this joint representation is cross-encoded with the $E_{hires}$ token stream.
4. Reconstruction (MAE): A decoder reconstructs all sensor modalities into a single fused patch. The reconstruction loss ($L_{Recon}$) is modified to include the high-resolution token stream, effectively fusing the input sensors.
5. Total Loss: The objective is the sum of Contrastive Loss ($L_{Con}$) and Reconstruction Loss ($L_{Recon}$).

C. Dataset Curation

Since no existing public dataset contained aligned low-res SAR, low-res optical, and high-res optical pairs, the authors curated a new dataset:

• Sources: ESA Sentinel-1 (SAR), Sentinel-2 (Optical), and USDA NAIP (High-res Optical).
• Processing: Images are segmented into 256x256 tiles. Sentinel-1 VV/VH bands are mapped to RGB channels. NAIP data is sourced for the same coordinates at both standard (Y) and double (Y+1) resolution levels.
• Scope: Limited to the Continental US and Puerto Rico due to NAIP coverage.
• Release: The dataset is released in three sizes (small, full, micro) to facilitate research.

3. Key Contributions

1. Scale-ALiBi Mechanism: The first extension of the ALiBi attention mechanism to explicitly incorporate GSD scaling, enabling Transformers to natively process multi-resolution and multi-modal inputs without rigid resizing.
2. Novel Architecture: A triple-contrastive, reconstructive architecture that aligns SAR, low-res optical, and high-res optical data simultaneously.
3. New Dataset: The release of a curated, aligned dataset of Sentinel-1, Sentinel-2, and NAIP imagery, filling a gap in available resources for multi-scale remote sensing research.
4. Benchmarking: Evaluation on the GEO-Bench benchmark, demonstrating the feasibility of this approach.

4. Results

The model was evaluated on the GEO-Bench benchmark (6 classification and 6 segmentation tasks) and compared against CROMA (a state-of-the-art contrastive radar-optical model).

• Performance: Scale-ALiBi performed similarly or better than CROMA across most tasks, despite being a preliminary model.
  • Example (Neural Network Classifier): On the m-pv4ger task, Scale-ALiBi (High-res) achieved 87.77%, outperforming CROMA's 88.53% (Note: CROMA was slightly higher here, but Scale-ALiBi Low-res was 83.27%). On m-forestnet, Scale-ALiBi (Low-res) achieved 27.83% vs CROMA's 28.27%.
  • Example (k-NN): On m-pv4ger, Scale-ALiBi (High-res) achieved 91.89%, nearly matching CROMA's 92.29%.
• Observation: While preliminary results were slightly below CROMA's published state-of-the-art in some specific metrics, the authors attribute this to the limited training run size. They express optimism that larger training runs will yield equivalent or superior results.
• Ablation: The inclusion of the high-resolution encoder provided marginal but consistent improvements over using only low-resolution encoders.

5. Significance and Future Work

• Significance: This work bridges the gap between multi-modal and multi-scale learning in remote sensing. By allowing models to "extrapolate" across resolutions using the GSD bias, it removes the need for complex image resampling or separate models for different sensor resolutions. The release of the aligned dataset is a critical resource for the community.
• Future Work:
  • Curating and releasing a significantly larger dataset to cover more geographic diversity.
  • Conducting longer, more extensive training runs to fully realize the potential of the Scale-ALiBi mechanism.
  • Further refining the architecture to close the performance gap with established baselines like CROMA.

In conclusion, the paper argues that the "alibi" (attention bias) is the key to unlocking robust, unified representation learning for the diverse and multi-scale nature of modern satellite imagery.