Technical Summary: Multi-Modal Contrastive Learning for Implicit Earth Embeddings via Location Tying

Problem Statement

Spatial prediction tasks, ranging from ecological modeling to population dynamics, are fundamentally constrained by the scarcity of high-quality, labeled ground-truth observations. While unlabelled geospatial data exists at scale across diverse modalities (e.g., satellite imagery, geotagged photographs, encyclopedic text), supervised models cannot leverage this data without labels.

Existing solutions, known as "Earth embedding models," aim to learn transferable vector representations of geospatial data. These can be explicit (extracted from raw data like imagery) or implicit (mapping geographic coordinates to an embedding via a location encoder). Implicit encoders are particularly valuable as they store location information within model weights, allowing inference on arbitrary downstream tasks without modality-specific inputs.

Currently, the dominant paradigm for pre-training these location encoders is Contrastive Learning (CL). However, existing methods typically align geographic coordinates with only one additional modality (e.g., coordinates + satellite imagery). This two-modality approach limits the entropy extractable from geographic data, as different modalities provide distinct semantic and visual contexts. While the theoretical potential of utilizing multiple unpaired modalities simultaneously has been noted, no prior work has empirically studied this in the context of implicit earth embeddings.

Methodology

The paper proposes two multimodal contrastive learning architectures designed to train a location encoder using unpaired geospatial data, utilizing geographic location as a shared anchor across multiple modalities.

Core Setup

• Modalities: A set $M = \{m_0, m_1, \dots, m_{K-1}\}$ where $m_0$ is the location modality (geographic coordinates) and $m_{1 \dots K-1}$ are non-location modalities (e.g., satellite, text, natural images).
• Unpaired Data: The datasets are unpaired; there is no one-to-one correspondence between non-location modalities. Instead, spatial co-location serves as the proxy for semantic relatedness.
• Positive Pairs: For a sample from a non-location modality $m$, the only positive counterpart is its co-located location projection.
• Encoder: The location encoder uses spherical harmonic basis functions (up to the 10th degree) combined with a SIREN MLP. Non-location modalities use frozen pre-trained backbones (e.g., MoCo-ResNet, DINO-ViT, BGE) with trainable linear adapters.

Proposed Architectures

1. MELT (Multimodal Embedding via Location Tying):

   • Mechanism: A batch-construction approach where all modalities are trained jointly in every step.
   • Process: In each training step, $N$ samples are drawn from each non-location modality. Each is paired with its co-located location embedding.
   • Objective: A symmetric NT-Xent loss is aggregated over all non-location modalities. All encoder parameters are updated jointly, creating a single shared embedding space without alternating updates.

2. SALT (Sequential Alternating Location Training):

   • Mechanism: An epoch-level schedule that alternates the active non-location modality while keeping the location encoder permanently active.
   • Process: At each epoch $e$, the location encoder is paired with a single non-location modality $m_{(e \mod (K-1))+1}$.
   • Objective: Only the parameters of the currently active modality and the location encoder are updated. Modalities are never directly compared; they are linked only via the shared training signal with the location encoder.

Key Contributions

1. Novel Architectures: Introduction of MELT and SALT, the first frameworks to extend implicit earth embedding training beyond two modalities using unpaired data and location as a tying element.
2. Empirical Investigation: A comprehensive study assessing whether increasing modality diversity (from 2 to 4 modalities) or scaling pre-training data volume improves downstream performance.
3. Performance Ceiling Finding: The identification that the location encoder itself, rather than modality diversity or pre-training volume, constitutes the primary performance bottleneck.

Experimental Results

Setup

• Data: Pre-training on ~100K globally sampled locations using three non-location modalities: Sentinel-2 satellite imagery, geotagged natural images (YFCC100M), and geotagged Wikipedia text.
• Downstream Tasks: Four global tasks: Elevation (Regression), Population Density (Regression), Country Codes (Classification), and Biomes (Classification).
• Baselines: Compared against strong existing location encoders (SATCLIP, GEOCLIP, CSP, RANGE) and non-learned baselines (MOSAIKS, IDENTITY).

Quantitative Findings

• Performance Parity: Both MELT and SALT match the performance of the strongest two-modality baseline (SATCLIP-ORG) across all four downstream tasks.
• No Consistent Gain from Modalities: Increasing the number of modalities (e.g., from 2 to 3 or 4) did not consistently improve performance. Statistical analysis (paired t-tests) showed that performance differences between multi-modal variants were largely attributable to random initialization rather than model design.
• No Gain from Data Volume: Scaling pre-training data volume (from 10k to 261k pairs) did not yield reliable improvements.
• Conclusion: The location encoder reaches its representational capacity early. Once it learns a stable coordinate-to-region mapping, additional modalities provide no exploitable gradient signal.

Qualitative Findings

• Training Stability: MELT demonstrated stable training with smooth loss curves. SALT exhibited characteristic loss spikes at epoch boundaries when the active modality switched, occasionally leading to early termination of training runs.
• Embedding Quality: Despite SALT's instability, both architectures produced spatially coherent predictions (e.g., in elevation maps) that were visually indistinguishable from each other but superior to baselines in terms of spatial continuity.
• Embedding Visualization: Independent Component Analysis (ICA) revealed that self-trained encoders produced geographically consistent color gradients (e.g., similar colors for tropical forests), whereas baselines showed grid artifacts or high variance.

Significance and Claims

The paper claims that the current paradigm of simply adding more modalities to contrastive learning for implicit earth embeddings has diminishing returns due to architectural limitations.

• The Bottleneck: The primary limitation is the location encoder architecture and the contrastive objective (NT-Xent). The objective forces all modalities to align to a single shared anchor, effectively collapsing the task into a low-complexity alignment that the current encoder can solve quickly. This process discards modality-specific features that might distinguish, for example, spectral signatures from textual context.
• Implications for Future Work:
  • Simply adding modalities or data volume is insufficient.
  • Future efforts should focus on alternative encoder architectures with higher representational capacity.
  • Alternative alignment objectives (beyond global instance-level similarity) are necessary to preserve modality-specific structures and unlock complementary information.
• Practical Recommendation: Between the two proposed methods, MELT is identified as the superior foundation for future scaling due to its training stability and lack of representational discontinuities, despite SALT achieving comparable final metrics.

The authors emphasize that while their methods are technically viable and match current state-of-the-art baselines, they do not surpass them, highlighting the need for fundamental changes in encoder design and loss functions to truly leverage multimodal geospatial data.