Towards Knowledge Guided Pretraining Approaches for Multimodal Foundation Models: Applications in Remote Sensing

1. Problem Statement

Existing pretraining paradigms for geoscience foundation models primarily rely on Masked Reconstruction (MR) or Next-Token/Variable-Step Forecasting (VSF).

• Limitations of MR: While effective for spatial feature extraction, MR focuses on reconstructing masked parts of an image (or sequence) without explicitly modeling the causal relationships between different environmental variables. It captures spatial structure but often fails to capture temporal dynamics driven by external factors.
• Limitations of Standard VSF: Standard forecasting models predict future states based on past observations of the same modality (e.g., predicting future satellite images from past satellite images). They often ignore the driver-response relationship where external variables (drivers, e.g., weather) causally influence the state of the land surface (response, e.g., vegetation growth).
• The Gap: Current approaches do not fully leverage the known causal interplay between geospatial variables (e.g., how rainfall and temperature drive vegetation phenology). Consequently, embeddings learned via these methods may lack the physical consistency required for tasks where understanding why a change occurred is as important as what changed.

2. Methodology: Knowledge Guided Variable-Step Forecasting (KG-VSF)

The authors propose KG-VSF, a novel pretraining task that frames forecasting as a conditional generation task. Instead of predicting the future based solely on past imagery, the model predicts future satellite imagery conditioned on past satellite imagery and future weather data (up to the forecast date).

Core Concept

• Driver Variables: Weather data (e.g., temperature, precipitation, wind) acts as the "driver."
• Response Variables: Satellite imagery (spectral bands) acts as the "response."
• Objective: Given a history of satellite images and weather, and weather data up to a future time $T + \delta t$, predict the satellite image at $T + \delta t$. This forces the model to learn how specific weather patterns causally alter land surface conditions.

Architecture

The model follows a Heavy Encoder / Lightweight Decoder design:

1. Satellite Encoder: A shared Vision Transformer (ViT) extracts spatial features from spectral imagery patches across timestamps.
2. Weather Encoder: A Unidirectional LSTM encodes weather time-series data (due to its coarse spatial resolution and sequential nature).
3. Temporal Encoders: Separate encoders for Day-of-Year (DOY) and time-delta ($\Delta t$) embeddings to provide temporal context.
4. Multimodal Sequence Encoder: A Transformer with forward-only (causal) attention fuses the satellite, weather, and temporal embeddings. This ensures no information leakage from the future.
5. Forecaster: A lightweight MLP layer that translates the current timestamp embeddings into future timestamp embeddings using the weather context.
6. Decoder: A lightweight MLP decoder reconstructs the satellite image from the forecasted embeddings.

Two-Phase Pretraining Strategy

To ensure the embeddings capture causal relationships rather than just correlations, the authors employ a two-phase training process:

• Phase 1 (Masked Reconstruction): The model is pretrained to reconstruct masked satellite images and weather data independently. This establishes robust baseline representations for each modality.
• Phase 2 (Forecasting): A forecaster is introduced. The model is trained to predict future satellite images using the current embeddings and future weather data. This phase forces the encoder to align the weather "driver" signals with the satellite "response" signals to minimize forecasting error.

3. Key Contributions

1. Novel Pretraining Objective: Introduction of KG-VSF, which explicitly models causal dependencies between modalities (weather $\to$ land surface) rather than just semantic correlations or self-supervised reconstruction.
2. Causal Embeddings: Demonstration that KG-VSF produces embeddings that inherently encode physical laws (e.g., rain leads to greening, snow leads to white cover) without explicit supervision.
3. Comprehensive Evaluation: Extensive benchmarking against standard baselines (SM-MR, MM-MR, SM-VSF, MM-VSF) and state-of-the-art models (Prithvi EO.2, WSTATT) across diverse downstream tasks.
4. Ablation & Verification: Rigorous experiments (Counterfactual Weather Probing, Mute/Shuffle tests) proving that performance gains stem from learning causal mechanisms, not merely the presence of an additional input modality.

4. Experimental Results

The authors evaluated the method on two datasets: a Global Dataset (diverse regions) and a US Cropland Dataset (agricultural focus).

Downstream Task Performance

• Pixel-Wise Crop Type Mapping: KG-VSF achieved the highest Mean F1 Score (0.7602), significantly outperforming MM-MR (0.7056) and MM-VSF (0.5426). It also showed superior performance with fewer training samples compared to models trained from scratch.
• Soil Moisture Estimation: KG-VSF achieved an $R^2$ of 0.8081, outperforming MM-VSF (0.7606) and Prithvi EO.2 (0.3377). This highlights the critical role of weather data in soil moisture prediction.
• Soil Moisture Forecasting: KG-VSF maintained superior accuracy across all forecast horizons (0–25 days to 100+ days), with the performance gap widening as the forecast horizon increased.
• Missing Image Prediction: KG-VSF demonstrated the lowest Mean Squared Error (MSE) in filling missing data (e.g., cloud cover), correctly inferring physical states (e.g., drying riverbeds) where other models hallucinated incorrect features (e.g., adding water to dry beds).
• Zero-Shot Future Image Forecasting: In a zero-shot setting (no fine-tuning), KG-VSF significantly outperformed VSF baselines in predicting future land cover changes (e.g., harvest, snowfall) over long horizons (up to 255 days), capturing terrain-specific dynamics that correlation-based models missed.

Embedding Quality Analysis

• t-SNE Visualization: KG-VSF embeddings formed tight, semantically consistent clusters for different crop types, whereas MR-based models produced diffuse, overlapping clusters.
• Counterfactual Probing: When fed counterfactual weather (e.g., "dry hot" vs. "cold wet"), KG-VSF's forecasted images changed physically correctly (e.g., adding snow only to open fields, not forests), proving it learned causal rules.
• Ablation (Mute/Shuffle): Muting or shuffling weather data caused a drastic drop in performance (e.g., Soil Moisture $R^2$ dropped from 0.8081 to 0.3143 when weather was zeroed), confirming the model relies on the causal signal, not just the presence of a second modality.

5. Significance and Impact

• Physical Consistency: KG-VSF moves foundation models from learning statistical correlations to learning physically consistent causal dynamics. This is crucial for geoscience applications where understanding the "why" (mechanism) is as important as the "what" (prediction).
• Data Efficiency: The method achieves high performance with limited fine-tuning data, making it suitable for domains where labeled data is scarce.
• Generalizability: While demonstrated in remote sensing, the paradigm of "Knowledge-Guided Pretraining" is applicable to other domains involving driver-response relationships, such as healthcare (clinical factors $\to$ patient outcomes) or finance (macro indicators $\to$ market dynamics).
• Scalability: Experiments show that performance scales linearly with the amount of pretraining data, suggesting that KG-VSF can benefit from the massive amounts of unlabeled satellite and weather data available globally.

In conclusion, the paper establishes that incorporating domain knowledge (causal drivers) directly into the pretraining objective yields foundation models that are more robust, interpretable, and effective for complex geoscientific tasks than traditional self-supervised approaches.