Densification and forecasting of Sentinel-2 time series from multimodal SAR and Optical satellite data using deep generative models

This paper proposes a probabilistic deep learning framework that leverages multimodal Sentinel-1 SAR and Sentinel-2 optical data to generate optical satellite images at arbitrary past or future dates, effectively addressing both temporal densification of cloud-obscured observations and future forecasting while explicitly modeling generation uncertainty.

The Gist

Imagine you are trying to watch a movie about a farmer's field changing through the seasons, but the projector is broken. Sometimes the film skips, sometimes it's covered in static (clouds), and sometimes the reels are missing entirely. You have two types of film strips:

1. Optical Film (Sentinel-2): Beautiful, colorful pictures of the field, but they only work when the sky is clear. If it's cloudy, the picture is white and useless.
2. Radar Film (Sentinel-1): Black-and-white, grainy pictures that can "see" through clouds and rain, but they don't show the vibrant colors of the crops.

The Problem:
Scientists want a perfect, continuous, colorful movie of the Earth. But because of clouds, there are huge gaps in the optical film. Existing tools can try to "fill in the blanks" between two known pictures (like guessing what happened in the middle of a skipped scene), but they can't guess what happens after the movie ends, and they can't tell you how sure they are about their guesses.

The Solution:
The authors built a smart AI "Director" that acts like a master editor. It takes the broken optical film and the grainy radar film and stitches them together to create a smooth, continuous, colorful movie for any date—whether that date is in the past (filling gaps) or in the future (predicting what comes next).

Here is how the AI Director works, using simple analogies:

1. The Two Specialized Eyes

The AI has two separate "eyes" to look at the data.

• The Optical Eye looks at the colorful pictures.
• The Radar Eye looks at the black-and-white pictures.
  Instead of forcing both eyes to see the same way, the AI lets them learn their own language first. This is like having a painter and a sculptor work separately before they collaborate; the painter understands color, and the sculptor understands shape and structure.

2. The "Time Travel" Calendar

The AI doesn't just look at the pictures; it knows when they were taken. It uses a special "Time Travel Calendar."

• If the AI needs to predict a picture for next Tuesday, it asks: "What did the field look like last Monday? What about three weeks ago?"
• It calculates the distance between the "now" and the "then." This helps it understand that a field looks very different in spring than in autumn, even if the pictures are blurry.

3. The Smart Spotlight (Cross-Attention)

This is the AI's most clever trick. Imagine a spotlight on a stage with many actors (the different satellite pictures). The AI needs to decide which actors to listen to for the final scene.

• Scenario A (Clear Sky Nearby): If there is a clear, colorful picture from yesterday, the spotlight shines brightly on that one. The AI ignores the radar pictures because it doesn't need them; the color is already there.
• Scenario B (Heavy Clouds): If the last few colorful pictures are covered in clouds (white static), the AI realizes, "I can't use these!" It immediately swings the spotlight to the Radar pictures. Even though they are black-and-white, they show the shape of the crops, helping the AI guess what the colors should be.
• Scenario C (The Cloudy Trap): If a picture is taken yesterday but is covered in clouds, the AI learns to ignore it completely, even though it's "close" in time. It knows that a cloudy picture is worse than a clear picture from a week ago.

4. The "Confidence Meter" (Uncertainty)

Most AI tools just give you a picture and hope for the best. This AI is different: it also hands you a "Confidence Meter" (an uncertainty map).

• If the AI is guessing based on a clear picture from yesterday, the meter says: "I am 100% sure."
• If the AI has to guess what the field will look like two months from now, or if it's guessing through a thick storm, the meter says: "I'm not so sure about this part."
• Why this matters: It's like a weather forecaster saying, "It will rain, but I'm only 60% sure," rather than just saying "It will rain." This helps users know when to trust the image and when to be careful.

5. The Results

The paper tested this "Director" on real farmland data:

• Filling Gaps: It successfully reconstructed missing days in the movie, especially for crops that change quickly (like growing wheat), doing a better job than simple math tricks or older AI models.
• Predicting the Future: It could guess what the field would look like weeks after the last photo was taken. It wasn't perfect (the further out it guessed, the fuzzier the image), but it kept the general colors and shapes right.
• The "Snow" Mistake: The authors admit the AI gets confused by snow. Since it was trained on clouds, it sometimes thinks snow is just another type of cloud and tries to "erase" it to show the ground underneath, which is wrong. It also gets confused by very bright city lights.

Summary

This paper presents a new way to watch Earth's story without missing a beat. By combining "color" cameras (that get blocked by clouds) with "shape" cameras (that see through clouds), and by teaching the AI to know when to trust which camera, they created a system that can fill in missing movie scenes and predict future scenes. Crucially, it also tells you how much it trusts its own predictions, acting like a responsible editor who admits, "I'm guessing here."

Technical Summary

Technical Summary: Densification and Forecasting of Sentinel-2 Time Series from Multimodal SAR and Optical Data

Problem Statement

Optical satellite image time series (SITS) are critical for Earth observation applications such as agriculture, climate monitoring, and land surface analysis. However, their utility is severely constrained by acquisition gaps and irregular sampling caused by cloud cover and swath edges. While existing deep learning approaches have successfully addressed cloud removal and temporal densification (interpolation) within the observed time window, they face two primary limitations:

1. Lack of Forecasting Capability: Most methods are restricted to reconstructing gaps within the temporal extent of available data and cannot predict future observations (extrapolation).
2. Absence of Uncertainty Quantification: Existing models typically provide point estimates without explicitly quantifying the reliability or confidence of their predictions, which is crucial for downstream decision-making.
3. Data Constraints: Many approaches rely on temporally aligned multimodal inputs or explicit cloud masks, limiting their robustness in real-world, sparse, and irregular scenarios.

Methodology

The authors propose a probabilistic deep learning framework designed to generate optical images at arbitrary past (interpolation) or future (extrapolation) dates from sparse, irregular, and multimodal time series (Sentinel-2 Optical and Sentinel-1 SAR). The approach is formulated as a target-conditioned image generation problem.

Architecture Overview

The model consists of three primary components:

1. Spatial Feature Extraction:

   • Separate 2D convolutional encoders process Sentinel-2 (RGB-NIR) and Sentinel-1 (VV/VH) data independently to capture modality-specific spatial patterns.
   • A Spatial Pyramid Pooling (SPP) mechanism is employed to aggregate multi-scale contextual information while preserving fine-grained spatial details and preventing over-smoothing.
   • Input sequences are reshaped to process each acquisition independently in the spatial dimension before temporal modeling.

2. Temporal Encoding and Cross-Attention:

   • Temporal Encoding: The model explicitly encodes time to handle irregular sampling.
     • The target date ($d_{target}$) is encoded using a continuous representation based on the day of the year (DOY) to capture seasonality.
     • Input acquisition dates ($d_i$) are encoded relative to the target date ($\Delta d = d_i - d_{target}$), capturing both temporal distance and direction (past/future).
   • Cross-Attention Mechanism: The temporal encoding of the target date serves as the query, while the spatio-temporal tokens from all available Sentinel-1 and Sentinel-2 acquisitions serve as keys and values.
   • This mechanism allows the model to selectively aggregate information from the most temporally relevant observations without requiring explicit cloud masks or temporally aligned inputs. The model learns end-to-end to weight reliable observations and ignore unreliable ones (e.g., cloudy pixels).

3. Probabilistic Decoder:

   • Instead of predicting a single deterministic image, the decoder predicts the parameters of a pixel-wise Laplace distribution (mean $\mu$ and scale $b$) for each spectral band.
   • The Laplace distribution is chosen for its heavier tails, which are more robust to large prediction errors and reduce over-smoothing compared to Gaussian assumptions.
   • The model outputs both the reconstructed optical image ($\mu$) and an uncertainty map (derived from the scale parameter $b$).
   • Training is performed by minimizing the negative log-likelihood of the Laplace distribution.

Key Contributions

1. Unified Interpolation and Extrapolation: Unlike prior works limited to gap-filling, this framework generates images at arbitrary target dates, supporting both reconstruction within the observation window and forecasting beyond it.
2. Explicit Uncertainty Modeling: The probabilistic formulation provides pixel-wise uncertainty estimates that are well-calibrated, offering a measure of prediction reliability that increases when data is sparse or temporally distant.
3. Robust Multimodal Fusion: The approach jointly exploits Sentinel-1 SAR and Sentinel-2 optical data without relying on external cloud masks or strict temporal alignment. The cross-attention mechanism adaptively leverages SAR data when optical observations are missing or contaminated.
4. End-to-End Learning: The model learns to handle irregular sampling and cloud contamination implicitly, removing the need for pre-processing steps like cloud masking.

Experimental Results

The method was evaluated on a dataset of Sentinel-1 and Sentinel-2 patches (96x96 pixels) covering diverse landscapes, with a focus on agricultural areas.

Interpolation Performance

• Quantitative Metrics: The proposed model outperformed both a linear interpolation baseline and the U-TILISE (optical-only sequence-to-sequence) model across all land-cover types (Urban, Forest, Cropland) in terms of MAE, RMSE, and PSNR.
• Dynamic Regions: The performance gap was most significant in cropland areas, where the model successfully captured complex, non-linear seasonal dynamics that linear interpolation and optical-only models failed to model accurately.
• Qualitative Analysis: In scenarios with large temporal gaps, the model produced sharper reconstructions than baselines. Crucially, regions with high reconstruction difficulty (e.g., large gaps) exhibited higher predicted uncertainty, demonstrating coherent uncertainty estimation.

Extrapolation Performance

• The model successfully generated plausible future observations, maintaining radiometric consistency despite the inherent difficulty of extrapolation.
• While errors were higher than in interpolation, the model preserved main vegetation dynamics (e.g., NDVI evolution) over long gaps (e.g., 1 month and 20 days).
• Uncertainty maps correctly highlighted areas of lower confidence during extrapolation.

Ablation Studies

• Multimodal vs. Optical-Only: The inclusion of Sentinel-1 SAR data consistently improved performance (lower MAE/RMSE, higher PSNR) compared to an optical-only variant, confirming the value of SAR's structural information in cloudy or sparse conditions.
• Temporal Encoding: Using relative temporal encoding ($\Delta d$) significantly outperformed absolute date encoding, suggesting that explicitly informing the network of the temporal offset to the target date is critical for performance.
• Attention Analysis: Visualizing attention weights revealed that the model adaptively shifts focus: it relies heavily on temporally close optical data when available, but significantly increases attention to the nearest SAR acquisition when optical data is missing or heavily clouded.

Significance and Limitations

Significance:
The paper claims that this framework represents a step forward in generating continuous, reliable optical time series by addressing the dual challenges of forecasting and uncertainty quantification. The ability to provide well-calibrated uncertainty estimates is highlighted as a key contribution for applications requiring decision support, scenario analysis, and risk assessment. The adaptive fusion of SAR and optical data without manual masking demonstrates a robust approach to real-world data irregularities.

Limitations:
The authors acknowledge several limitations:

• Extreme Reflectance: The model tends to underestimate very high reflectance values (e.g., artificial surfaces, bright crops) due to regularization from the training distribution.
• Snow and Mountains: The model lacks training on mountainous or snow-covered regions. It frequently confuses snow with clouds, attempting to reconstruct the underlying surface (e.g., soil) rather than preserving the snow signal, leading to physically incorrect predictions.
• Out-of-Distribution: The model shows reduced robustness in conditions not present in the training set (e.g., snow), though the associated high uncertainty serves as a warning indicator.

The authors conclude that while the framework is effective for general land surface monitoring, future work should focus on expanding training datasets to include diverse environmental conditions (snow, mountains) and potentially incorporating additional spectral bands to better discriminate between highly reflective surfaces.