💻 computer science

From Surface Forecasting to Observability Forecasting: A Latent World Model for Cloud-Aware EO Monitoring

This paper introduces LeWorldModel, a latent world model adapted for Earth Observation that leverages multispectral imagery and weather data to significantly outperform persistence and LightGBM baselines in forecasting cloud cover and predicting the timing of future usable satellite acquisitions.

Original authors: Mohanad Albughdadi

Published 2026-07-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohanad Albughdadi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to watch a live sports game on a giant screen, but the view is constantly blocked by thick, moving clouds. Sometimes the clouds part for a second, then cover the field again for hours. In the world of Earth Observation, scientists use satellites to take pictures of our planet, but they face the exact same problem. The real challenge isn't that the satellites stop working or that the pictures don't arrive; the challenge is that the pictures often arrive with the ground completely hidden. It's like trying to read a book while someone keeps shuffling a deck of cards over the pages.

To solve this, researchers use "weather drivers"—data about rain, temperature, and pressure—to guess what the sky will look like. They also use a concept called a "world model." Think of a world model not as a machine that draws a perfect picture of the future, but as a smart student who learns the rules of how the world changes. Instead of trying to memorize every single pixel of a cloudy photo, this student learns the hidden pattern of "when the clouds usually move away." This paper asks a very practical question: Can we teach a computer to stop worrying about drawing the perfect future image and start focusing on answering a simpler, more useful question: "Is the view going to be clear enough to see the ground, and if not, how long do we have to wait?"

The Cloudy Window Problem

In the world of Earth Observation (EO), pipelines don't fail because data stops flowing; they fail because the data that arrives is useless. Clouds, haze, and missing pixels can hide the Earth's surface for days or weeks. This turns the main challenge from "getting data" to "knowing if the data is usable."

The authors of this paper, Mohanad Albughdadi from the European Centre for Medium-Range Weather Forecasts, decided to tackle this by changing the goal. Instead of trying to predict exactly what the next picture of the Earth will look like (a task called "surface forecasting"), they asked: Can we predict if the next picture will be usable, and if it's cloudy, when the clouds will finally clear?

They tested this idea using a dataset called EarthNet2021, which contains sequences of satellite images paired with weather data. To solve the problem, they used a special type of AI called LeWorldModel (LeWM).

The "Smart Student" vs. The "Artist"

Most AI models trying to predict the future act like artists. They try to reconstruct every single detail of the next image, pixel by pixel. If the sky is cloudy, the artist tries to guess what the clouds look like and what the ground underneath might be.

LeWorldModel, however, acts more like a smart student studying a pattern. It uses a "Joint-Embedding Predictive Architecture" (JEPA). Instead of trying to draw the next picture, it learns a latent state—a compressed, hidden summary of the situation. It asks: "Given the last few pictures and the current weather, what is the state of visibility?"

The authors argue that for monitoring the Earth, you don't need to know if the next image will show a specific tree or a specific road. You just need to know the "state" of the scene: Is it hidden? Is it about to become visible? This makes the model much more efficient. It has 18.0 million trainable parameters and was trained on 23,904 episodes of satellite data.

The Experiment: Can It Predict the Clearing?

The researchers set up a strict test to see if their "smart student" could beat two other methods:

Persistence: This is the "lazy" method. It assumes the future will look exactly like the present. If it's cloudy now, it predicts it will be cloudy next.
LightGBM: A powerful, standard machine learning tool that looks at the numbers (weather history, calendar dates) but doesn't "see" the images.

They tested the model on four different scenarios:

Valsplit: A standard test set.
IID: Data that looks very similar to the training data.
OOD: Data that is different from the training data (Out-of-Distribution).
Extreme: Data with very heavy cloud cover and longer time gaps.

The Results: When to Wait, When to Act

The results showed that LeWorldModel is a fantastic "observability forecaster," but it has specific strengths and weaknesses compared to the other tools.

1. Beating the "Lazy" Method
The model consistently crushed the "persistence" method.

Next-step usability: When asked if the very next picture would be usable, LeWorldModel achieved a balanced accuracy between 0.769 and 0.887. Persistence only managed 0.493 to 0.556.
Timing the return: When asked to predict exactly when the view would clear up, LeWorldModel was much more accurate (0.602 to 0.806) compared to persistence (0.120 to 0.369).

2. The Battle with LightGBM
The comparison with LightGBM revealed a split in the team's skills:

The Simple Question: For the easy question "Will any usable picture appear in the next six steps?", LightGBM was actually the winner on most tests. It is very good at looking at weather numbers and making a quick "yes/no" decision.
The Complex Question: For the harder questions—like predicting the exact moment the clouds will clear, or estimating the continuous quality of the view (how much of the ground is visible)—LeWorldModel was the champion.
- On the Extreme split (the hardest, cloudiest data), LeWorldModel predicted the exact timing of the first usable view with 0.705 accuracy, while LightGBM got 0.633.
- In terms of error, LeWorldModel was off by an average of 0.627 steps on the Extreme split, whereas LightGBM was off by 0.885 steps.

3. The Weak Spot
The paper notes that LeWorldModel is not perfect everywhere. On the OOD (Out-of-Distribution) split, where the data was very different from what it was trained on, LightGBM actually performed better. This suggests that while the "smart student" is great at learning patterns it has seen before, it can get confused when the weather behaves in a totally new way.

What This Means for the Future

The authors are careful not to overhype the results. They clarify that this model is not a tool for planning actions (like telling a satellite exactly where to fly) because the weather data used is external and cannot be controlled. It is also not a video generator that creates new pictures of the Earth.

Instead, the paper concludes that LeWorldModel is a highly effective latent observability forecaster. It is a tool that tells operators: "Don't waste time analyzing the next image; it's still cloudy. But good news, the clouds should clear up in about 3 hours."

In the end, the value of this work isn't that it can draw a perfect future picture. It's that it can tell us when we will finally be able to see the ground again, saving time and resources for everyone watching the Earth from space.

Technical Summary: From Surface Forecasting to Observability Forecasting

Problem Definition
The paper addresses a critical bottleneck in Earth Observation (EO) processing: the arrival of new imagery does not guarantee usability. Clouds, haze, and missing pixels often render observations unusable for days or weeks. While traditional EO research focuses on reconstructing future surface imagery (pixel forecasting) or predicting surface indices, this work reframes the problem as observability forecasting. The core question is not "what will the surface look like?" but rather:

Will the next acquisition be usable?
If not, will a usable observation return within a specific horizon?
Exactly when will the first usable observation return?

The authors utilize the EarthNet2021 dataset, which provides multispectral Sentinel-2 imagery paired with mesoscale meteorological drivers, to train a model that predicts observation quality under cloud occlusion.

Methodology
The study adapts LeWorldModel (LeWM), a Joint-Embedding Predictive Architecture (JEPA) world model, to the EO domain. Unlike generative models that reconstruct pixels, LeWM learns a latent state representation that evolves over time based on past observations and exogenous covariates.

Data Pipeline: Raw EarthNet2021 "minicubes" are converted into episodic HDF5 sequences. Each observation $x_t$ consists of five channels (Blue, Green, Red, NIR, and a binary Cloud Mask). Covariates $u_t$ include time deltas, cyclical day-of-year features, and aggregated weather statistics (precipitation, pressure, temperature) between frames. Crucially, cloudy frames are not dropped; they are preserved with zeroed spectral values and an active cloud mask to maintain temporal alignment.
Model Architecture: The adapted LeWM contains 18.0M trainable parameters. It features:
- A channel adapter mapping 5 EO channels to a 3-channel ViT-tiny encoder input.
- A ViT-tiny encoder (patch size 16) producing a 192-dimensional latent state $z_t$ .
- An action encoder mapping 8-dimensional weather covariates to the same latent width.
- A 6-layer conditional autoregressive transformer predictor that forecasts future latent states $(\hat{z}_{t+1}, \dots)$ given the history of latents and covariates.
Training Objective: The model is trained from scratch using a teacher-forced loss over a 10-step history to predict the next 10 latent states. The objective combines a latent prediction loss ( $L_{pred}$ ) with an isotropic Gaussian regularizer ( $R_{sig}$ ) to enforce stable latent geometry.
Evaluation Protocol: A "locked" protocol is employed to ensure rigorous generalization testing. The model is trained once on the training split. Linear probes and calibration thresholds are fitted only on the internal validation split (valsplit). These heads are then frozen and evaluated on four distinct splits: valsplit, IID (Independent and Identically Distributed), OOD (Out-of-Distribution), and Extreme (longer, cloudier sequences).

Key Contributions

Problem Reframing: The authors shift the EarthNet2021 benchmark from generic surface forecasting to observability forecasting, defining explicit targets for next-step usability, horizon-based usability, and exact recovery timing.
Reproducible Pipeline: A complete adaptation pipeline is provided, including cloud-aware data conversion, explicit cloud-mask handling, and dataset-specific normalization.
Rigorous Evaluation: The paper establishes a locked evaluation protocol that separates training, probe fitting, and testing across multiple splits to distinguish true generalization from split-specific adaptation.
Empirical Findings: The study demonstrates that LeWM is most effective not as a spatial video predictor, but as a latent observability forecaster. It outperforms simple persistence baselines significantly and offers distinct advantages over tabular baselines (LightGBM) in specific temporal tasks.

Results
The model was evaluated against two baselines: Persistence (copying the last observed state) and a frozen LightGBM baseline trained on explicit quality histories and weather covariates.

Latent Forecasting: LeWM consistently outperforms persistence in predicting next-step latent states and six-step rollouts across all splits, confirming it learns non-trivial temporal dynamics.
Next-Step Usability (Binary): LeWM achieves balanced accuracy ranging from 0.769 to 0.887, significantly beating persistence (0.493–0.556). Against LightGBM, LeWM performs better on valsplit and Extreme splits but is slightly outperformed on OOD and tied on IID.
Exact Recovery Timing: This is where LeWM shows the strongest advantage. For predicting the exact horizon of the first usable observation, LeWM achieves accuracies of 0.602 to 0.806, vastly superior to persistence (0.120–0.369) and generally better than LightGBM on valsplit, IID, and Extreme splits.
Continuous Quality Regression: LeWM provides lower RMSE for clear/cloud fraction regression on most splits compared to LightGBM, except on the OOD split where LightGBM is slightly more robust.
Anomaly Detection: The model produces strong ranking-based signals for synthetic temporal inconsistencies (e.g., shuffled future frames), serving as a temporal consistency checker.

Significance and Claims
The paper modestly claims that the value of LeWorldModel in this context is not as a spatial forecaster or a planning system, but specifically as a latent observability forecaster for cloud-aware monitoring.

Task Specialization: The results reveal a clear task split: LightGBM excels at simpler binary decisions ("Will any usable observation appear in the next 6 steps?"), likely because explicit quality histories contain sufficient signal for this. LeWM, however, is superior for continuous quality regression and exact timing prediction, suggesting its latent representation captures richer temporal state evolution that tabular models miss.
Limitations: The authors acknowledge that OOD performance remains a weak point, where the latent distribution shifts significantly from the training regime. Furthermore, the model relies on known future weather covariates (exogenous forcing) and does not perform weather forecasting or policy optimization.
Conclusion: The primary contribution is demonstrating that a JEPA-based world model can effectively track the hidden evolution of scene visibility. It provides a practical tool for EO pipelines to determine when to expect usable data, rather than just attempting to reconstruct obscured surfaces.