🔬 physics

Do AI Forecast Ensembles Sample the Correct Conditional Distribution?

This paper demonstrates that AI forecast ensembles, specifically diffusion models for coastal sea level, can achieve positive marginal skill while failing to correctly capture the joint spatial distribution of outcomes—a structural inadequacy invisible to standard energy scores but detectable via variogram scores and reproducible by linear baselines.

Original authors: Lucas J. Howard, Elizabeth A. Barnes

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lucas J. Howard, Elizabeth A. Barnes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to predict the weather for an entire city, not just a single neighborhood. You know that if it rains in the north, it's likely to rain in the south too; the weather doesn't happen in isolated bubbles. This is the heart of ensemble forecasting: instead of guessing one single future, scientists run a computer model dozens or hundreds of times to create a "cloud" of possible outcomes. The goal isn't just to get the average right, but to make sure the spread of those outcomes looks like the real world. If the real world has a tight connection between different places, the forecast cloud should show that same connection.

For a long time, these clouds were made by massive, physics-based supercomputers that simulate the laws of nature. But recently, a new kind of "AI weather forecaster" has arrived. These are like brilliant students who have memorized thousands of years of past weather patterns and learned to guess the future based on that memory. They are incredibly fast and can generate huge clouds of possibilities. But here is the big question: Just because these AI models are fast and smart at guessing the weather for one specific spot, does their cloud of possibilities actually capture how different spots are connected to each other? If the AI thinks it's raining in Boston and sunny in Miami, but in reality, they are usually both rainy or both sunny, the forecast is broken, even if the individual guesses look good.

The Great AI Weather Disconnect

In this study, researchers at Boston University put a popular type of AI weather model to the test. They trained a "diffusion model"—a fancy AI that learns by slowly turning noise into a clear picture—to predict sea levels along the US East Coast. They wanted to see if the AI could correctly predict not just the water level at one tide gauge, but how the water levels at eight different stations (from Maine to Florida) moved together.

Think of it like a choir. The AI is great at singing the right note for every single singer (the individual stations). If you ask, "What is the water level in Boston?" the AI's answer is usually spot on. But when you listen to the whole choir, the AI is singing out of sync. It fails to capture the harmony between the singers.

The researchers found a surprising "skill gap." When they checked the AI's performance using standard tests that look at one station at a time, the AI was a star student, beating random guesses. However, when they used a special test designed to check if the stations were "singing together" correctly, the AI actually performed worse than just picking random historical weather patterns. It was as if the AI knew the lyrics to every song perfectly but had no idea how to harmonize with the other singers.

The "More Data" Myth

You might think, "Maybe the AI just needs to study harder! If we give it more training data, it will learn the harmony." The researchers tested this by running experiments in a simplified, computer-generated world (called the Lorenz-96 system) that mimics chaotic weather. They fed the AI everything from a tiny bit of data up to a massive amount—equivalent to 170 years of weather records.

The result? The gap didn't close. Even with a mountain of data, the AI got better at predicting individual spots but remained terrible at predicting how those spots were connected. It turns out this isn't a problem of the AI being "under-studied"; it's a structural flaw in how these AI models learn. They seem to learn the "solo" parts of the song very well but struggle to learn the "ensemble" parts.

The "Copycat" vs. The "Physics"

To figure out why this was happening, the team compared three different types of forecasters:

The AI Diffusion Model: The fancy new student.
The Dynamical Model: The traditional physics-based computer that solves equations of motion (the old-school expert).
The Deterministic Emulator: An AI that acts like a copycat, taking a starting point and running it forward without any randomness.

They found that the traditional physics model (the dynamical ensemble) didn't have this problem; it kept the harmony intact. The AI copycat, however, failed just like the diffusion model. This suggests the issue isn't with "ensemble forecasting" in general, but specifically with how these data-driven AI models are built. They seem to have a hard time learning the complex, multi-dimensional relationships between different parts of the system, even when they are good at the simple parts.

Why This Matters

The study suggests that we can't just assume that bigger AI models or more training data will automatically fix these hidden errors. If we rely on these AI forecasts for things like coastal flood warnings, we might get a great prediction for one town but a completely wrong picture of how the storm surge moves along the whole coast.

The researchers conclude that while AI is a powerful tool, it currently has a blind spot regarding how different parts of the Earth connect. To get the full picture, we likely need to keep using traditional physics-based models alongside AI, and we need to develop new ways to test AI that don't just look at the solo performance but check the harmony of the whole choir. Until then, the AI might be a brilliant soloist, but it's not quite ready to lead the orchestra.

Technical Summary: Do AI Forecast Ensembles Sample the Correct Conditional Distribution?

Problem Statement
Ensemble forecasting aims to generate samples from a conditional distribution that accurately represents the true uncertainty of outcomes given specific conditions (e.g., initial states). While the rapid advancement of Artificial Intelligence (AI) in earth system science offers the potential for larger, more computationally efficient ensembles, a foundational question remains untested: do AI forecast ensembles correctly sample the joint distribution of outcomes, or do they only capture marginal statistics? Current evaluation metrics may mask failures in multivariate structure, leading to ensembles that appear skillful at individual locations but fail to represent the spatial correlations inherent in the true uncertainty.

Methodology
The authors employ a multi-faceted approach combining real-world oceanographic data with idealized dynamical systems to isolate the source of potential distributional failures.

Real-World Application (Coastal Sea Level):
- Model: A Denoising Diffusion Probabilistic Model (DDPM) is trained to generate probabilistic forecasts of weekly coastal sea surface height (SSH) anomalies at eight tide gauge stations along the US East Coast.
- Data: Inputs include GLORYS SSH and Sea Surface Temperature (SST), ERA5 wind and sea level pressure (NAO index), and Niño 3.4 indices, compressed via Empirical Orthogonal Functions (EOFs). The model is trained on 1993–2018 data and evaluated on a locked 2022–2025 test set.
- Evaluation Metrics: The study utilizes three scoring rules:
  - Continuous Ranked Probability Score (CRPS): Assesses marginal distribution skill at single locations.
  - Energy Score (ES): A multivariate extension of CRPS, noted for being potentially insensitive to joint structure when marginal skill dominates.
  - Variogram Score (VS): A multivariate score specifically designed to be sensitive to dependence structures.
- Decomposition: A shuffle-based permutation decomposition is applied to isolate the contribution of joint spatial structure to the scores. By independently permuting ensemble values at each station, the authors destroy cross-station correlations while preserving marginal distributions, allowing for the quantification of the "joint contribution" ( $\Delta$ ).
Idealized Experiments (Lorenz-96):
- To determine if the observed failures are due to insufficient training data or structural model inadequacy, the authors utilize the Lorenz-96 (L96) chaotic system.
- Design: DDPMs are trained on L96 trajectories with varying dataset sizes (equivalent to 0.7 to 170 years of ERA5 data).
- Comparisons: The DDPM is compared against:
  - A linear baseline (Ordinary Least Squares with full residual covariance, MVN).
  - A dynamical ensemble (integrating L96 equations from perturbed initial conditions).
  - A deterministic AI emulator (a fixed forward map applied to perturbed initial conditions).

Key Contributions and Results

Decoupling of Marginal and Joint Skill: In the coastal SSH forecasts, the DDPM achieves statistically significant positive CRPS skill (marginal) at all lead times. However, the Variogram Score (VS) is consistently negative, indicating that the joint spatial structure of the ensemble is worse than random climatological draws.
Insensitivity of Energy Score: The shuffle decomposition reveals that the Energy Score (ES) is largely insensitive to this failure. While the DDPM's joint contribution to ES is statistically significant, it is only modestly better than climatology. In contrast, VS detects a substantial joint contribution that is nonetheless miscalibrated (worse than observed).
Structural Inadequacy vs. Data Volume: Lorenz-96 experiments demonstrate that the gap between CRPS and VS skill persists regardless of training dataset volume, even when scaled to 170 equivalent years (exceeding the ERA5 record). A linear MVN baseline exhibits the same qualitative pattern, suggesting the failure is not specific to the non-linear DDPM architecture but is a structural limitation of learned distributions in this context.
AI vs. Dynamical Models: When comparing forecast types in the L96 system, a dynamical ensemble (perturbed initial conditions) maintains a VS/CRPS ratio near 1.0, indicating proper joint calibration. Conversely, both the probabilistic DDPM and a deterministic AI emulator show a significantly lower VS/CRPS ratio. This suggests the failure is specific to data-driven emulation rather than a general property of ensemble forecasting in chaotic systems.
Heterogeneity of Failure: In the coastal application, the joint skill deficit is spatially heterogeneous. Pairs of stations with high observed correlations (e.g., Atlantic City and The Battery) show the largest deficits, suggesting the model may over-correlate highly correlated pairs, while other pairs show near-zero failure.

Significance
The paper concludes that probabilistic generative AI forecast tools may sample from a distribution that does not properly represent the joint conditional structure of true uncertainty, despite achieving positive marginal skill. The authors argue that:

Evaluation Metrics: Standard metrics like CRPS and ES are insufficient for identifying this failure mode; VS is required to detect structural miscalibration.
Training Data: Increasing the volume of training data cannot be assumed to remedy this specific failure mode, as it appears to be a structural limitation of the learned distribution.
Model Nature: The failure is reproduced in deterministic AI emulators but not in dynamical models, highlighting a specific vulnerability of data-driven approaches.
Implications: These findings support the continued necessity of dynamical and physics-based models to complement AI forecasts and suggest that current practices in model evaluation and training require adjustment to ensure multivariate reliability. The results align with recent literature indicating that AI emulators can produce distorted background error covariances, leading to instabilities in data assimilation systems.

The Great AI Weather Disconnect

The "More Data" Myth

The "Copycat" vs. The "Physics"

Why This Matters

More like this