📊 statistics

Reliability of Probabilistic Emulation of Physical Systems

This paper introduces a framework and the AutoCast toolkit to systematically evaluate probabilistic emulation of physical systems, finding that CRPS-trained ensembles generally offer more reliable uncertainties and faster inference than generative models, particularly when the latter are constrained to latent spaces.

Original authors: Sam F. Greenbury (The Alan Turing Institute), Radka Jersakova (The Alan Turing Institute), Paolo Conti (The Alan Turing Institute, Autodesk Research), Marjan Famili (The Alan Turing Institute, Physics

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sam F. Greenbury (The Alan Turing Institute), Radka Jersakova (The Alan Turing Institute), Paolo Conti (The Alan Turing Institute, Autodesk Research), Marjan Famili (The Alan Turing Institute, PhysicsX), Christopher Iliffe Sprague (The Alan Turing Institute, Orbital), Edwin Brown (The Alan Turing Institute, University of Sheffield), Jason D. McEwen (The Alan Turing Institute, University College London)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, the flow of traffic, or how a drop of ink spreads in water. Scientists use complex computer simulations to do this, but these simulations are often so slow and expensive that they can't be used for real-time decisions.

To fix this, researchers have built "emulators"—AI models that act like fast, cheap stand-ins for the real physics simulations. But there's a catch: these AI models need to tell us not just what will happen, but how sure they are about it. If an AI says, "It will rain," but it's actually just guessing, that's dangerous.

This paper compares two different ways of building these "uncertainty-aware" AI models to see which one gives the most reliable confidence levels.

The Two Contenders

Think of the two approaches as two different types of weather forecasters:

1. The "Generative" Artist (The Diffusion/Flow Model)

How it works: Imagine an artist who starts with a blank canvas full of random noise (static on an old TV). They slowly "paint" over the noise, step-by-step, turning the static into a clear picture of the future.
The Catch: To make this fast, the artist usually works in a tiny, compressed sketchbook (called "latent space") rather than the full-sized canvas. They draw a rough sketch and then try to expand it back to full size.
The Problem: The paper found that while these artists are good at drawing the picture, they often get the confidence wrong. They might draw a beautiful storm but act like they are 100% sure it will happen, even when they are actually guessing. Also, the step-by-step painting process is slow to run.

2. The "Ensemble" Crowd (The CRPS-Trained Model)

How it works: Imagine asking 8 different experts to predict the weather at the same time. Each expert is given a tiny, random "twist" in their thinking (noise injection) so they don't all give the exact same answer. You then look at the group's answers together. If they all agree, you are very confident. If they disagree wildly, you know it's a risky prediction.
The Catch: This method is trained using a specific scoring rule (CRPS) that rewards the group for being both accurate and spread out correctly.
The Benefit: This approach is incredibly fast to run (just ask the 8 experts once) and, according to the paper, gives much more honest answers about how uncertain the future really is.

The Big Experiment

The researchers set up a "taste test" using four different types of physical systems (like swirling fluids, chemical reactions, and quantum physics simulations). They made sure both teams used the same amount of computing power and model size so the race was fair.

What they found:

Honesty Wins: The "Ensemble Crowd" (CRPS) was much better at telling the truth about its uncertainty. If it said there was a 90% chance of an event, it actually happened about 90% of the time. The "Generative Artist" often underestimated the risks, acting too confident when it should have been worried.
Speed: The Ensemble method was lightning-fast. The Generative method was slow because it had to take many small steps to "paint" the result.
The Sketchbook Issue: When the Generative Artist worked in the tiny sketchbook (latent space) to save time, their confidence levels got even worse. However, if they worked on the full canvas (ambient space), they were as honest as the Ensemble, but the process was so slow it wasn't practical for big problems.
Accuracy: Both methods were actually very good at predicting what would happen (the average prediction was accurate for both). The main difference was in how well they described the uncertainty.

The Toolkit Released

To help other scientists, the authors released two new software tools:

AutoSim: A "Lego kit" for quickly building different types of physics simulations to test AI models.
AutoCast: A "construction set" that lets researchers easily build and test both the "Artist" and the "Crowd" models to see which works best for their specific problem.

The Bottom Line

If you need a fast, reliable AI that tells you not just the answer, but also how much you should trust that answer, the "Ensemble" approach (training many models with a specific scoring rule) is currently the winner. It is faster, more honest about its mistakes, and just as accurate as the more complex generative methods.

Technical Summary: Reliability of Probabilistic Emulation of Physical Systems

Problem Statement

The modeling of physical systems is critical across domains from climate science to materials engineering. While computational simulations are accurate, their high computational cost often bottlenecks practical deployment. Data-driven emulators offer a faster alternative, yet current benchmarks predominantly focus on deterministic point-prediction accuracy, overlooking the critical need for reliable uncertainty quantification (UQ). Real-world deployment requires emulators that can quantify both epistemic uncertainty (model limitations) and aleatoric uncertainty (inherent system stochasticity).

Two dominant probabilistic approaches have emerged for physical system emulation:

Generative Models: Such as denoising diffusion and flow matching (FM), which learn to sample from the conditional forecast distribution by transforming noise. These are often trained in compressed latent spaces to improve efficiency.
CRPS-Trained Ensembles: Deterministic model ensembles with injected stochasticity, trained using the Continuous Ranked Probability Score (CRPS) loss. These have gained traction in weather forecasting for their computational efficiency.

Despite demonstrated predictive accuracy in both approaches, the reliability of their predictive intervals (i.e., whether the stated uncertainty actually captures the true value) has not been systematically assessed or compared under controlled conditions. Miscalibrated uncertainties can lead to catastrophic failures in downstream decision-making.

Methodology

The authors developed a framework to evaluate and compare these two approaches across diverse 2D spatiotemporal physical systems under matched model sizes and computational budgets.

Datasets and Simulation (AutoSim)

To facilitate rapid prototyping and controlled experimentation, the authors introduced AutoSim, an open-source Python package for generating flexible datasets. They selected four distinct physical systems simulated on a $64 \times 64$ grid:

Advection–Diffusion (AD): A single-channel transport problem isolating coherent structure preservation.
Gray–Scott (GS): A two-species reaction–diffusion system producing diverse morphologies (spots, worms, spirals).
Conditioned Navier–Stokes (CNS): A buoyancy-driven flow coupling passive scalar transport with incompressible fluid dynamics.
Gross–Pitaevskii Equation (GPE): A nonlinear quantum-fluid model of a Bose–Einstein condensate, extending benchmarks beyond classical fluids.

Modeling Pipelines (AutoCast)

The authors released AutoCast, a modular framework implementing both pipelines:

Generative Pipeline (Flow Matching): Trained in latent space. An autoencoder compresses input/output states; FM learns a flow map from noise to the latent forecast distribution. Sampling involves solving an Ordinary Differential Equation (ODE).
Ensemble Pipeline (CRPS): Trained in ambient space. A deterministic backbone (Vision Transformer) receives injected noise via conditional layer normalization. The ensemble is trained using the $\alpha$ -fair CRPS loss ( $\alpha$ fCRPS), which optimizes for both accuracy and dispersion.

Evaluation Metrics

The study prioritizes empirical coverage as the primary metric for reliability. This measures the proportion of times a $(1-\alpha)$ prediction interval contains the true value. Ideally, empirical coverage should match nominal coverage. Secondary metrics include:

Predictive Accuracy: Variance-scaled Root Mean Squared Error (VRMSE) and CRPS.
Calibration: Spread-Skill Ratio (SSR) and Winkler interval score.
Efficiency: Training epoch time and inference latency.

Experiments covered single-step predictions and 100-step autoregressive rollouts.

Key Results

1. Reliability and Coverage

CRPS-trained ensembles generally provide more reliable uncertainties than latent-space generative models. They demonstrate better empirical coverage across single-step predictions and autoregressive rollouts.
Latent Space Limitations: Generative models trained in latent space tend to underestimate uncertainty, particularly on complex datasets like CNS and GPE.
Ambient Space Generative Models: When generative models are trained directly in ambient space (bypassing the latent bottleneck), they achieve coverage comparable to CRPS ensembles. However, this comes at a substantial cost in inference latency, making it often infeasible for high-dimensional problems.
Latent Space Ensembles: Training CRPS ensembles in latent space does not significantly degrade coverage compared to ambient space, offering a flexible alternative.

2. Accuracy and Efficiency

Accuracy: Both approaches demonstrate comparable predictive accuracy (VRMSE) on single-step tasks. In autoregressive rollouts, CRPS ensembles often maintain a slight accuracy advantage, particularly on the CNS dataset.
Inference Speed: CRPS-trained ensembles are significantly faster at inference time. Each ensemble member requires only a single forward pass, whereas generative models require multiple ODE integration steps.
Training Speed: Generative models (FM) train faster per epoch because CRPS training must evaluate multiple ensemble members simultaneously.

3. Ablation Studies

Compression: Increasing autoencoder compression (e.g., from 6x to 24x) degrades the performance of latent-space generative models, confirming that latent space size is a limiting factor.
Loss Variants: Using the Winkler score for checkpoint selection improved coverage on the GPE dataset compared to standard validation loss. The fCRPS variant showed better coverage than standard CRPS.
Architecture: ViT backbones outperformed U-Net backbones in CRPS ensembles regarding both accuracy and coverage. Diffusion and Flow Matching showed comparable performance in latent space.
Conditioning: Injecting global simulation parameters via channel concatenation outperformed backbone modulation for CRPS ensembles.

Significance and Contributions

The paper makes three primary contributions to the field of scientific machine learning:

Systematic Assessment of UQ Reliability: It provides the first systematic comparison of generative models and CRPS-trained ensembles regarding empirical coverage across diverse physical systems. The findings challenge the assumption that both approaches inherently yield reliable uncertainties, highlighting that latent-space generative models often suffer from under-dispersion.
Practical Trade-off Analysis: The study clarifies the trade-offs between accuracy, reliability, and computational cost. It suggests that for applications requiring fast, reliable uncertainty estimates (e.g., real-time risk assessment), CRPS-trained ensembles in ambient space are superior. Conversely, while ambient-space generative models can match reliability, their inference cost is prohibitive for many high-dimensional applications.
Open-Source Tooling: The release of AutoSim and AutoCast lowers the barrier to entry for researchers. AutoSim enables rapid dataset generation for controlled PDE experiments, while AutoCast provides a unified, modular framework for training and benchmarking both generative and ensemble-based probabilistic emulators, including interoperability with existing large-scale benchmarks like "The Well."

Conclusion

The authors conclude that while both approaches are theoretically well-motivated to learn full conditional distributions, approximations in practice (latent space compression, ODE discretization, finite ensemble sizes) impact reliability. CRPS-trained ensembles currently offer a more robust balance of reliability and efficiency for physical system emulation. Future research directions identified include improving latent representations to preserve calibration and exploring distributional objectives for autoencoders. The authors note that while conformal prediction can calibrate uncertainties post-hoc, accurate initial UQ remains essential to avoid uninformative error bars.