Stochasticity and probabilistic trajectory scoring are essential for data-driven closures of chaotic systems

This paper demonstrates that accurately modeling chaotic systems requires combining stochastic closures with trajectory-based calibration using proper scoring rules, as deterministic pointwise losses suppress essential predictive variance and fail to capture long-term statistics.

The Gist

Imagine you are trying to predict the weather, but you only have a low-resolution map. You can see the big storms and the general wind patterns, but you can't see the tiny swirls, eddies, and gusts that happen in the gaps between your map's grid lines.

In science, this is called a coarse-grained model. You are ignoring the tiny details to save computing power. The problem is, those tiny details matter. They interact with the big picture, and if you ignore them, your model eventually goes off the rails. It might predict a calm day when a storm is coming, or it might slowly lose all its "energy" and become a boring, flat gray blob.

This paper, written by Martin T. Brolly, solves a specific puzzle about how we teach computers to fill in those missing details. Here is the breakdown in simple terms:

The Problem: The "Perfect Path" Trap

For a long time, scientists taught their computer models to be deterministic. This means they asked the computer: "If the wind is blowing this way right now, what is the one single, perfect path it will take next?"

They trained the computer by looking at a short time step (like one hour) and saying, "You were wrong by 5%; try to be closer next time."

The Analogy: Imagine you are trying to teach a robot to walk through a crowded, chaotic dance floor.

• The Old Way: You tell the robot, "Take one step. If you hit a person, you failed. Try to find the one perfect step that avoids everyone."
• The Result: The robot gets scared. To avoid hitting anyone, it decides to just stand still or shuffle very slowly in a straight line. It stops dancing. It becomes "over-smoothed." It loses the chaotic, energetic nature of the dance floor.

In the paper, the author proves that if you train a model to find that "one perfect path" over a long period, the math forces the model to kill off all the natural randomness. It suppresses the variance. The model becomes a boring, predictable, and physically wrong version of reality.

The Solution: Embrace the Chaos

The paper argues that because the tiny details we ignored are chaotic and unpredictable, we shouldn't try to predict a single path. Instead, we should predict a range of possibilities.

The New Way: Instead of asking, "What is the one path?", we ask, "What is the cloud of possible paths the wind could take?"

• The Analogy: Now, you tell the robot: "You can't know exactly where everyone will move. So, imagine 20 different versions of yourself walking through the crowd at the same time. Some might bump into people, some might dodge. Your job isn't to pick the perfect path; your job is to make sure that the group of you covers the whole dance floor realistically."

This is called a Stochastic Closure. It adds a little bit of "random noise" (like a dice roll) to the model to represent the missing tiny details.

The Secret Sauce: Scoring the "Cloud," Not the "Point"

The paper also points out that you can't just use the old scoring method (checking if the robot hit a person) anymore. You need a new way to grade the robot's performance.

• Old Score (Mean Squared Error): "Did your single path match the reality?" (This punishes the robot for having a wide range of guesses).
• New Score (Strictly Proper Scoring Rule): "Did your cloud of guesses cover the reality correctly?"

The author uses a specific scoring rule called the Energy Score. Think of it like this:

• If the robot's cloud of guesses is too tight (too confident), it gets penalized if it misses.
• If the robot's cloud is too wide and messy, it gets penalized for being useless.
• If the robot's cloud perfectly matches the spread and shape of the real chaotic dance floor, it gets a perfect score.

What Happened in the Experiment?

The author tested this on a simulated ocean/atmosphere system (Quasi-Geostrophic turbulence).

1. The "One-Step" Trained Model: Failed miserably. It couldn't even keep the simulation running for long; it became unstable.
2. The "Long-Term Deterministic" Model: It ran, but it became a "boring blob." It smoothed out all the interesting swirls and eddies. It looked like a calm pond instead of a stormy sea.
3. The "Stochastic + Trajectory" Model: This was the winner. By training the model to predict a range of outcomes over a long period using the new scoring rules, it successfully recreated the chaotic, swirling energy of the real system. It kept the big storms and the tiny eddies.

The Big Takeaway

If you are trying to model a chaotic system (like weather, climate, or fluid flow) and you have to ignore some details, do not try to predict a single perfect future.

1. Accept Uncertainty: The missing details are random, so your model must be random too.
2. Train on the Long Haul: Don't just check if you were right for one hour; check if your model behaves correctly over weeks or months.
3. Score the Distribution: Don't grade the model on hitting a single target. Grade it on whether its "cloud of possibilities" looks like the real world.

By doing this, we stop our models from becoming "boring blobs" and allow them to capture the beautiful, chaotic mess of the real universe.

Technical Summary

1. Problem Statement

The paper addresses the fundamental challenge of closure modeling in chaotic systems (e.g., fluid turbulence, climate models). When complex systems are coarse-grained (reduced-order modeling), unresolved degrees of freedom ($x'$) are neglected, inducing structured model error.

• The Core Issue: Standard data-driven closures are typically trained offline to minimize one-step prediction errors (e.g., Mean Squared Error, MSE). This implicitly assumes the dynamics are Markovian (memoryless). However, coarse-grained dynamics are inherently non-Markovian and stochastic due to the interaction between resolved and unresolved scales.
• The Gap: While recent methods attempt to train over multi-step trajectories (online learning) to account for memory, they often still use deterministic pointwise losses (like MSE). The authors argue this creates a fundamental mathematical degeneracy: optimizing deterministic losses over chaotic trajectories forces the model to suppress predictive variance, leading to unphysical "over-smoothing" and distorted long-term statistics.

2. Methodology

The authors propose a framework combining generative stochastic modeling with trajectory-based probabilistic calibration.

A. Theoretical Framework

• Generative Closures: The model error is represented as a conditional generative model: $\hat{m}_n = G_\theta(\hat{x}_n, \xi_n)$, where $G_\theta$ is a neural network and $\xi_n$ is auxiliary noise. This allows the model to output a probability distribution rather than a single point.
• Loss Functions:
  • Deterministic Baseline: Uses Euclidean distance (equivalent to MSE) over trajectories.
  • Probabilistic Approach: Uses Strictly Proper Scoring Rules, specifically the Energy Score. The Energy Score evaluates the quality of a forecast distribution against an observation without requiring a tractable likelihood function. It is defined as:
    $$S(\nu, y) = \mathbb{E}_\nu \|Y - y\| - \frac{1}{2}\mathbb{E}_\nu \|Y - Y'\|$$
    where $Y, Y'$ are independent samples from the forecast distribution $\nu$.
• Training Strategy: Models are trained online (a posteriori) over finite time windows ($w$). The loss is the average Energy Score over the window, forcing the model to balance short-term accuracy with long-term statistical consistency.

B. Theoretical Proofs

The paper provides rigorous mathematical proofs regarding the asymptotic behavior of these training objectives:

• Proposition 1 (Degeneracy of Deterministic Loss): It proves that for any pointwise deterministic loss (like MSE) applied to chaotic systems with mixing dynamics, the long-lead terms in the loss function increasingly penalize predictive variance. As the lead time increases, the target conditional mean converges to the climatological mean. Consequently, the optimizer minimizes loss by collapsing the forecast distribution to a single point (the Fréchet median of the invariant measure), resulting in variance collapse.
• Proposition 2 (Asymptotic Consistency of Proper Scoring Rules): It proves that strictly proper scoring rules (like the Energy Score) do not penalize spread itself. Instead, the asymptotic objective minimizes the divergence between the model's long-lead predictive distribution and the true invariant measure of the system. This ensures the model preserves the correct statistical variability.

C. Numerical Experiments

• System: Quasi-Geostrophic (QG) turbulence, a canonical chaotic system for atmospheric/oceanic dynamics.
• Setup: A high-resolution simulation serves as the "truth." A coarse-grained operator creates a low-resolution state.
• Models:
  1. Offline Deterministic: Trained on single time steps.
  2. Online Deterministic: Trained on trajectories using Euclidean distance.
  3. Online Stochastic: Trained on trajectories using the Energy Score with a generative architecture.
• Data: 100 years of data for training and 100 years for validation.

3. Key Contributions

1. Identification of Variance Collapse: The paper establishes that training deterministic models over long trajectories using pointwise losses is mathematically degenerate. It forces the model to dampen natural variability to minimize error, leading to "over-smoothed" solutions.
2. Necessity of Stochasticity: It demonstrates that stochasticity is not merely an optional feature for uncertainty quantification but a structural requirement for faithfully representing coarse-grained chaotic dynamics.
3. Role of Proper Scoring Rules: The authors show that strictly proper scoring rules (specifically the Energy Score) are essential to align the learning objective with the system's invariant measure, avoiding the bias against predictive spread inherent in metric-based losses.
4. Unified Framework: They present a unified generative framework where deterministic and stochastic closures are structurally identical, differing only in the inclusion of noise and the choice of loss function, isolating the impact of the training objective.

4. Results

• Finite-Time Skill: Stochastic closures trained with the Energy Score significantly outperform deterministic counterparts across all lead times. Deterministic models, even when trained online, fail to capture fine-scale variability.
• Stability: Offline-trained models and short-window online models often become numerically unstable. Stochastic closures require sufficiently long training windows to stabilize, but once stabilized, they remain robust.
• Long-Term Statistics (Kinetic Energy Spectrum):
  • Deterministic Closures: Exhibit a substantial artificial decrease in energy at small scales (over-dissipation). Even with an optimally tuned window length, they fail to recover the correct energy spectrum.
  • Stochastic Closures: Successfully reproduce the correct kinetic energy spectrum down to the dissipation scale.
• Visual Validation: Snapshots of potential vorticity show that deterministic closures produce overly smooth, laminar flows lacking coherent eddies. In contrast, stochastic closures maintain physically realistic turbulent structures and jet coherence.

5. Significance

This work fundamentally shifts the paradigm for data-driven modeling of chaotic systems:

• Beyond Ad Hoc Engineering: The shift toward probabilistic forecasting (seen in recent weather models like GenCast) is not just an engineering choice but a mathematical necessity dictated by the nature of coarse-grained dynamics.
• General Applicability: While tested on fluid dynamics, the findings apply to any reduced-order model of a chaotic system where unresolved scales induce memory and stochasticity.
• Methodological Guidance: The paper provides a clear prescription for future research: Trajectory-based calibration must be probabilistic. Using deterministic losses over trajectories is counterproductive. To build statistically faithful reduced models, one must combine stochastic architectures with strictly proper scoring rules.

In summary, the paper proves that stochasticity and probabilistic trajectory scoring are coupled, essential requirements for closing chaotic systems, preventing the systematic loss of physical variability that plagues current deterministic approaches.