🤖 machine learning

DAW: Dynamics-Aware Weighting for Deep Learning Forecasts of Chaotic Systems

This paper introduces Dynamics-Aware Weighting (DAW), a data-centric framework that leverages local dynamical dimension to reweight training objectives, thereby enabling deep learning surrogates to better capture rare, complex transitions in chaotic systems and significantly reducing long-term forecast errors compared to standard and statistically reweighted baselines.

Original authors: Zhou Fang, Gianmarco Mengaldo

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhou Fang, Gianmarco Mengaldo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Predicting the future of complex systems, from the swirling patterns of a storm to the flow of fluids in a pipe, is one of the most enduring challenges in science. These systems are often chaotic, meaning that tiny differences in their starting conditions can lead to vastly different outcomes over time. For decades, scientists have relied on deep learning, a form of artificial intelligence, to act as a shortcut for these difficult calculations. These digital models learn from past data to guess what happens next, offering a faster alternative to traditional physics simulations. However, a persistent problem has limited their usefulness: when asked to predict far into the future, these models tend to accumulate errors rapidly. They often get the easy, repetitive parts of the system right but fail spectacularly when the system undergoes rare, sudden, and complex changes. This failure is not just a glitch; it stems from how the models are taught. Standard training methods treat every moment in a simulation as equally important, causing the model to focus its attention on the common, boring moments and ignore the rare, critical ones that actually drive the system's behavior.

Researchers at the National University of Singapore have developed a new approach to fix this imbalance, a method they call Dynamics-Aware Weighting. Instead of treating all data points equally, this technique teaches the model to pay extra attention to the moments when the system is most complex and difficult to predict. The team focused on a specific mathematical model known as the Kuramoto-Sivashinsky equation, which describes how waves form and interact in a fluid. In this system, the fluid spends most of its time in a calm, predictable state, but occasionally undergoes dramatic events where waves crash into each other and merge. These merging events are rare, but they are also the moments where prediction errors tend to explode. The researchers realized that the standard way of training models was blind to this distinction. By using a concept from dynamical systems theory called the "local dimension," they created a way to measure the complexity of the system at any given moment. This measure acts like a complexity score: a low score indicates a simple, repetitive state, while a high score signals a rare, intricate event where many variables are changing at once.

The new method uses this complexity score to reshape how the model learns. During training, the researchers assigned higher importance to the data points with high complexity scores. This forced the neural network to dedicate more of its learning capacity to understanding those rare, difficult moments rather than just memorizing the common, easy ones. To test if this approach worked, the team compared their method against standard training techniques and other existing methods that tried to fix the problem by simply looking at how often certain outcomes appeared. The results were clear. When the models were asked to predict the future of the fluid system over long periods, the model trained with the new weighting method stayed accurate for much longer. It maintained a strong match with the true physical patterns of the fluid, whereas the other models quickly fell apart, losing their ability to predict the shape and movement of the waves.

The study showed that the improvement was not just a general benefit of using different numbers in the training process. When the researchers shuffled the importance scores so they no longer matched the actual complexity of the system, the model's performance dropped significantly. This proved that the key was not just giving some data more weight, but specifically giving more weight to the moments when the system was dynamically complex. The analysis revealed that the new method worked by suppressing the sudden spikes in error that usually occur during those rare wave-merging events. By learning to handle these difficult transitions better, the model prevented small mistakes from snowballing into total failure. The researchers also tested their method on a different, simpler chaotic system known as the Lorenz-63 model, which describes atmospheric convection. There, too, the new method extended the time the model could predict accurately, suggesting that this approach works across different types of chaotic systems, not just the one they first studied.

This work highlights a shift in how scientists might approach machine learning for physical systems. Rather than just building bigger or more complex neural networks, the focus can shift to how the data is presented to the model. By recognizing that some moments in a system are inherently more important for long-term accuracy than others, and teaching the model to respect that hierarchy, researchers can significantly improve the reliability of their forecasts. The method is designed to be a plug-and-play addition to existing tools, requiring no changes to the underlying architecture of the models themselves. It offers a way to make deep learning surrogates more robust, allowing them to navigate the treacherous waters of chaotic systems with greater confidence and for longer periods. While the intrinsic unpredictability of chaos cannot be eliminated, this approach pushes back the point at which predictions become useless, providing a clearer, more reliable window into the future of complex physical phenomena.

Technical Summary: Dynamics-Aware Weighting (DAW) for Deep Learning Forecasts of Chaotic Systems

1. Problem Statement

Deep learning (DL) surrogates have demonstrated success in modeling complex dynamical systems but suffer from catastrophic error accumulation during long-term autoregressive rollouts. This failure is attributed to the uneven visitation of phase space in chaotic systems, where trajectories spend the majority of time in recurrent, low-dimensional quiescent states (e.g., near-laminar flows) and only rarely visit sparse, high-complexity regimes (e.g., wave-merging events or topological transitions).

Standard training objectives, such as the sample-wise uniform Mean Squared Error (MSE), implicitly treat all states as equally informative. Consequently, neural surrogates allocate their finite representational capacity to the statistically dominant low-dimensional bulk, systematically under-representing the rare, high-dimensional transient regimes. This leads to a specific instance of imbalanced regression, where the model minimizes average error by fitting the majority class while failing on the sparse regions that trigger disproportionate, localized forecast errors.

Existing solutions in deep imbalanced regression typically reweight samples based on statistical target-space density (e.g., using Kernel Density Estimation on the target values). However, the paper argues that statistical rarity in the target space does not necessarily coincide with intrinsic dynamical rarity—the geometric structure of the attractor that drives the imbalance. There is a lack of dynamically meaningful indicators to describe this intrinsic imbalance and a principled, data-centric scheme to exploit them.

2. Methodology: Dynamics-Aware Weighting (DAW)

The authors propose Dynamics-Aware Weighting (DAW), a data-centric framework that reshapes the loss landscape to prioritize dynamically complex states. The core of DAW is the use of the local dimension ( $d$ ) from dynamical systems theory as an a priori measure of a state's active degrees of freedom and complexity.

2.1 Theoretical Foundation: Local Dimension ( $d$ )

The local dimension $d_\mu(x)$ quantifies how the invariant measure scales within a small neighborhood of a state $x$ .

Low $d$ : Corresponds to geometrically dense, frequently revisited regions with few active degrees of freedom (simple dynamics).
High $d$ : Corresponds to sparse, rarely visited regions with many active degrees of freedom (complex dynamics, e.g., wave-merging).
Estimation: $d$ is estimated using Poincaré recurrence and Extreme Value Theory (EVT). Specifically, the negative logarithmic distance of trajectory points to a reference state is modeled using a Generalized Pareto Distribution (GPD). The local dimension is defined as the inverse of the GPD scale parameter ( $d = 1/\sigma$ ).

2.2 The DAW Framework

DAW constructs sample weights through a four-step process:

Density Estimation: Compute the local dimension $d_i$ for each training sample (input-output pair) and estimate the empirical probability density function (PDF) $P(d)$ using Kernel Density Estimation (KDE).
Base Weighting: Assign a base weight $w_{base}$ that decreases affinely with the normalized density $P'(d)$ , up-weighting rare states (similar to the DenseWeight baseline).
Dynamical Reshaping (The "Tilt"): To ensure weights align with dynamical complexity rather than just statistical rarity, the base weight is multiplied by the normalized local dimension $\tilde{d}_i$ . This suppresses weights for infrequent but simple (low- $d$ ) states and amplifies weights for rare, complex (high- $d$ ) states.
$w_{reshaped} = \max(w_{base} \cdot \tilde{d}, \epsilon)$
Global Normalization: Weights are normalized so their expectation over the dataset equals 1.0 to preserve the global gradient scale.

The final objective is a weighted MSE:
$L_{DAW}(\theta) = \frac{1}{N} \sum_{i=1}^N w_i \|\hat{f}_\theta(x_i) - y_i\|_2^2$

2.3 Theoretical Motivation

The paper provides a theoretical justification based on error propagation. The forecast error at step $n$ is decomposed into one-step model errors ( $\tilde{\varepsilon}_k$ ) amplified by the system's dynamics (the cocycle $M_{k \to n}$ ).

Empirical evidence suggests one-step errors are systematically larger at high- $d$ states.
Uniform training minimizes the average error, effectively smoothing over high- $d$ excursions.
DAW acts by reducing the one-step error $\|\tilde{\varepsilon}_k\|$ specifically at high- $d$ states. Since the amplification factor is determined by the true dynamics (independent of the model), reducing the injected error at these critical points shrinks the terms in the error sum that the dynamics will subsequently amplify, thereby delaying the collapse of spatial correlation.

3. Experimental Setup

Datasets:
- Kuramoto-Sivashinsky (KS) Equation: A 1D spatiotemporal chaotic system ( $L=22$ ) exhibiting quiescent regimes, traveling waves, and wave-merging events.
- Lorenz-63 System: A low-dimensional chaotic attractor used to validate generalizability.
Model: A Multilayer Perceptron (MLP) with 6 hidden layers and 128 neurons.
Baselines:
- Standard: Uniform weighting ( $w_i = 1$ ).
- DenseWeight: State-of-the-art imbalanced regression reweighting based on target-space statistical density.
- RandomWeight: An ablation where DAW's weights are randomly permuted to test if the gain comes from weight magnitude or alignment with dynamics.

4. Key Results

Evaluated on the KS equation over autoregressive rollouts up to 1.0 Lyapunov Time (LT):

Long-Term Forecasting Accuracy: DAW consistently outperforms all baselines.
- At 1.0 LT, DAW achieves the lowest Mean Absolute Error (MAE) and highest preserved spatial Pearson correlation.
- DAW delays the "spatial correlation collapse" (crossing the 0.5 predictability threshold) to approximately 0.85 LT, compared to ~0.5 LT for Standard and ~0.6 LT for RandomWeight.
One-Step Error Analysis:
- One-step error increases monotonically with local dimension $d$ , validating the premise that high- $d$ states are the source of large errors.
- DAW reduces error across the entire range of $d$ , including the low- $d$ bulk, rather than just the high- $d$ tail.
Event-Level Analysis:
- During high- $d$ events (e.g., wave-merging), DAW systematically suppresses localized error jumps.
- In a win-rate analysis across 21,146 high- $d$ event windows, DAW achieved the lowest cumulative error in 42.4% of cases, significantly outperforming DenseWeight (24.4%), RandomWeight (28.1%), and Standard (5.1%).
Generalizability:
- On the Lorenz-63 system, DAW similarly delayed correlation collapse from ~2.3 LT (Standard) to ~4.1 LT, confirming the method's applicability beyond the KS geometry.
Ablation Studies:
- RandomWeight: Performs better than Standard but worse than DAW, proving that the alignment of weights with dynamical complexity (not just non-uniform weighting) is the source of the improvement.
- DenseWeight: Improves over Standard but degrades faster than DAW at long horizons, indicating that target-space statistical rarity is an insufficient proxy for dynamical complexity.
- Input vs. Target $d$ : Using local dimension computed on inputs ( $d(x)$ ) or input-output pairs ( $d(x,y)$ ) yields similar results, with $d(x)$ being slightly more robust.

5. Significance and Claims

The paper claims that DAW addresses a fundamental gap in deep learning for chaotic systems: the absence of a principled, data-centric scheme that leverages the intrinsic geometry of the attractor to handle imbalanced regression.

Shift in Paradigm: Unlike recent Scientific Machine Learning (SciML) advances that focus on model architecture or physics-informed constraints, DAW reframes the problem as data-centric distribution reshaping. It uses the local dimension $d$ as a model-free, a priori proxy for dynamical complexity.
Mechanism of Action: The significance lies in the mechanism: DAW forces the network to allocate representational capacity to the sparse, high- $d$ regimes where forecast errors are systematically large. By suppressing the one-step errors at these critical junctures, it prevents the system's inherent dynamics from amplifying them into catastrophic long-term failures.
Practicality: DAW is a "plug-and-play" framework. Since $d$ is computed offline from historical trajectories, it introduces no architectural changes, no inference overhead, and leaves the surrogate lightweight.
Modesty: The authors explicitly state that DAW does not eliminate the intrinsic limits of chaotic predictability (Lyapunov instability) but systematically delays the loss of spatial coherence and improves long-term accuracy relative to uniform training and purely statistical weighting methods.

The work concludes that aligning representational capacity with local dynamical complexity is a more effective strategy for long-term forecasting than relying on statistical target-space rarity or uniform objectives.