Technical Summary: Watch your neighbors: Training statistically accurate chaotic systems with local phase space information

Problem Statement

Data-driven discovery of dynamical systems faces fundamental challenges when applied to chaotic systems. While "black-box" surrogate models (such as Neural ODEs) trained via standard trajectory-based optimization can match observed short-term trajectories, they often fail to capture the underlying dynamics accurately over longer horizons. This is due to the exponential sensitivity of chaotic systems to initial conditions and model parameters; small modeling errors lead to rapidly diverging trajectories, rendering exact long-term prediction impossible.

Consequently, the community has shifted toward training models that are "statistically accurate," meaning they reproduce the long-term invariant physical measures (e.g., attractor geometry and statistical moments) of the ground truth, even if individual trajectories diverge. However, existing statistical methods often neglect the local geometric properties of the flow, specifically the Jacobian, which governs local expansion and contraction rates. Conversely, methods focusing on Jacobian accuracy often rely on oracle access to ground-truth derivatives or fail to maintain long-term statistical consistency. There is a need for a framework that simultaneously achieves accurate long-term statistical behavior and captures the local dynamical properties (Jacobian) without requiring ground-truth derivative labels.

Methodology

The authors propose a novel training framework for Neural ODEs that bridges the gap between global statistical accuracy and local dynamical fidelity. The core idea is to shift from a global perspective to a local one by analyzing how the dynamics transport and deform local neighborhoods (coverings) of the chaotic attractor.

1. Local Covering and Pushforward Measures

Instead of optimizing solely for trajectory matching or global invariant measures, the method constructs a finite cover $\{U_i\}$ of the chaotic attractor using data points. The training objective is to ensure that the Neural ODE reproduces the pushforward measure of these local sets under the ground-truth dynamics. Specifically, the method enforces that the distribution of points evolved from a neighborhood $U_i$ under the surrogate model matches the distribution of points evolved from the same neighborhood under the true dynamics.

2. Neighborhood Construction and Noise Robustness

To handle observational noise, the authors define neighborhoods not as simple balls, but as annuli with inner radius $r_{min}$ and outer radius $r_{max}$.

• $r_{min}$ is set to a multiple of the estimated noise standard deviation to avoid the "noise floor" where time evolution is obscured.
• $r_{max}$ is chosen such that each neighborhood contains approximately 5% of the total dataset points to prevent overfitting to sparse local data.
• Neighbors are identified efficiently using KD-Trees.

3. Evolution via Taylor Expansion

Rather than evolving individual neighbor points directly (which is computationally expensive and sensitive to integration errors), the method evolves perturbations relative to a center point using a Taylor expansion of the Neural ODE vector field.
The perturbation $\omega_k$ between a center point $u_j$ and its neighbor $u_{j,k}$ is evolved according to:
$$\frac{d}{dt} \begin{bmatrix} \tilde{u} \\ \tilde{\omega}_k \end{bmatrix} = \begin{bmatrix} f_\theta(\tilde{u}) \\ \partial f_\theta|_{\tilde{u}} [\tilde{\omega}_k] + \frac{1}{2}\partial^2 f_\theta|_{\tilde{u}} [\tilde{\omega}_k, \tilde{\omega}_k] \end{bmatrix}$$
This formulation explicitly incorporates the first and second-order derivatives of the learned vector field, allowing the model to learn the local deformation of the phase space.

4. Loss Function

The training objective minimizes a composite loss:
$$L(\theta) = L_{traj}(\theta) + \lambda \cdot \ell_{neighbor}$$

• $L_{traj}$: Standard trajectory-based loss (MSE) to ensure short-term accuracy.
• $\ell_{neighbor}$: A pairwise loss quantifying the discrepancy between the pushforward measures of the ground truth and the surrogate model on the evolved neighborhoods. This is computed using the Maximum Mean Discrepancy (MMD) squared.
• Theoretical justification: The authors prove that if the pushforward measures agree on a countable cover of the attractor, the global physical measures must be identical (Theorem 1).

Key Contributions

1. Novel Framework: Introduction of a training method for Neural ODEs that jointly optimizes for long-term statistical accuracy and local dynamical properties (Jacobian) by regularizing the evolution of local phase-space neighborhoods.
2. Noise-Robust Design: Development of an annular neighborhood strategy and a perturbation-based evolution scheme that effectively handles observational noise, a critical requirement for real-world applications where ground-truth derivatives are unavailable.
3. Theoretical Connection: Demonstration that enforcing local measure agreement on a cover implies global measure invariance, providing a theoretical bridge between local Jacobian learning and global statistical accuracy.
4. Practical Implementation: A fully differentiable algorithm that does not require oracle access to ground-truth Jacobians, making it applicable to real-world data.

Experimental Results

The method was evaluated on three chaotic systems of varying dimensions: the 3D Lorenz 63 system, the 4D Hyperchaotic Chen system, and the 6D Lorenz 96 system. Experiments were conducted with training data corrupted by Gaussian noise (1%, 5%, and 10% noise strength). The proposed method ("Neighborhood") was compared against:

• Vanilla: Standard trajectory-based Neural ODE training.
• DySLIM: A state-of-the-art method optimizing for global invariant measures.

Findings:

• Short-term Accuracy: The Neighborhood method maintained competitive Valid Prediction Times (VPT) with DySLIM and significantly outperformed Vanilla training, especially under high noise conditions, indicating effective regularization against overfitting.
• Long-term Statistical Accuracy: The method successfully learned the invariant measure of the attractor. While DySLIM achieved slightly higher statistical accuracy in some metrics (due to direct optimization of global measures), the Neighborhood method remained highly competitive.
• Local Dynamics (Jacobian): The Neighborhood method demonstrated substantially improved Jacobian accuracy compared to both Vanilla and DySLIM across all noise levels and systems. This was evidenced by lower relative Jacobian errors and more accurate reconstruction of vector fields, particularly in sensitive regions like the lobe-switching mechanism of the Lorenz 63 attractor.
• Lyapunov Exponents: As a direct consequence of improved Jacobian learning, the method yielded the most accurate estimates of Lyapunov exponents among the tested methods, correctly capturing both positive and negative exponents even in noisy settings.

Significance and Claims

The paper claims that the "Neighborhood" method provides a robust solution for learning chaotic dynamics from noisy data by explicitly accounting for the local phase-space structure. The authors argue that while previous methods focused on either global statistics or required oracle Jacobians, this approach successfully bridges the two paradigms.

The significance lies in the ability to train surrogate models that are not only statistically accurate (preserving the attractor's shape) but also dynamically faithful (preserving the local expansion/contraction rates). This dual capability is crucial for applications requiring an understanding of local instability, such as identifying interacting subdomains in complex systems (e.g., the brain) or estimating predictability. The authors position this work as a foundational step toward training Neural ODEs with local phase-space information, noting that future work could extend this to higher-dimensional and partially observed systems.