Panda: A pretrained forecast model for chaotic dynamics

The paper introduces Panda, a pretrained foundation model for chaotic dynamics that, despite being trained exclusively on synthetic low-dimensional ordinary differential equations, demonstrates emergent zero-shot forecasting capabilities for unseen chaotic systems, real-world experimental data, and even partial differential equations.

The Gist

Imagine you are trying to predict the weather. You know that if you get the temperature wrong by just a tiny fraction of a degree today, your prediction for next week could be completely off. This is the nature of chaotic systems: they are incredibly sensitive to small errors. Whether it's the flutter of a butterfly's wings, the flow of blood in your veins, or the movement of a double pendulum, these systems are notoriously hard to forecast.

For a long time, scientists had two main ways to try and predict these systems:

1. The Specialist: Train a specific model on just one system (like just the weather) and hope it works for that specific case.
2. The Generalist: Train a massive AI on tons of data, but without really understanding the underlying physics, so it just memorizes patterns rather than learning the rules.

Enter PANDA (Patched Attention for Nonlinear DynAmics), a new AI model that tries to do something smarter. Here is how it works, explained simply:

1. The "Evolutionary" Training Camp

Instead of just feeding the AI existing data, the researchers created a digital petri dish.

• The Parents: They started with 129 famous chaotic systems (like the Lorenz attractor, which looks like a butterfly shape).
• The Mutation: They took these systems and randomly tweaked their settings, like changing the weight on a pendulum or the speed of a fluid.
• The Mating: They "mated" these systems together, combining their equations to create entirely new, never-before-seen chaotic systems.
• The Result: They generated 20,000 unique chaotic systems. They then trained PANDA on this massive, synthetic playground.

The Analogy: Imagine teaching a child to recognize animals. Instead of showing them photos of just cats and dogs, you generate millions of hybrid creatures (cat-dogs, dog-birds) and teach the child to understand the rules of anatomy. When you finally show them a real, unknown animal, they can figure it out because they understand the underlying logic, not just the pictures.

2. The "Patchwork" Brain

Most AI models look at time series data (like a stock chart) one point at a time. PANDA looks at the data in chunks, or "patches."

• The Metaphor: Think of a movie. A standard AI looks at one frame at a time. PANDA looks at a 16-second clip at a time. By looking at the whole clip, it can see the flow and the shape of the movement, not just the individual dots.
• The "Channel" Connection: In chaotic systems, different variables (like temperature and pressure) are deeply connected. PANDA has a special "attention" mechanism that lets it look at how these variables talk to each other, rather than treating them as separate lists of numbers.

3. The Magic Tricks (Emergent Abilities)

After training only on simple, low-dimensional math equations (like a 3D pendulum), PANDA started doing things the researchers didn't explicitly teach it to do:

• The "Zero-Shot" Superpower: PANDA can look at a system it has never seen before (like a specific electronic circuit or the movement of a worm) and predict its future with high accuracy. It didn't need to be retrained; it just applied the rules it learned in the training camp.
• The "Dimensional" Leap: This is the coolest part. PANDA was trained only on simple 3D systems. But when asked to predict Partial Differential Equations (PDEs)—which describe complex, high-dimensional things like fluid turbulence or flame fronts—it succeeded!
  • Analogy: It's like teaching a child to ride a tricycle, and then watching them hop on a motorcycle and ride it perfectly without ever having seen one. The model learned the essence of chaos, which applies to everything from a simple pendulum to a swirling storm.
• The "Resonance" Discovery: When the researchers analyzed the AI's internal "brain" (its attention maps), they found it was developing complex patterns that look like nonlinear resonance. This is a deep physics concept where a system vibrates in complex ways based on input frequencies. The AI "invented" this physics concept on its own just by trying to predict the future.

4. Why This Matters

Usually, to predict a complex system, you need a massive amount of data specific to that system. PANDA shows that if you train a model on a diverse enough set of synthetic chaos, it learns the fundamental "grammar" of how chaotic systems behave.

• The Scaling Law: The researchers found that the more different types of chaotic systems they trained the model on, the better it got at predicting new systems. It's not about seeing more data of the same thing; it's about seeing more variety.

Summary

PANDA is like a student who didn't just memorize the answers to a math test but learned the fundamental laws of physics so well that they can solve problems in a subject they've never studied before. By training on a vast, artificially evolved universe of chaotic systems, it learned to predict the unpredictable, from the wobble of a worm to the swirl of a storm, all without needing a specific textbook for each new challenge.

Technical Summary

Here is a detailed technical summary of the paper "PANDA: A PRETRAINED FORECAST MODEL FOR CHAOTIC DYNAMICS" (ICLR 2026).

1. Problem Statement

Predicting chaotic dynamical systems (e.g., fluid flows, neuronal activity, weather) is a fundamental challenge in scientific machine learning (SciML). These systems are intrinsically sensitive to initial conditions, causing small errors to grow exponentially, which precludes long-term point-wise forecasting.

• Limitations of Current Approaches:
  • Specialized Models: Existing methods often train on a single time series to learn a local propagator. These fail at out-of-domain generalization (predicting unseen systems).
  • Foundation Models: General time-series foundation models (e.g., Chronos, TimesFM) are trained on vast databases but often lack the underlying dynamical structure, leading to "parroting" of motifs rather than learning the governing physics.
  • Data Scarcity: There is a lack of large-scale, diverse datasets of chaotic systems required to train global forecast models that can generalize across different dynamical regimes.

2. Methodology

A. Dataset Generation: Evolutionary Discovery of Chaotic ODEs

To address the data scarcity, the authors created a novel, extensible dataset of $2 \times 10^4$ chaotic Ordinary Differential Equations (ODEs) using an evolutionary algorithm.

1. Founding Population: Started with 129 human-curated chaotic systems (e.g., Lorenz attractor, double pendulum).
2. Mutation: Added Gaussian noise to system parameters.
3. Recombination: Used skew-product coupling to combine two systems ($f_a$ as driver, $f_b$ as response):
   $$\dot{x} = f_a(x, t)$$
   $$\dot{y} = \kappa_b f_b(y, t) + \kappa_a f_a(x, t)$$
   This asymmetric coupling preserves chaoticity.
4. Selection: Integrated trajectories and filtered them using a suite of dynamical tests:
   • Rejection of fixed points or divergence.
   • 0-1 Test for Chaos: Distinguishes chaos from quasiperiodicity.
   • Lyapunov Exponent: Ensures positive maximum Lyapunov exponent.
   • Stationarity Tests: KPSS and ADF tests.
5. Augmentation: Applied dynamics-preserving transformations (random time-delay embedding per Takens' theorem, convex combinations, affine transforms).

B. Model Architecture: PANDA

PANDA (Patched Attention for Nonlinear DynAmics) is an encoder-only Transformer designed specifically for multivariate dynamical systems.

• Patching: Tokenizes time series into patches (stride = patch size = 16) to leverage Takens' Embedding Theorem, which suggests that time-delayed copies preserve topological features of the attractor.
• Dynamics-Informed Embedding: Unlike standard positional encodings, PANDA lifts patches into a higher-dimensional space by concatenating:
  • The raw patch.
  • Random Polynomial Features: Motivated by Extended Dynamic Mode Decomposition (eDMD) and Koopman operator theory.
  • Random Fourier Features.
• Attention Mechanisms:
  • Temporal Attention: Mixes information across time steps using Rotary Positional Encoding (p-RoPE).
  • Channel Attention: Crucial for capturing strong channel coupling inherent in dynamical systems. The model interleaves channel attention layers (transposing tokens to treat channels as the sequence dimension) to model functional dependencies between variables.
• Training Objectives:
  • Forecasting: Predict future states (fixed horizon).
  • Masked Language Modeling (MLM): Reconstruct masked patches to learn dynamical continuity and cross-channel dependencies.

3. Key Contributions

1. Novel Dataset: A framework for algorithmically discovering $\sim 20,000$ unique chaotic ODEs via evolutionary recombination, providing a rich, diverse training ground for out-of-domain generalization.
2. Global Forecast Model: Pretraining a global model purely on simulated chaotic data, achieving zero-shot forecasting on unseen systems (including real-world experimental data) without fine-tuning.
3. Architectural Innovations: Demonstrated the necessity of Channel Attention for dynamical coupling and Polynomial/ Fourier embeddings for capturing nonlinear dynamics, outperforming univariate baselines.
4. Emergent Capabilities:
   • Zero-shot PDE Forecasting: Despite being trained only on low-dimensional ODEs, PANDA spontaneously generalizes to predict high-dimensional Partial Differential Equations (PDEs) like the Kuramoto-Sivashinsky equation and Von Kármán vortex streets.
   • Neural Scaling Law for Dynamics: Showed that performance scales with the diversity of unique dynamical systems encountered, rather than just the total number of time steps.

4. Results

• Zero-Shot Performance: On a held-out set of 9,300 unseen systems, PANDA (21M parameters) outperformed state-of-the-art time-series foundation models (Chronos, TimesFM, TimeMoE) and specialized models (DynaMix) across multiple metrics (sMAPE, MAE, KL divergence, Spectral Hellinger distance).
• Experimental Data: Successfully forecasted real-world data from:
  • Double pendulum motion.
  • C. elegans worm body posture.
  • Networks of coupled electronic oscillators.
  • Key Finding: PANDA's advantage over baselines increased as the coupling strength between variables increased, validating the efficacy of channel attention.
• Scaling Laws: Performance improved significantly as the number of unique dynamical systems in the training set increased (holding total timepoints constant), confirming that dynamical diversity is a critical factor for generalization.
• PDE Generalization: PANDA outperformed fully trained Fourier Neural Operators (FNO) and DeepONets on zero-shot PDE forecasting, capturing complex nonlinear phenomena (e.g., flame front merging, vortex pinchoff) without PDE-specific training.
• Long-Term Behavior: While point-wise error diverges (as expected in chaos), PANDA maintained accurate invariant measures (Lyapunov exponents, correlation dimension, attractor geometry) over horizons 8x longer than the training horizon.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the notion that chaotic systems cannot be forecasted globally. It demonstrates that a foundation model trained on diverse synthetic dynamics can learn the "language" of chaos, enabling transfer to unseen physical systems.
• Scientific Insight: The emergence of PDE prediction from ODE training suggests that the model learns abstract mathematical structures (topology of attractors) rather than specific equations.
• Future Directions: The work highlights the potential of "dynamics-informed" pretraining for SciML. It suggests that low-dimensional chaotic systems serve as the fundamental building blocks for understanding complex, high-dimensional phenomena like weather and turbulence.
• Limitations: The model currently focuses on low-dimensional dynamics; extending to high-dimensional sparse systems requires architectural adaptations (e.g., custom attention masks). Additionally, Masked LM pretraining was found to slightly degrade autoregressive rollout performance, indicating a need for further research into optimal pretraining tasks for dynamics.

In summary, PANDA establishes a new benchmark for forecasting chaotic systems, proving that pretrained models can generalize across dynamical domains, capture invariant physical properties, and even bridge the gap between ODEs and PDEs.