Technical Summary: In-context learning to predict critical transitions in dynamical systems

Problem Statement

Critical transitions—abrupt, often irreversible shifts in system dynamics—occur across climate, ecology, infrastructure, and biological systems. Predicting these events is challenging because real-world observations of such shifts are scarce, and the data is often noisy, short, and partially observed.

Conventional early warning signals (EWS), such as increasing variance or lag-1 autocorrelation (AR1), rely on restrictive assumptions including near-equilibrium linearization, stationarity, and sufficiently long records. These indicators often fail under realistic conditions, particularly for rate-induced (r-tipping) and noise-induced (n-tipping) transitions where critical slowing down (CSD) may be absent or weak.

From a machine learning perspective, predicting tipping points is an out-of-distribution (OOD) task. Standard deep learning classifiers trained on specific dynamical regimes often fail to generalize to unseen systems or different tipping mechanisms. Existing data-driven approaches typically require extensive retraining for new systems and struggle to extrapolate beyond their training distribution.

Methodology: TipPFN

The authors introduce TipPFN, a transformer-based architecture utilizing Prior-Data Fitted Networks (PFNs) and In-Context Learning (ICL). Unlike traditional models that fit parameters to a single dataset, PFNs are trained on millions of synthetic tasks to learn a mapping from data to a posterior predictive distribution, enabling inference via ICL.

1. Training Framework and Data Generation (TipBox)

TipPFN is trained exclusively on a novel synthetic data generator called TipBox. This generator creates diverse stochastic dynamical systems by embedding canonical tipping systems into high-dimensional, randomized nonlinear stochastic differential equations (SDEs).

• Generative Process: The prior $p(\psi)$ factorizes over a canonical tipping system $M$ (driving the bifurcation), a random interaction graph $G$, and an auxiliary nonlinear SDE system $Z$.
• Diversity: The training data includes various bifurcation types (fold, Hopf, transcritical) and is coupled with randomized stochastic dynamics and forcing schedules.
• Task Construction: The model is trained on context-query pairs. A query episode represents the system of interest, while context episodes provide additional trajectories from the same underlying generative process but with different noise realizations or forcing schedules.
• Prediction Target: Instead of predicting the binary event of a crash, the model predicts the Relative Distance to Criticality (RDTC), denoted as $\Lambda$. For bifurcation-induced tipping, $\Lambda = 1 - \tilde{\lambda}$, where $\tilde{\lambda}$ is the normalized control parameter. $\Lambda=0$ indicates the bifurcation point. This continuous target allows the model to learn the proximity to a transition rather than just the event itself.

2. Architecture and Inference

TipPFN adapts the TabPFN architecture for time-series and dynamical systems:

• Input Representation: Time series are encoded as tabular multi-episode sequences with identifiers for episodes and time steps. Variables are normalized using only observed values to prevent target leakage.
• Attention Mechanism: The transformer employs causal attention. Query tokens can attend to all context tokens and their own past/present tokens, but never future tokens. Context tokens can attend to all other context tokens bidirectionally.
• Inference Modes:
  • Nowcasting: Estimates the current RDTC given observations up to time $t$.
  • Forecasting: Predicts the future trajectory of RDTC given observations up to time $t$.
• Zero-Shot and ICL Capabilities: The model can operate in a zero-shot setting (no context episodes) relying solely on the learned prior, or leverage context episodes (sim-to-real or real-world) to refine its posterior predictive distribution.

Key Contributions

1. Model (TipPFN): A transformer-based PFN designed to infer proximity to critical transitions directly from short, noisy time series using ICL.
2. Training and Benchmark (TipBox): A comprehensive benchmarking suite and synthetic data generator covering diverse tipping regimes (b-tipping, r-tipping, n-tipping) and complexities, including high-dimensional stochastic dynamics.
3. Generalization: Demonstrated robust performance across previously unseen tipping regimes, sim-to-real cases, and real-world systems in both ICL and zero-shot settings.

Results

The authors evaluated TipPFN against classical EWS (variance, AR1, skewness) and state-of-the-art deep learning baselines (Bury, Huang, Zhuge, TabPFN) across 14 datasets spanning synthetic, semi-real, sim-to-real, and real-world systems.

• Performance: TipPFN consistently achieved state-of-the-art AUROC scores, outperforming classical indicators and specialized deep learning models. For example, on canonical bifurcation systems, TipPFN achieved AUROCs around 0.95, significantly higher than the ~0.6–0.8 range of baselines.
• Generalization to Unseen Mechanisms: Crucially, TipPFN generalized to r-tipping and n-tipping scenarios despite being trained primarily on b-tipping synthetic data. It remained competitive on Bautin, compost-bomb, and saddle-node r-tipping systems where classical EWS and other ML models often failed.
• Role of Context: Performance improved significantly when context episodes were provided. In the Daphnia extinction experiment, adding simulated or real context episodes allowed TipPFN to outperform baselines, particularly in sparse data regimes ($W=16$).
• Real-World Application: In zero-shot settings (no context), TipPFN provided early warnings for real-world systems including:
  • AMOC: Detected increased variance coinciding with an abnormal reversal in slowdown.
  • Power Grid: Signaled hidden instability 2 minutes before a blackout.
  • Microcosm: Estimated a cyanobacteria population collapse approximately a week in advance.
• Comparison to Baselines: While specialized models like Bury (trained on specific bifurcations) performed well on matching tasks, they failed to generalize to r-tipping. TipPFN maintained high performance across all regimes without system-specific retraining.

Significance and Claims

The paper claims that TipPFN represents a shift from system-specific, indicator-based prediction to a universal, system-agnostic framework for critical transition detection.

• Universal Prediction: By inferring hidden proximity to a transition rather than relying on explicit early warning indicators (which assume specific dynamical structures), TipPFN provides a universal estimate of criticality.
• Data Efficiency: The framework effectively leverages limited observational data by utilizing ICL to incorporate context from simulations or other similar systems, addressing the scarcity of real-world tipping data.
• Operational Utility: The ability to provide early predictions in zero-shot settings enables deployment in high-stakes domains (climate, energy, health) where early intervention is critical and system-specific retraining is impractical.

The authors note limitations, particularly regarding online settings where observations arrive sequentially. They suggest future work could combine ICL with near-real-time model adaptation (e.g., low-rank updates) and investigate spatial dependencies in critical transitions.