Learning Transferable Predictability Representations

Technical Summary: Learning Transferable Predictability Representations

Problem Formulation

The paper addresses the challenge of assigning a scalar score to a short trajectory window that reflects its position on an ordered continuum of predictability regimes. While existing methods effectively discriminate between deterministic and stochastic dynamics within a single system, they fail to produce scores with consistent numerical interpretations across different systems.

The authors formalize this as an ordinal estimation problem over a five-level "predictability ladder" ($L_0$ to $L_4$):

1. $L_0$ (Stable Deterministic): Fixed points or limit cycles ($\lambda_{max} \le 0$).
2. $L_1$ (Weakly Chaotic): Deterministic with small positive instability ($0 < \lambda_{max}\tau \ll 1$).
3. $L_2$ (Strongly Chaotic): Deterministic with rapid divergence ($\lambda_{max}\tau \gg 0$).
4. $L_3$ (Structured Stochastic): Surrogates preserving linear statistics (e.g., power spectrum) but disrupting nonlinear geometry.
5. $L_4$ (Unstructured Stochastic): i.i.d. noise.

A critical structural barrier identified is gauge freedom: ordinal supervision alone determines the ordering of scores but leaves the numerical coordinate unfixed up to a monotone reparameterization. Consequently, scores trained on one system are not directly comparable to those on another, even if the ordinal ranking is correct. The paper argues that cross-system transfer requires fixing this coordinate ambiguity, as zero-shot scenarios lack target-system data for post-hoc recalibration.

Methodology: The Gauge-Fixed Ordinal Network (GON)

The proposed solution, GON, is a temporal convolutional model designed to learn a shared score coordinate across systems. It comprises three key components:

1. 2-Jet Preprocessing:
   Instead of using raw state coordinates (which carry system-specific scaling and units), GON operates on 2-jet features: position ($x$), velocity ($\dot{x}$), and acceleration ($\ddot{x}$). These features, estimated via Savitzky–Golay filtering and central finite differences, expose local trajectory geometry. This representation is preserved by smooth deterministic flows but disrupted by stochastic surrogate procedures, making it ideal for distinguishing deterministic structure from structured noise.

2. Architecture:
   The model uses a Temporal Convolutional Network (TCN) with dilated residual blocks to encode the 2-jet input into a hidden sequence. Multi-scale average pooling aggregates this into a vector, which is mapped to a scalar score via a two-layer MLP. Outputs are clipped using a hyperbolic tangent function to align with the target range.

3. Gauge-Fixed Ordinal Objective:
   To resolve gauge freedom, the training objective assigns fixed anchor targets to the mean scores of each ladder level: $\{t_k\} = \{-4, -2, 0, +2, +4\}$. The loss function ($\mathcal{L}$) consists of:

   • Anchor Term: Penalizes the deviation of level-wise score means ($\mu_k$) from their targets ($t_k$).
   • Variance Term: Penalizes excessive within-level spread ($\sigma_k$), encouraging tight clustering around the anchors.

   This "anchor-and-variance" objective pins the score coordinate, ensuring that a specific numerical value corresponds to a specific predictability regime across different systems.

Experimental Setup

• Training Data: 12 three-dimensional dissipative chaotic ODE systems (e.g., Lorenz-63, Rössler, Chen) covering a range of attractor geometries.
• Test Data: Five fully held-out systems (e.g., Thomas, Tigan, forced pendulum) with no data used during training.
• Evaluation Metrics:
  • Monotonicity Slope ($\beta$): Measures global ordinal coherence of level means.
  • Scale-Normalized $\beta$ ($\beta_{norm}$): Used for cross-method comparison.
  • Anchor Drift ($\delta_{anchor}$): Measures how closely learned means match target anchors.
  • Pairwise AUROC: Evaluates discrimination at specific boundaries (e.g., $L_2 \to L_3$).

Key Results

1. Zero-Shot Transfer:

   • GON achieves a scale-normalized monotonicity slope of $\beta_{norm} = 0.559$ on held-out systems, outperforming all supervised baselines (which cluster around 0.46–0.48).
   • Transfer is strongest at the stochastic boundary ($L_2 \to L_3$ and $L_3 \to L_4$), where surrogate procedures disrupt nonlinear geometry. Deterministic-side boundaries ($L_0 \to L_2$) show weaker separation, suggesting that fine-grained chaotic distinctions are more system-specific.
   • In-distribution calibration is excellent, with mean anchor drift $\delta_{anchor} = 0.0097$.

2. Adaptation Efficiency:

   • Initializing from a pretrained GON checkpoint significantly outperforms training from scratch, even with very limited labeled windows ($k=5$).
   • Full fine-tuning (pre_all) consistently beats random initialization, while freezing the encoder (pre_head) improves discrimination but fails to recover global ordinal structure ($\beta$).
   • Adaptation depth correlates with geometric proximity: systems with vector fields similar to the training set adapt immediately, while those with unseen structures (e.g., sinusoidal forcing) require more data and may fail to resolve finer boundaries.

3. Ablation Studies:

   • Anchor vs. Margin: Replacing the anchor objective with a margin loss (which enforces separation but not fixed coordinates) preserves in-distribution discrimination but causes anchor drift to rise from 0.005 to 2.556 and reduces transfer $\beta$ by 47%. This confirms that coordinate stability, not just ordinal coherence, is necessary for transfer.
   • 2-Jet Features: Using coherent 2-jet features significantly improves transfer ($\beta = 1.504$) compared to raw states ($\beta = 1.132$) or shuffled derivatives. This highlights the importance of geometric structure over mere dimensionality.

Significance and Claims

The paper claims to identify gauge-fixing as the missing ingredient for transferable ordinal predictability scores. The central contribution is demonstrating that assigning a stable numerical coordinate to ordinal levels is a necessary condition for cross-system comparability.

• Theoretical Insight: The work establishes that pairwise discrimination and globally coherent ordinal scoring are distinct properties; a stable coordinate is required to bridge them under transfer.
• Practical Utility: The framework enables predictability assessment, model selection, and early-warning diagnostics across natural and engineered systems without system-specific calibration.
• Limitations: The authors note the benchmark is currently limited to low-dimensional, dissipative ODE systems. Extensions to high-dimensional, conservative, or real-world signals remain future work. The method operates on synthetic data and involves no personal data or foreseeable negative societal impacts.

In conclusion, the paper posits that learning a transferable ordinal predictability coordinate via gauge-fixing allows for meaningful comparisons of predictability regimes across heterogeneous dynamical systems, a capability previously unattainable with standard ordinal regression or classification approaches.