🧬 biology

Learning the Koopman Operator using Attention Free Transformers

This paper introduces a robust Koopman predictor that combines an attention-free latent memory block for local temporal context and dynamic re-encoding mechanisms to correct latent drift, achieving superior long-horizon accuracy and lower inference latency compared to existing autoencoder and attention-based models.

Original authors: Mohammed Nagdi, Evangelos-Marios Nikolados, Alexey Yermakov, Mars Gao, Nathan Kutz, Filippo Menolascina

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohammed Nagdi, Evangelos-Marios Nikolados, Alexey Yermakov, Mars Gao, Nathan Kutz, Filippo Menolascina

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The Big Problem: The "Drifting" GPS

Imagine you are trying to predict the path of a rollercoaster or a swinging pendulum using a computer. Scientists use a mathematical tool called the Koopman Operator to turn these complex, wiggly, non-linear movements into simple, straight-line predictions. It's like trying to predict the path of a car by pretending it only drives on a perfectly flat, straight highway.

For a short trip (a few seconds), this works great. But if you try to predict the ride for a long time (hours or days), the prediction starts to "drift." The computer thinks the car is still on the straight highway, but in reality, the car has veered off a cliff or is spinning in a loop. The math gets slightly wrong at every step, and those tiny errors pile up until the prediction is completely useless.

The paper calls this drift. It happens because the "straight highway" (the learned model) isn't a perfect map of the real, bumpy world.

The Solution: Two New Tools

The authors, a team from the University of Edinburgh and the University of Washington, introduced two new "safety features" to fix this drifting problem. They added these to the standard Koopman model to make it robust enough to predict for thousands of steps without losing its way.

1. The "Short-Term Memory" (AFT Block)

The Analogy: Imagine you are walking through a foggy forest. If you only look at the ground directly in front of your feet, you might trip over a root you didn't see coming. But if you glance back at the last few steps you took, you can feel the slope of the path and adjust your balance before you fall.

The Tech: The authors added a component called Attention-Free Latent Memory (AFT).

What it does: Before the computer makes its next prediction, it looks at a short "window" of the last few steps it just took. It uses this history to correct the current position slightly, fixing small errors before they get bigger.
Why it's special: Usually, looking at the past requires a very heavy, slow computer process (like a Transformer with "attention"). This new method is "attention-free," meaning it does the same job but is much faster and uses very little memory (only about 30,000 extra parameters). It's like having a GPS that remembers the last 10 turns without needing a supercomputer to calculate them.

2. The "Reality Check" (Dynamic Re-encoding)

The Analogy: Imagine you are navigating a ship. Even with a good map, you might drift off course due to wind or currents. A smart captain doesn't just keep steering; they occasionally stop, look at the stars (the real world), and say, "Wait, we aren't where the map says we are." They then snap the ship's position back to the correct location on the map before continuing.

The Tech: This is called Dynamic Re-encoding.

What it does: The system constantly monitors itself. It asks, "Is my prediction starting to look weird compared to what the model should look like?" It uses simple, fast statistical alarms (like a smoke detector) to spot when the prediction is drifting off the "safe path" (the manifold).
The Fix: If the alarm goes off, the system instantly "snaps" the prediction back onto the correct path and continues. This prevents the small errors from turning into a catastrophic failure.

How They Tested It

The team tested these tools on three very different "rollercoasters":

The Duffing Oscillator: A system that can switch between two different states (like a ball rolling between two valleys). It's chaotic and hard to predict.
The Repressilator: A synthetic gene circuit that acts like a perfect, rhythmic clock (a limit cycle).
IRMA: A complex network of five genes interacting with each other, like a tangled web of feedback loops.

The Results

Better Accuracy: On all three systems, the new method (Koopman + AFT + Reality Check) made far fewer mistakes over long periods than the old methods.
Beating the Giants: They compared their method to massive, complex AI models (like Transformers and GRUs). Even though those models are huge and slow, they still drifted more than the new, lightweight Koopman method.
Speed: The new method is incredibly fast. It can predict 1,000 steps into the future in a fraction of the time it takes the other models.

The Takeaway

The paper shows that you don't need a giant, slow AI to predict complex systems for a long time. Instead, you can take a simple, fast linear model and give it two things:

A short-term memory to smooth out small bumps.
A self-check mechanism to snap back to reality if it starts to wander.

This creates a predictor that is fast, small, and stays on track for a very long time, making it perfect for systems where you need to know what happens next without waiting for a supercomputer to crunch the numbers.

Technical Summary: Learning the Koopman Operator Using Attention Free Transformers

Problem Statement

Learning Koopman operators via autoencoders (KAE) offers a principled approach to analyzing nonlinear dynamics by lifting states into a latent space where evolution is linear. While KAEs often achieve high single-step accuracy, they frequently suffer from error accumulation during long-horizon rollouts. Specifically, trajectories tend to drift off the learned manifold, leading to phase slips in oscillators, amplitude decay or explosion, and divergence from attractors. These failures are particularly pronounced in systems with:

Continuous or mixed spectra (e.g., undamped oscillators).
Switching between metastable basins.
Transient regimes where small errors compound rapidly.

Existing solutions often rely on multi-head attention (MHA) to capture temporal context, but MHA incurs quadratic computational costs ( $O(T^2)$ ) and adds significant parameter overhead. Furthermore, standard KAEs lack mechanisms to correct for drift once it begins, leading to catastrophic failure over extended time horizons.

Methodology

The authors propose augmenting the standard KAE with two complementary components designed to enhance robustness while maintaining computational efficiency: Attention-Free Latent Memory (AFT) and Dynamic Re-encoding.

1. Attention-Free Latent Memory (AFT)

To address local phase and amplitude drift, the authors integrate an AFT block that aggregates a short window of past latents ( $H_t = [z_{t-T}, \dots, z_{t-1}]$ ) to produce a corrected latent state ( $\tilde{z}_{t-1}$ ) before each Koopman update.

Mechanism: Unlike MHA, AFT operates in linear time ($O(Td)$) and adds minimal parameters ( $\approx 3d^2 + T^2$ , roughly 30k parameters for typical settings). It utilizes learned linear maps ( $W_Q, W_K, W_V$ ) and positional biases to compute attention weights via element-wise operations, avoiding the quadratic cost of dot-product attention.
Function: The corrected latent $\tilde{z}_{t-1}$ is fed into the Koopman operator $K$ ( $z_t = K \tilde{z}_{t-1}$ ). This acts as a local correction step, suppressing error divergence before it propagates through the linear dynamics.
Variants: The block can be applied as a direct replacement for the latent state or in a residual form ( $\tilde{z}_{t-1} = z_{t-1} + \text{AFT}(H_t)$ ).

2. Dynamic Re-encoding (E–D–E Projection)

To bound accumulated drift, the authors introduce a mechanism to periodically project predictions back onto the learned autoencoder manifold.

Drift Detection: The system computes a "drift proxy" $\delta_t$ by comparing two one-step predictions: one from the raw latent ( $z_{pred} = K \tilde{z}_{t-1}$ ) and one from the re-encoded latent ( $z_{re-pred} = K \phi(\phi^{-1}(\tilde{z}_{t-1}))$ ). A large discrepancy indicates the trajectory is leaving the manifold.
Streaming Triggers: Lightweight online change-point detectors monitor $\delta_t$ $δ_{t}$ . The paper evaluates four trigger families:
- Windowed Z-score (mean + std).
- Exponentially Weighted Moving Average (EWMA).
- CUSUM (Cumulative Sum).
- Sequential Two-Sample tests (non-parametric).
Correction: When a trigger fires, the latent state is snapped back to the manifold via an Encode–Decode–Encode (E–D–E) projection ( $z_{t-1} \leftarrow \phi(\phi^{-1}(\tilde{z}_{t-1}))$ ) before the next Koopman step. This mechanism is applied only during inference.

Key Contributions

Attention-Free Latent Memory for Koopman Prediction: A linear-time, low-overhead block that aggregates short latent histories to correct local errors before propagation. It outperforms matched-capacity MHA in accuracy and scalability.
Dynamic Re-encoding via Streaming Detection: An inference-time strategy using lightweight statistical triggers (EWMA, CUSUM, two-sample tests) to detect latent drift and project predictions back to the learned manifold, preventing catastrophic divergence.
Comprehensive Evaluation: Extensive testing on three representative systems (Duffing oscillator, Repressilator, IRMA) and additional benchmarks (pendulum, Rössler, fluid flow). The study includes ablations on operator size, context length, and trigger policies, comparing against GRU and Transformer autoencoders.

Results

The proposed model (Koopman + AFT, with optional re-encoding) consistently reduces error accumulation compared to baseline KAEs and matched-capacity MHA models across all tested systems.

Duffing Oscillator (Switching/Mixed Spectrum): Dynamic re-encoding significantly outperforms AFT alone and periodic re-encoding, achieving the lowest Mean Squared Error (MSE) at 200 and 500 steps. It effectively handles switching transitions between potential wells.
Repressilator (Clean Limit Cycle): AFT alone is the superior variant. Dynamic re-encoding degrades performance here, as unnecessary phase resets introduced by triggers can harm smooth oscillations. AFT achieves MSE values $\approx 5-6\times$ lower than MHA.
IRMA (High-Dimensional Feedback): Dynamic re-encoding provides the most robust performance, preventing the divergence seen in vanilla KAEs (which fail at 1000 steps). It maintains low error accumulation where AFT alone begins to degrade.
Comparison to RNNs/Transformers: While GRU and Transformer autoencoders benefit from context, they underperform on long horizons compared to the Koopman-based approach. The Koopman+AFT model achieves markedly lower long-horizon error with significantly lower inference latency (e.g., ~13ms vs. ~491ms for Transformers on IRMA).
Efficiency: The AFT block adds only $\approx 30k$ parameters and maintains linear complexity, making it suitable for embedded monitoring and rapid simulation.

Significance and Claims

The paper claims to provide a fast, compact predictor that maintains stability over long horizons by staying on the learned manifold. The significance lies in:

Pragmatic Robustness: It offers a practical alternative to complex attention mechanisms, achieving better long-horizon fidelity with linear computational costs.
Orthogonal Mechanisms: The combination of AFT (preventing local error growth) and re-encoding (bounding accumulated drift) addresses different failure modes, allowing the model to adapt to both smooth limit cycles and complex switching dynamics.
Inference-Time Adaptability: The use of streaming triggers allows the model to adapt to drift without retraining, keeping the core Koopman operator and autoencoder weights fixed.

The authors note that their stability claims are empirical; they do not provide theoretical convergence guarantees for the learned operator. They acknowledge that trigger policies introduce hyperparameters and can be detrimental on clean limit cycles if not tuned correctly. The work focuses on autonomous systems, leaving inputs and control as future directions.