⚛️ quantum physics

Attention in Krylov Space: Transformer-Based Extrapolation of Lanczos Coefficients

This paper introduces a transformer-based model that autoregressively predicts Lanczos coefficients from short prefixes, outperforming traditional asymptotic fits in accuracy and enabling zero-shot transfer across system sizes to efficiently probe operator dynamics in Krylov space.

Original authors: Zihao Qi, Christopher Earls

Published 2026-08-03

📖 3 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihao Qi, Christopher Earls

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future of a chaotic system, like a swirling galaxy or a bouncing ball in a room full of mirrors. In the world of quantum physics, scientists use a special mathematical tool called the "Lanczos algorithm" to track how information spreads through these systems. Think of this algorithm as a long, winding staircase where each step represents a moment in time. To know what happens at the top of the stairs (the far future), you need to know the height of every single step below it. These step heights are called "Lanczos coefficients."

However, there's a catch. Calculating these steps is like trying to count grains of sand on a beach while the tide is coming in; the math gets messy, the numbers get huge, and eventually, the computer runs out of memory or gets confused by tiny rounding errors. For a long time, scientists have tried to guess the rest of the staircase by drawing a simple straight line through the first few steps they could calculate. But this is like trying to predict the rest of a rollercoaster ride just by looking at the first hill—it misses all the twists, turns, and sudden drops that make the ride what it is. These missing details are crucial because they determine how the system actually behaves.

Now, imagine if you could teach a computer to look at those first few steps and "feel" the rhythm of the entire staircase, predicting the rest with incredible accuracy. That is exactly what this paper does. The researchers, Zihao Qi and Christopher Earls, treated the sequence of Lanczos coefficients not as a simple math problem, but as a story with a beginning, middle, and end. They used a type of artificial intelligence called a "Transformer"—the same kind of technology that powers advanced language models—to read the early part of the sequence and write the rest.

Instead of forcing the data into a boring straight line, their AI model learned to spot subtle, hidden patterns, like a secret code written in the way the steps wiggle up and down. They tested this on three different types of systems: a chaotic spinning top, a chaotic quantum magnet, and a perfectly ordered (integrable) quantum chain. In every case, the AI didn't just guess; it outperformed the old straight-line method by a huge margin, often reducing the error by ten times. Even more surprisingly, they trained the AI on small, easy-to-calculate systems, and it successfully predicted the behavior of much larger, harder-to-calculate systems without needing any extra training. It's as if you taught a child to recognize the pattern of a small drumbeat, and they could instantly predict the rhythm of a massive, complex orchestra.

The researchers also peeked inside the AI's "brain" to see how it worked. They found that the model didn't just look at the most recent step to guess the next one. Instead, it paid attention to the very first steps of the sequence (like an anchor) and the most recent ones simultaneously, while also noticing a specific "even-odd" wobble in the steps that the old straight-line method completely ignored. This suggests that the history of the system is deeply connected to its future in a way that simple math formulas can't capture. By using this AI approach, scientists can now explore the "deep future" of quantum systems—places where brute-force calculations used to be impossible—opening up new ways to understand how information scrambles and evolves in the quantum world.

Technical Summary: Attention in Krylov Space

Problem Statement
The study of operator growth in quantum and classical many-body systems relies heavily on the Lanczos formalism, which maps Heisenberg time evolution to a hopping problem on a semi-infinite Krylov chain. The dynamics are governed by Lanczos coefficients $\{b_n\}$ , which act as hopping amplitudes. While the asymptotic behavior of these coefficients (e.g., linear growth $b_n \sim \alpha n$ for chaotic systems) encodes universal properties like information scrambling rates, practical computation is severely limited. Exact Lanczos iterations suffer from numerical instability due to finite-precision arithmetic and exponential memory costs as system size increases. Consequently, researchers typically compute only a short prefix of coefficients and extrapolate the remainder using simple asymptotic fits (e.g., linear or linear-plus-logarithmic forms). However, these standard fits fail to capture systematic, subleading structures—such as even-odd staggering and model-dependent oscillations—that are crucial for accurately reconstructing physical observables like autocorrelation functions and Krylov complexity.

Methodology
The authors propose a data-driven approach that treats the sequence of Lanczos coefficients as a causal time series. Instead of fitting to a predefined asymptotic function, they employ a Transformer-based architecture to autoregressively predict future coefficients from a short input prefix.

Data Representation: To handle the large dynamical range of $b_n$ , the model predicts the differences between neighboring coefficients, $\Delta b_n = b_n - b_{n-1}$ , rather than the coefficients themselves. This stabilizes the training target, as $\Delta b_n$ approaches a constant in the asymptotic limit.
Architecture: The model utilizes a Decoder-only Transformer. Scalar inputs ( $\Delta b_n$ ) are mapped to embedding vectors and augmented with positional encodings. The core mechanism is a masked multi-head self-attention layer, which enforces causality (predictions at step $n$ depend only on $n' \leq n$ ) and allows the model to weigh the influence of historical coefficients on future predictions.
Training Objective: The model is trained to minimize the mean squared error between the predicted next-step difference and the ground truth, averaged over a dataset of Lanczos sequences generated from various Hamiltonian instances.
Baseline Comparison: Performance is benchmarked against standard asymptotic fits (linear or linear-plus-logarithmic with even-odd staggering terms) derived from the Universal Operator Growth Hypothesis (UOGH).

Key Results
The authors validate their approach on three paradigmatic models: the classical XYZ spin top, the chaotic Transverse-Field Ising Model (TFIM), and the integrable Heisenberg XXZ model.

Superior Extrapolation Accuracy: In both classical and quantum chaotic systems, the Transformer model significantly outperforms asymptotic fits. It achieves an order-of-magnitude reduction in Root-Mean-Squared Error (RMSE) when extrapolating coefficients far beyond the input window. Crucially, the model successfully captures non-linear, subleading deviations and even-odd staggering that linear fits miss.
Physical Observable Reconstruction: The improved coefficient extrapolation translates directly to physical accuracy. When used to reconstruct the Krylov complexity $K(t)$ and autocorrelation function $C(t)$ , the Transformer-based method yields errors orders of magnitude smaller than the baseline fits at later times ( $tJ \gtrsim 2$ ).
Zero-Shot Transferability: A striking result is the model's ability to generalize across system sizes. A model trained exclusively on Lanczos sequences from a small system ( $L=8$ ) can accurately extrapolate coefficients for a larger system ( $L=12$ ) without retraining. This suggests the model learns universal structural features of the coefficient sequence rather than memorizing specific system parameters.
Integrable Regimes: The model also performs well in integrable systems (XXZ model), where no universal asymptotic form exists. It accurately reproduces sub-linear growth and complex staggering patterns without being explicitly programmed with these functional forms.
Attention Mechanism Analysis: By analyzing the learned attention weights, the authors identify that the model relies on three key patterns:
- Local Context: Strong attention to the most recent coefficients.
- Parity Sensitivity: A checkerboard pattern indicating sensitivity to even-odd staggering.
- Global Anchoring: Significant attention weights assigned to the very first few coefficients across the sequence, suggesting they serve as "anchors" for inferring global properties.
  Ablation studies confirm that removing long-range attention or masking early coefficients drastically degrades performance, validating the necessity of capturing long-range, history-dependent correlations.

Significance and Claims
The paper claims that modern sequence models, specifically Transformers, serve as practical and superior surrogates for probing operator dynamics deep within Krylov space, where brute-force Lanczos iteration is computationally prohibitive. The work demonstrates that treating Lanczos coefficients as a structured causal sequence allows for the extraction of subleading, history-dependent structures that are invisible to standard asymptotic fitting procedures.

The authors emphasize that this approach enables more reliable access to long-time operator dynamics from limited input data. Furthermore, the ability to train on computationally cheap, small-system data and transfer the model to larger systems offers a pathway to studying larger quantum many-body systems without incurring the exponential memory costs of direct simulation. The paper concludes by noting limitations, specifically that the current models are tied to specific Hamiltonian families and initial operators, and suggests future work toward a "foundation model" for quantum dynamics formulated in Krylov space.

Technical Summary: Attention in Krylov Space

More like this