🔭 astrophysics

When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms

This controlled study demonstrates that multi-token prediction is the primary factor enabling stable long-horizon autoregressive forecasting of synthetic seismograms, while also identifying a critical context-ratio threshold for generalization and highlighting the need for phase-aware objectives to address residual polarity inversion failures.

Original authors: Waleed Esmail, Stuart Russell, Jana Klinge, Alexander Kappes, Christine Thomas

Published 2026-06-10

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Waleed Esmail, Stuart Russell, Jana Klinge, Alexander Kappes, Christine Thomas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to sing a long, complex song by ear. You play it a few seconds of the melody, and it has to keep singing the rest of the song on its own, note by note, for several minutes.

This paper is about a specific type of robot (an AI model) trying to do exactly that, but instead of singing, it's predicting seismic waves (the vibrations of the Earth during an earthquake). The researchers wanted to figure out: Why does the robot sometimes get the rhythm wrong after a while, and how can we fix it?

Here is the breakdown of their findings using simple analogies:

The Problem: The "Drifting Metronome"

When the robot predicts the next few seconds of a wave, it does a great job. But when it has to keep predicting hundreds of seconds into the future using its own previous guesses as the starting point, a problem called "error accumulation" happens.

Think of it like a game of "Telephone."

The Setup: You whisper a sentence to the first person, who whispers it to the next, and so on.
The Issue: Even if everyone whispers perfectly, tiny mistakes happen. By the time the message reaches the end, the sentence has changed completely.
In the AI: The robot makes a tiny mistake in timing (a "phase drift") every single step. At first, it's unnoticeable. But after 200 steps, the robot is singing the same song, just completely out of sync with the real Earth. It might be singing the right notes at the wrong time, or even singing them backwards (upside down).

The Solution: Three Tools in the Toolbox

The researchers tested a specific AI model (SeismoGPT) and tried different "tools" to stop this drifting. They found three main things that helped:

1. The "Group Forecast" (Multi-Token Prediction) – The Big Hero

The Old Way: The robot was asked, "What is the very next second?" It guessed, then was asked, "What is the next second?" over and over. This is like walking blindfolded, taking one step, checking your footing, and taking another.
The New Way: The robot is asked, "What are the next four seconds?" It looks at the whole chunk at once.
The Result: This was the most important fix. It's like giving the robot a map of the next few steps instead of just one. It stopped the robot from getting lost immediately. The paper says this single change did almost all the heavy lifting to keep the prediction stable.

2. The "Specialized Coach" (Horizon-Embedding Hybrid Head) – The Small Helper

The Idea: Even when looking at four seconds, the robot treats the 1st second and the 4th second the same way. But the 4th second is further away and needs a different kind of "focus."
The Fix: The researchers gave the robot a special "coach" for each specific second in the group. It's like having a different coach for the sprint, the middle, and the finish line, all working together.
The Result: This helped a little bit, making the prediction slightly more accurate, but it wasn't a game-changer on its own.

3. The "Rhythm Check" (Cross-Horizon Coherence Loss) – The Quality Control

The Idea: Sometimes the robot sings the right notes but the texture of the sound changes weirdly between the first chunk and the second chunk.
The Fix: The researchers added a rule that checks if the "sound texture" (the frequency spectrum) flows smoothly from one chunk of time to the next.
The Result: This helped keep the "sound quality" consistent. However, it had a blind spot (see below).

The Catch: The "Polarity" Blind Spot

There is one thing the robot still struggles with, and the researchers were honest about it.

Imagine the robot is predicting a wave that goes Up, Down, Up, Down.

The robot might predict: Down, Up, Down, Up.
To a human ear (or a standard "volume" check), this sounds exactly the same. The "loudness" and "rhythm" are perfect.
But to the Earth, this is a disaster. The wave is inverted (upside down).

The researchers found that their "Rhythm Check" tool (Tool #3) looks only at the volume and frequency, not the direction. So, it couldn't tell the robot when it was singing the song backwards. This is the biggest remaining failure: the robot gets the rhythm right but flips the sign of the wave.

The "Golden Rule" of Context

The researchers also discovered a strict rule for when this works:

The Rule: The robot needs to see at least one full cycle of the main earthquake waves (the time between the first "P" wave and the second "S" wave) before it starts guessing.
The Analogy: If you try to teach someone a song but only let them hear the first half-second of the intro, they will guess the rest of the song wrong. They need to hear the whole "verse" (the P-S interval) to understand the pattern. If the robot sees less than this, it fails immediately.

Summary

What worked best? Asking the robot to predict a block of time at once (Multi-Token Prediction) rather than one second at a time. This stopped the "Telephone game" errors.
What helped a little? Giving the robot specific coaches for different time steps and checking that the sound texture flows smoothly between steps.
What still fails? The robot sometimes flips the wave upside down (predicting "Down" when it should be "Up"). The current tools can't see this because they only check the "volume," not the "direction."
The Bottom Line: This study wasn't about building the perfect earthquake predictor yet. It was a controlled experiment to figure out which parts of the robot's brain were responsible for keeping the rhythm steady. They found that "looking ahead in chunks" is the key to stability, but they still need to teach the robot to recognize "up" vs. "down."

Technical Summary: When Do Autoregressive Sequence Models Forecast Physical Wavefields?

Problem Statement

The paper addresses the challenge of long-horizon autoregressive forecasting for oscillatory physical signals, specifically synthetic three-component seismograms. While sequence models (particularly causal transformers) perform well at short horizons, they suffer from error accumulation during free-running rollout (where the model feeds its own predictions back as input).

For oscillatory wavefields, this accumulation manifests primarily as phase drift. Small per-step timing errors compound over hundreds of steps, causing the predicted waveform to slide out of phase with the ground truth. While the envelope and frequency content may remain plausible, the waveform becomes locally anti-correlated with the target. Standard pointwise metrics often fail to detect this failure because local errors remain modest even as the global phase alignment collapses. Existing remedies like scheduled sampling or non-autoregressive forecasting do not explicitly target the spectral boundary between successive predictions where this drift originates.

Methodology

The study utilizes SeismoGPT, an autoregressive transformer-based forecaster, as a testbed to isolate specific design choices governing rollout stability. The authors conduct a controlled intra-architecture ablation study on synthetic seismograms (spanning epicentral distances of 10–90°, depths of 5–100 km, and magnitudes of 3–7).

The experimental setup involves:

Tokenization: Raw time-series data is tokenized into 16-sample chunks across 3 channels.
Backbone: A vanilla causal transformer encoder (512 dimensions, 8 heads) remains fixed across all configurations to ensure comparisons isolate specific components.
Ablations: The authors compare a single-token baseline against four configurations:
- Multi-Token Prediction (MTP): The model predicts a horizon of $H=4$ tokens jointly rather than one at a time.
- Horizon-Embedding Hybrid Head: A single shared MLP head conditioned on learned per-horizon embeddings (vs. independent heads for each horizon).
- Cross-Horizon STFT Magnitude Coherence Loss: A loss term applied to the concatenation of adjacent predicted horizons, measuring the magnitude coherence of their Short-Time Fourier Transforms (STFT) across multiple resolutions (128, 256, 512 FFT sizes).
- Auxiliary Losses: Log-cosh base loss, first-difference MSE, and per-horizon STFT loss.

The evaluation focuses on free-running rollout over 240 seconds (approx. 28 tokens) using a context ratio ( $\rho$ ) relative to the P-S wave arrival interval. Performance is measured via Normalized Cross-Correlation (NCC), Signal-to-Residual Ratio (SRR), and Power Spectral Density (PSD) error. Statistical significance is established via paired Wilcoxon signed-rank tests with bootstrap confidence intervals over 10,000 held-out events.

Key Contributions

The paper explicitly frames itself as a controlled study of rollout stability, not a benchmark of forecasting architectures. Its contributions are:

Quantification of Multi-Token Prediction (MTP): The study isolates MTP as the dominant stabilizer for long-horizon oscillatory forecasting. Enabling MTP accounts for almost the entire performance gain over a single-token baseline.
Marginal Gains from Secondary Components: The authors quantify that the horizon-embedding hybrid head and the cross-horizon STFT magnitude coherence loss provide small but consistent additive improvements (+0.009 and +0.005 median NCC, respectively) only when combined with MTP.
Regime Identification: The study identifies a sharp performance threshold at a context ratio of $\rho \approx 1$ . Reliable long-horizon rollout requires a context length roughly equal to the full P-S interval of the observed signal; below this, generalization collapses.
Failure Mode Analysis: The paper identifies that the dominant residual failure is polarity inversion (phase drift leading to sign flips). It demonstrates that magnitude-based spectral losses (like the STFT coherence loss) are inherently incapable of penalizing these inversions because $|STFT(x)| = |STFT(-x)|$.

Results

Dominance of MTP: Switching from single-token to multi-token prediction (Ablation C) improves the median NCC by +0.040 (95% CI [0.038, 0.042]) over the baseline. The full model improves by +0.045.
Additive Effects: Within the MTP setting, the hybrid head adds +0.009 NCC, and the coherence loss adds +0.005 NCC. The auxiliary losses alone (without coherence) slightly degrade NCC relative to MTP-only but improve spectral shape (PSD error).
Spectral Fidelity: While NCC is driven by MTP, the auxiliary losses and coherence term significantly reduce PSD error (from 15.2 for baseline to 12.1 for the full model), improving spectral shape fidelity.
Context Ratio: Performance remains high and stable for $\rho \ge 1$ but degrades sharply for $\rho < 1$ .
Failure Modes: In worst-case scenarios, the model produces waveforms with correct frequency content but inverted polarity, resulting in negative NCC values (down to -0.56). The magnitude-based coherence loss cannot detect these errors.

Significance and Claims

The paper modestly claims to identify which design choices govern long-horizon stability within a specific autoregressive family, rather than proposing a new state-of-the-art system.

Primary Finding: Multi-token prediction is the critical mechanism for stabilizing autoregressive rollout on oscillatory physical signals.
Secondary Finding: Horizon-embedding and cross-horizon spectral coherence provide marginal, consistent gains but are insufficient to solve the fundamental phase-drift problem on their own.
Limitation: The study is confined to a single synthetic dataset and a single training seed per configuration. The authors do not claim that causal transformers are the best model class for this task, nor do they claim the component effects hold across different seeds or real-world data without further validation.
Future Direction: The inability of magnitude-based losses to correct polarity inversion identifies phase-aware objectives (e.g., complex STFT, group-delay, or anti-wrapping losses) as the necessary next step for addressing the dominant residual failure mode.