🤖 machine learning

LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series

This paper demonstrates that language-pretrained transformers can effectively forecast time series because pretraining establishes a reusable geometric manifold of structural dynamics, allowing finetuning to merely align numerical data onto these existing directions rather than learning temporal primitives from scratch.

Original authors: Alexis Roger, Prateek Humane, Zhenghan Tai, Gwen Legate, Andrei Mircea, Vasilii Feofanov, Irina Rish

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexis Roger, Prateek Humane, Zhenghan Tai, Gwen Legate, Andrei Mircea, Vasilii Feofanov, Irina Rish

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Can a Language Brain Predict the Weather?

Imagine you have a super-smart robot that has read every book, article, and website on the internet. It is a master at predicting the next word in a sentence. Now, you ask it to predict the next number in a stock market chart or the next temperature reading for a city.

Usually, we think these are two totally different jobs. One is about words (language), and the other is about numbers (time series). You might think the robot needs to "unlearn" how to read and "relearn" how to do math.

This paper argues that the robot doesn't need to unlearn anything. Instead, it turns out that reading books already taught the robot how to see patterns, rhythms, and trends. The paper proves that the robot's "brain" (its internal structure) is already shaped in a way that makes it great at predicting numbers, even before it ever sees a single number.

The Core Idea: The "Manifold" (The Shape of the Brain)

The authors use a fancy word called a "manifold." Let's call it the "Shape of the Brain."

The Old View: When you train a model on numbers from scratch, it's like building a house from a pile of bricks. You have to lay every single brick (learn every pattern) yourself.
This Paper's View: Language pretraining is like buying a house that is already built. The walls, the roof, and the floor are already there. When you want to use it for a new purpose (predicting numbers), you don't rebuild the house. You just move the furniture around to fit the new room.

The "furniture" here represents the directions in the model's brain where it stores information about repetition, trends, and sudden changes.

The Evidence: How They Proved It

The researchers ran several experiments to show that the "house" was already built. Here is what they found:

1. The "Magic Decoder" (Linear Probe)

They took the language-trained robot and froze its brain so it couldn't learn anything new. Then, they tried to use a very simple tool (a "linear probe") to translate the robot's thoughts about text into predictions about numbers.

The Result: Even without any training on numbers, the robot's internal thoughts about text could be translated into realistic-looking number patterns (like sine waves or stock trends).
The Analogy: Imagine you have a dictionary written in a secret code. You try to use a simple decoder ring to translate it into a map. Surprisingly, the map looks perfect. This means the "secret code" (the language training) already contained the geometry of the map.

2. The "Gradient Coherence" (The Smooth Path)

When you train a model from scratch (randomly), it stumbles around in the dark. It tries a direction, fails, tries another, and eventually finds a path. This is messy and slow.

The Result: The language-trained robot started with a clear, smooth path. Its internal "gradients" (the signals telling it which way to improve) were already aligned and working together from the very first second.
The Analogy: Training a random model is like trying to find your way out of a maze in the dark with a blindfold. Training a language model is like walking out of that same maze with a flashlight. The path was already lit up by the language training.

3. The "Low-Rank" Adjustment (Moving Furniture, Not Rebuilding)

They compared training the whole model from scratch vs. just tweaking a tiny part of the language model (using a method called LoRA).

The Result: The tiny tweak worked almost as well as training the whole thing from scratch.
The Analogy: If you want to turn a living room into a home office, you don't need to knock down the walls and pour a new foundation. You just move the desk and add a lamp. The paper shows that for time series, we only need to "move the furniture" (low-rank updates) because the "walls" (the core structure) are already perfect.

4. The "Shared Features" (The Same Brain Cells)

They looked at specific parts of the robot's brain to see what it was actually "thinking" about.

The Result: They found specific brain cells that fired when the robot saw:
- In Text: A story about a storm getting stronger, or a list of dates and measurements.
- In Numbers: A sudden spike in temperature or a repeating pattern.
The Analogy: The robot uses the exact same "neurons" to understand a sentence about a "sudden drop in pressure" and a graph showing a "sudden drop in pressure." It's not learning two different things; it's recognizing the same shape of change in two different languages.

The Conclusion: Geometry, Not Semantics

The most important takeaway is this: The robot isn't predicting numbers because it "understands" what a stock market is.

It predicts numbers because language is full of patterns (repetition, trends, cycles, sudden changes). By learning to predict the next word in a sentence, the robot accidentally learned how to predict the next number in a sequence.

Language Pretraining: Builds the "shape" of the brain to handle sequences.
Time Series Finetuning: Just points the brain in the right direction to use that shape for numbers.

The paper concludes that we don't need to teach these models math from scratch. We just need to show them that the patterns they already know from reading books apply to numbers, too. It's a "geometric" transfer, not a "semantic" one.

Technical Summary: LLM Pretraining Shapes a Generalizable Manifold

Problem Statement

The paper investigates the mechanism behind the cross-modal transfer of Large Language Models (LLMs) to time-series forecasting. While recent studies have demonstrated that LLMs can be adapted for time-series tasks, empirical results regarding the source of this benefit remain contradictory. Some literature attributes performance gains to architectural biases or tokenization effects rather than the pretrained representations themselves, while others observe significant improvements in low-data or distribution-shift regimes. The central question addressed is: Do language-pretrained transformers become effective time-series forecasters because they learn reusable sequential structures, or is the transfer merely semantic?

Methodology

The authors repurpose Qwen3-0.6B, a 0.6B parameter language model, as a probabilistic time-series forecaster. The experimental setup involves:

Tokenization: Continuous time-series values are normalized and discretized into $V=1024$ uniform bins over the range $[-5, 5]$ . These bins are mapped to the model's vocabulary.
Training Objective: The model is trained using a next-token prediction paradigm with a weighted quantile loss (pinball loss) over quantile levels $Q = \{0.1, \dots, 0.9\}$ , following the Chronos Bolt approach.
Adaptation Strategies: The study compares four regimes on the GiftEval benchmark:
1. Full Finetuning (all parameters updated).
2. IO Only (frozen backbone, trainable input/output heads).
3. LoRA Attn (LoRA adapters on attention projections only).
4. LoRA Attn+IO (LoRA on attention plus trainable IO).
Comparative Baselines: All strategies are tested against models with random initialization (RandInit) and against established time-series baselines (Chronos variants, ARIMA).

Key Contributions and Findings

1. The Shared Manifold Hypothesis

The paper proposes that language pretraining constructs a reusable temporal manifold rather than just semantic knowledge. This is evidenced by three primary experiments:

Linear Probing without Supervision: A simple linear probe trained on frozen LLM hidden states (using only WikiText inputs) can decode realistic time-series trajectories. When matched via nearest-neighbor to a bank of 10,000 real time series, the pretrained model produces diverse, structured outputs (MSE 0.23–0.38). Crucially, this structure collapses when using random weights or random tokens, proving that the combination of pretrained weights and meaningful text is required to access the manifold.
Retrieval-Based Forecasting: A projection retrieved from the pretrained manifold based on the first half of a time series can forecast the second half with lower error (MSE 1.91) than a naive last-value carry-forward baseline (MSE 2.27). This suggests the manifold contains predictive continuation signals before any time-series fine-tuning occurs.
Gradient Coherence: Pretrained models exhibit high gradient alignment (mean pairwise cosine similarity) from the very first step of fine-tuning. In contrast, randomly initialized models undergo a "warmup" phase with near-zero alignment until step 64–256, after which gradients become coherent. This indicates that pretraining provides a favorable optimization landscape with coherent, mutually reinforcing gradients, bypassing the destructive interference phase seen in random initialization.

2. Low-Rank Adaptation and Geometric Alignment

The paper argues that fine-tuning acts as a low-dimensional alignment rather than learning temporal primitives from scratch:

Effective Data Transfer: Using the framework of Hernandez et al. [2021], the authors show that parameter-efficient methods (LoRA) recover most of the transfer benefit. Full finetuning achieves an effective data transfer (DT) of 6.8, while LoRA Attn achieves 5.2. This implies that only a small perturbation of the pretrained weights is necessary.
Effective Rank Dynamics: Randomly initialized models start with near-isotropic hidden states and undergo a rapid "rank collapse" to a low-dimensional structure. Conversely, language-initialized models start with a compressed representation and undergo selective compression: middle layers increase effective rank while early and late layers decrease. This suggests fine-tuning redistributes capacity, repurposing middle-layer circuitry rather than building new structures.
Feature Reuse: Cross-domain crosscoder analysis reveals that specific latent features in the pretrained model fire on both time-series motifs (e.g., magnitude jumps, missing data, regime changes) and semantically coherent text passages (e.g., unit conversions, tropical cyclone narratives, naval battle timestamps). This confirms that the transferred subspace contains interpretable temporal primitives shared across modalities.

3. Geometric vs. Semantic Transfer

The authors demonstrate that while both pretrained and randomly initialized models eventually learn to encode periodic structures (e.g., sine waves), they do so via different geometric paths:

RandInit learns simple, low-dimensional arcs with high variance explained by principal components.
LangInit retains complex, heterogeneous, layer-varying loops inherited from pretraining, which are geometrically richer but functionally aligned with the task.
Circuit Analysis: Zero-ablation identifies a specific superadditive circuit in Layer 1 (MLP and attention head) critical for periodic prediction. This same circuit is shown to be highly sensitive to sequential repetitive structures in text, providing causal evidence that the model reuses abstract structural mechanisms rather than semantic content.

Significance and Claims

The paper concludes that the mechanism enabling LLM-to-time-series transfer is geometric rather than semantic.

Pretraining as Manifold Construction: Language pretraining organizes the model into a sequential manifold encoding dynamics like repetition, trends, and periodicity.
Fine-tuning as Projection: Fine-tuning does not learn forecasting structure from scratch; it projects numerical dynamics onto task-relevant directions already present in the manifold.
Reconciliation of Conflicting Literature: This geometric account explains why transfer is most effective in low-data and distribution-shift regimes (where pretrained directions provide structure that limited data cannot establish) and why parameter-efficient methods like LoRA are effective (as transfer is fundamentally direction selection).

The authors maintain a modest stance, noting that their findings are based on a single backbone (Qwen3-0.6B) and corpus, and that the causal circuit evidence is preliminary. However, they assert that any sufficiently rich autoregressive sequence model with latent temporal structure may support similar transfer, extending the relevance of these findings beyond language models.