🤖 machine learning

Why Do Time Series Models Need Long Context Windows?

This paper argues that long context windows in time series forecasting are essential not only for capturing long-range dependencies but primarily for reducing uncertainty in identifying the underlying data-generating process, a necessity proven to exceed the process's memory length for achieving minimum error.

Original authors: Luca Butera, Giovanni De Felice, Andrea Cini, Cesare Alippi

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Luca Butera, Giovanni De Felice, Andrea Cini, Cesare Alippi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Why Do We Need So Much History?

Imagine you are trying to predict the weather for tomorrow. A standard approach is to look at the last 24 hours of temperature and rain. But modern AI models for time series (like predicting electricity usage or traffic) are often fed massive amounts of past data—sometimes weeks or months of history.

For a long time, experts thought these models needed so much history simply to spot "long-range patterns," like realizing that a storm today might be related to a pressure system from three weeks ago.

This paper argues that's only half the story. The authors claim the real reason models need long windows is to solve a different problem: figuring out exactly what kind of process is generating the data in the first place.

The Two Jobs of a Time Series Model

The authors break down the job of a forecasting model into two distinct tasks:

Conditional Forecasting (CF): "Given what just happened, what happens next?"
- Analogy: If you see a car swerving left, you predict it will likely hit the curb. You only need the last few seconds of video to make this guess.
Generative Process Identification (GPI): "What are the rules of the game?"
- Analogy: Before you can predict the car's movement, you need to know: Is this a human driver? A self-driving car? A drunk driver? A toy car on a track? Each "driver" follows different rules.

The Core Insight:
In many real-world scenarios (like a "Foundation Model" trained on data from thousands of different companies), the AI doesn't know a priori which specific "driver" it is looking at. It has to infer the rules by looking at the data.

The paper claims that GPI is the reason we need long windows. You might need a month of data just to figure out, "Ah, this specific time series behaves like a seasonal retail store," before you can even start predicting next week's sales.

The "Detective" Analogy

Imagine a detective trying to solve a crime.

Conditional Forecasting is like looking at the suspect's footprints right now to guess where they are walking next.
Generative Process Identification is like looking at the suspect's entire life history, their fingerprints, and their past crimes to figure out who they are.

If you only look at the footprints from the last 5 minutes (a short window), you might not know if the suspect is a sprinter, a walker, or someone in a wheelchair. You need a longer history (a long window) to identify the "process" (the person) so you can accurately predict their next move.

The "Too Many Options" Problem

The paper uses a mathematical proof to show that even if a process only depends on the last $P$ steps (like a simple memory), a global model still needs more than $P$ steps to be perfect.

The Scenario: Imagine you have a bag of different dice. Some are weighted to roll high numbers, some low. You don't know which die you are holding.
The Short Window: If you roll the die twice, you might not know if it's a "high" die or a "low" die. You are guessing.
The Long Window: If you roll the die 50 times, you can be 99% sure it's the "high" die. Now you can predict the next roll with high confidence.

The paper proves that to get the best possible prediction, the model needs that long window to reduce the uncertainty about "which die" (which process) it is dealing with.

The Cost of Being a "Generalist"

The paper highlights a trade-off for Foundation Models (AI models trained on everything from traffic to weather to energy).

Specialist Model: A model trained only on traffic data knows the "rules" of traffic immediately. It needs a short window to predict the next traffic jam.
Generalist Model: A model trained on everything doesn't know if it's looking at traffic or weather. It needs a longer window to "read the room" and figure out, "Okay, this looks like traffic data, not weather data."

The authors show that foundation models require much longer input windows than specialist models to achieve the same accuracy, simply because they have to do extra work to identify the context.

The Solution: Split the Job

The paper proposes a clever way to make these models faster and cheaper without losing accuracy. Instead of feeding the whole long history into the main prediction engine every single time, they suggest decoupling the two jobs:

The "Context Reader" (GPI): A dedicated module looks at a long history of data once to figure out the rules. It creates a summary (a "latent embedding") of what kind of process this is.
The "Predictor" (CF): This module takes the recent data (just the last few steps) plus the summary from step 1 to make the prediction.

The Analogy:
Imagine a chef (the Predictor) who needs to cook a dish.

Old Way: Every time the chef needs to cook, they have to read the entire 500-page cookbook from page 1 to find the recipe. This is slow.
New Way: A sous-chef (the Context Reader) reads the whole book once, figures out "We are making Italian Pasta," and writes a sticky note saying "Italian Pasta Rules." The main chef just looks at the fresh ingredients and the sticky note.

This allows the model to use a massive amount of historical data to understand the context (GPI) without having to re-process that huge history every single time it makes a prediction (CF). This saves computing power and memory.

Summary of Findings

Why long windows? Not just to see far back in time, but to identify the rules of the specific data stream you are looking at.
The Proof: Even for simple processes, you mathematically need more data points than the "memory length" of the process to be sure of the rules.
The Benefit: By separating "figuring out the rules" from "making the prediction," we can build models that are just as accurate but much faster and cheaper to run.

The paper concludes that future time series models should be designed with this separation in mind, treating the input window as a tool for identification (GPI) rather than just a source of temporal dependencies (CF).

Technical Summary: Why Do Time Series Models Need Long Context Windows?

Problem Formulation

Modern deep learning approaches for forecasting groups of time series, particularly foundation models, increasingly rely on long observation windows. While the prevailing explanation attributes this need to capturing long-range temporal dependencies, the authors argue this view is incomplete. The paper posits that training a global model on a collection of time series generated by diverse processes involves two distinct, coupled objectives:

Generative Process Identification (GPI): Inferring the specific latent stochastic process ( $\phi_n$ ) that generated the observed input sequence.
Conditional Forecasting (CF): Predicting future values conditioned on the past, assuming the process is known.

The core problem addressed is understanding why global models, especially those trained inductively on heterogeneous data (foundation models), require input windows significantly larger than the theoretical memory length ( $P$ ) of the underlying processes to achieve optimal performance. The paper challenges the assumption that long windows are solely for capturing temporal dependencies, suggesting instead that they are necessary to reduce uncertainty about the data-generating process.

Methodology and Theoretical Framework

Theoretical Decomposition

The authors formalize the forecasting task as minimizing the expected loss over a family of stochastic processes. By applying the law of total conditional expectation, they decompose the optimal predictor into a Bayesian posterior average:
$\hat{x}_t = \int \mathbb{E}[x_t | x_{t-W:t}, \phi] \cdot p(\phi | x_{t-W:t}) \, d\phi$
Here, the term $\mathbb{E}[x_t | x_{t-W:t}, \phi]$ represents the Conditional Forecasting (CF) component, while $p(\phi | x_{t-W:t})$ represents the Generative Process Identification (GPI) component. The input window $W$ serves two roles:

For CF, only the most recent $P$ observations (where $P$ is the process memory) are strictly necessary.
For GPI, the entire window provides evidence to contract the posterior distribution over the process parameters $\phi$ .

Theoretical Proofs

The paper proves Theorem 3.2, which states that for processes with memory length $P$ , an input window size $W > P$ is a necessary condition to achieve the minimum attainable error (the irreducible noise variance) in a global setting.

Assumption: The paper assumes that different processes in the family cannot be uniquely identified from only the last $P$ observations (i.e., the variance of the conditional expectation across possible processes is non-zero given $P$ observations).
Implication: Even if the true temporal dependencies are short, the uncertainty about which process is generating the data necessitates additional observations to resolve that uncertainty. This is framed as a form of In-Context Learning (ICL), where the model infers a latent task variable (the process parameters) from the input sequence.

Experimental Design

The authors validate their theory through:

Synthetic Experiments: Generating time series from Non-Linear Autoregressive (NAR) and Linear AR processes with known orders ( $P$ ). They compare local models (trained on single series) against global models (trained on multiple series) and models utilizing a "context" (a historical sequence separated by a gap from the prediction window).
Foundation Model Probing: Testing pre-trained foundation models (Moirai2, TimerXL) by feeding them a short contiguous window plus a distant historical context. If performance relies on long-range dependencies, the gap should degrade accuracy; if it relies on GPI, the distant context should maintain or improve accuracy.
Domain-Specific vs. Foundation Models: Comparing models trained on a single domain against those trained on multiple domains to quantify the "cost" of GPI in terms of required window length.
Decoupled Architecture: Proposing a design where GPI and CF are separated. A module $G(\cdot)$ processes a long historical context to produce a latent embedding $e_n$ , which is then used by a lightweight predictor $F(\cdot)$ operating on a short recent window.

Key Contributions

Problem Formulation: The paper redefines global time series forecasting as a dual task of GPI and CF, providing a framework to characterize time series foundation models.
Theoretical Necessity of $W > P$ : It proves that for global models, the input window must strictly exceed the process memory length to minimize error, driven by the need to identify the generating process.
Empirical Evidence for GPI: Experiments show that foundation models benefit from historical context even when separated by a temporal gap, confirming that they utilize input data for process identification rather than just temporal dependency modeling.
Cost Analysis: The study demonstrates that foundation models require significantly longer windows than domain-specific global models to match performance, attributing this to the higher uncertainty (variance) in the pretraining distribution.
Methodological Proposal: The authors propose decoupling GPI and CF to improve scalability. By amortizing the cost of GPI (processing long contexts) and reusing the resulting embeddings for multiple forecasts, the approach reduces inference costs (FLOPs, memory, latency) without compromising accuracy.

Results

Synthetic Data: Global models required $W \gg P$ to approach the error floor of local models. However, when provided with a long historical context (separated by a gap) and a short window ( $W=P$ ), global models achieved near-optimal error, confirming that the extra data was used for GPI, not temporal dependency.
Foundation Models: When tested on real-world datasets (traffic, energy, climate), foundation models (Moirai2, TimerXL) maintained performance with a short window ( $W=24$ ) if a long historical context ( $C=720$ ) was provided, despite the temporal gap. This suggests the models were inferring the process from the context rather than relying on contiguous long-range dependencies.
Multi-Domain Training: Models trained on multiple domains (simulating foundation models) required larger windows to match the performance of single-domain models. Increasing the window size allowed multi-domain models to recover performance by reducing process uncertainty.
Decoupled Architecture: The proposed decoupled design (long context for GPI, short window for CF) achieved Pareto-optimal performance-cost trade-offs. It matched the accuracy of standard models with large windows while significantly reducing computational costs (up to 27x reduction in FLOPs and memory in some scenarios) by caching the GPI embedding.

Significance and Claims

The paper claims to offer a fundamental shift in understanding why time series foundation models require long context windows. Rather than viewing long windows solely as a mechanism for capturing long-range temporal dependencies, the authors argue they are essential for reducing uncertainty about the data-generating process (GPI).

The significance of this insight lies in:

Design Principles: It suggests that future foundation models should not simply scale up window sizes monolithically. Instead, architectures should explicitly decouple the tasks of identifying the process context and performing the forecast.
Efficiency: By separating GPI (which can be computed once and cached) from CF (which is computed frequently), models can achieve high accuracy with lower computational overhead, making them more scalable for real-world deployment.
Interpretability: The work provides a theoretical link between time series forecasting and In-Context Learning (ICL), framing the input window as a set of examples used to infer a latent task variable.

The authors remain modest, noting that while they demonstrate the relevance of these insights, the development of a full-scale foundation model based on this decoupled architecture is left as future work. They also acknowledge that for highly time-varying processes, the static nature of the GPI embedding might require dynamic updates, which is an open direction for future research.