🌀 nonlinear sciences

Temporal horizons in forecasting: a performance-learnability trade-off

This paper establishes a fundamental trade-off in autoregressive forecasting, proving that while long-horizon training yields models with superior generalization to short-term predictions, it simultaneously creates a rougher loss landscape that makes optimization inherently more difficult.

Original authors: Pau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange

Published 2026-09-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pau Vilimelis Aceituno, Jack William Miller, Noah Marti, Youssef Farag, Victor Boussange

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Predicting the future is one of humanity's oldest and most persistent challenges, whether we are trying to forecast the weather, model the spread of a disease, or guide a robot through a cluttered room. In the modern era, scientists and engineers rely heavily on a specific type of computer program called an autoregressive model to do this work. These programs operate on a simple, iterative logic: they look at the current state of a system, make a prediction for the very next moment, and then feed that prediction back into the machine as if it were a real observation to predict the moment after that. By repeating this process, the model attempts to build a picture of what happens days, weeks, or years down the line. However, a fundamental question has long lingered in the training of these systems: how far into the future should the computer be asked to look while it is learning? Should it be taught to predict just the next second, or should it be forced to plan for the next hour? For a long time, the answer was largely a matter of guesswork or tradition, with researchers often choosing a time frame without a clear understanding of how that choice shaped the machine's ability to learn.

A team of researchers from institutions in Switzerland, Australia, and the United States has now provided a rigorous answer to this question, revealing a surprising trade-off that governs how these models learn. They discovered that the choice of the training time horizon is not just a minor setting; it fundamentally alters the shape of the mathematical landscape the computer navigates while learning. When a model is trained to predict only the immediate future, the mathematical terrain it explores is relatively smooth and easy to traverse, allowing it to find a good solution quickly. However, this ease comes at a cost: the solutions found in this smooth, short-term landscape often fail when asked to look further ahead, leading to predictions that drift wildly off course over time. Conversely, when the model is trained to predict far into the future, it learns a much more accurate understanding of the system's true behavior, one that generalizes well to both short and long-term forecasts. The catch is that this long-term training landscape is incredibly rough and jagged, filled with countless tiny peaks and valleys that make it extremely difficult for the computer to find the best solution without getting stuck or losing its way.

To understand this phenomenon, the researchers turned to the mathematics of dynamical systems, which describe how things change over time, such as the swirling patterns of a hurricane or the swinging motion of a double pendulum. They proved mathematically that as the time horizon for training increases, the "roughness" of the learning landscape grows exponentially for chaotic systems and linearly for systems that repeat in cycles. This roughness means that the computer's learning process, which relies on following the slope of the terrain to find the lowest point (the best prediction), becomes increasingly prone to stumbling. The steeper and more complex the terrain becomes with longer time horizons, the harder it is for standard training methods to succeed. Yet, the researchers also demonstrated that the few solutions found in these difficult, long-term landscapes are superior. A model that manages to find a good minimum in a long-horizon training session will naturally perform well on short-term tasks as well, whereas a model trained only on short horizons cannot reliably be trusted to predict the distant future.

The team validated these theoretical insights through a series of computer experiments using diverse systems, ranging from the famous chaotic Lorenz attractor, which mimics atmospheric convection, to a complex food web model representing ecological interactions. They also tested their findings on real-world data, including sea surface temperatures and stock market prices. In every case, the results followed the same pattern: the performance of the models formed a U-shaped curve relative to the training time horizon. Models trained on very short horizons performed poorly on long-term tasks, and models trained on excessively long horizons often failed to learn anything at all because the landscape was too rough to navigate. The sweet spot, the optimal training horizon, was found somewhere in the middle. Crucially, this optimal point was rarely the same as the specific time horizon the researchers wanted to test the model on later. Instead, the best training horizon depended on the intrinsic nature of the system being modeled, such as how chaotic or stable its behavior was, rather than the specific length of the forecast the user desired.

These findings offer a new, principled way to approach the training of predictive machines. The study suggests that the common practice of training a model to predict only the next step, or simply matching the training horizon to the testing horizon, is rarely the most effective strategy. Instead, the ideal approach involves selecting a training horizon that balances the system's inherent dynamics with the computational resources available. While training on longer horizons yields better, more robust models, it requires significantly more computing power and careful tuning to overcome the roughness of the learning landscape. The researchers propose that future systems could dynamically adjust this training horizon during the learning process, starting with shorter, easier horizons to find a good starting point and gradually extending the view as the model becomes more stable. This work provides a clear theoretical foundation for a field that has often relied on trial and error, showing that the key to better forecasting lies in understanding the hidden geometry of the learning process itself.

Technical Summary: Temporal horizons in forecasting: a performance-learnability trade-off

Problem Statement

Training autoregressive (AR) models to forecast dynamical systems presents a critical, yet theoretically underexplored, challenge: determining the optimal temporal forecast horizon ( $T$ ) to use during training. While practitioners often rely on single-step prediction or arbitrarily long horizons without rigorous justification, there is no existing theoretical framework to guide this choice. This gap affects both data-driven neural network models and mechanistic modeling approaches (e.g., differential equations) that utilize similar horizon-selection strategies like multiple shooting or piecewise regression. The core question addressed is how the training horizon influences model performance and the geometry of the loss landscape.

Methodology

The authors employ a combination of dynamical systems theory and machine learning analysis to derive a connection between the training horizon and the geometry of the loss landscape.

Theoretical Framework:
- Assumptions: The theory assumes smooth, deterministic, ergodic dynamical systems with fully observable states and sufficient data coverage of the stationary distribution.
- $\epsilon$ -bounded Regions: The authors define a region in parameter space where the model $f(\cdot, \theta)$ approximates the true dynamics $\phi$ within a bounded error $\epsilon$ . This allows the model's Jacobian to be closely aligned with the system's true Jacobian.
- Gradient and Hessian Analysis: They analyze the scaling of the loss gradient $\nabla_\theta L$ and the Hessian $H$ with respect to the temporal horizon $T$ . They utilize the Lyapunov exponent ( $\lambda$ ) for chaotic systems and the frequency ( $\omega$ ) for limit cycles to characterize system dynamics.
- Roughness Metric: They define the "roughness" of the loss landscape as the number of local minima and maxima along a line segment connecting two parameter points.
Empirical Validation:
- Synthetic Systems: Experiments were conducted on four dynamical systems: the Lorenz attractor, a double pendulum, a simple limit cycle, and a chaotic food web model.
- Real-world Datasets: The theory was tested on three empirical datasets: the ClimSIM climate simulation, NOAA Sea Surface Temperature (SST) data, and Amazon (AMZN) stock prices.
- Architectures: The study primarily utilized Multi-Layer Perceptrons (MLPs) and residual MLPs, though the theoretical implications extend to RNNs and Transformers via embedding arguments.

Key Contributions and Theoretical Results

1. Gradient and Hessian Scaling (Theorems 4.3 & 4.4)

The paper proves that the magnitude of the loss gradient and the Hessian (specifically the sum of eigenvalues) scale with the training horizon $T$ :

Chaotic/Unstable Systems: The gradient and Hessian scale exponentially, $O(e^{\lambda T})$ , where $\lambda$ is the Lyapunov exponent.
Limit Cycles: The scaling is linear, $O(\omega T)$ .
This implies that as the horizon increases, the loss landscape becomes increasingly steep and "rough."

2. Generalization Across Horizons (Theorem 4.6)

The authors establish that minima found with a long training horizon ( $T_h$ ) generalize better to shorter horizons ( $T_l$ ) than vice versa. Specifically, the ratio of performance differences between minima scales as $O(e^{\lambda(T_h - T_l)})$ for chaotic systems.

Implication: Models trained on long horizons capture the global dynamics of the system more effectively, making them robust for short-term predictions. Conversely, models trained only on short horizons may fail to capture long-term dependencies, leading to divergence in long-term forecasts.

3. Loss Landscape Roughness (Theorem 4.7)

A critical trade-off is identified: while long horizons yield better generalization, they make the loss landscape significantly harder to navigate.

The number of local minima and maxima along a parameter line segment grows as $O(e^{\lambda T})$ for chaotic systems.
As $T \to \infty$ , the loss landscape approaches a fractal structure, rendering gradient descent methods ineffective.
Implication: There is a "learnability" limit; training with excessively long horizons becomes computationally intractable due to the roughness of the landscape, even if the theoretical minimum is superior.

4. Mechanistic Models

The paper extends these insights to mechanistic models (e.g., ODEs with unknown parameters). In the ideal case of a perfect deterministic model, the loss is zero at the true parameters regardless of $T$ . However, in the presence of noise or model mismatch, the same performance-learnability trade-off observed in neural networks applies, suggesting an optimal intermediate horizon exists.

Experimental Results

U-Shaped Performance Curves: Across all synthetic and real-world datasets, the Mean Squared Error (MSE) exhibits a U-shaped curve relative to the training horizon.
- Performance improves as $T$ increases from 1 step (validating Theorem 4.6).
- Performance degrades after a certain point as the loss landscape becomes too rough to optimize effectively (validating Theorem 4.7).
Optimal Horizon vs. Evaluation Horizon: The optimal training horizon is rarely equal to the single-step prediction horizon, nor does it necessarily match the desired evaluation horizon. Instead, it is inextricably linked to the system's intrinsic dynamics (e.g., Lyapunov exponents).
Noise Sensitivity: In noisy datasets (NOAA SST, AMZN stocks), the optimal horizon is constrained by the noise level. High noise limits the effective horizon beyond which predictions become random, consistent with Proposition A.14.
Iterative Scheduling: The authors propose and test an iterative algorithm that slowly increases $T$ while adjusting the learning rate. This approach showed promise in outperforming fixed-horizon training in some cases, particularly for limit cycles, by navigating the trade-off between landscape roughness and gradient magnitude.

Significance and Claims

The paper claims to provide a principled foundation for hyperparameter optimization in autoregressive forecasting. Its primary significance lies in:

Resolving the Trade-off: It formally articulates the tension between learnability (ease of optimization, favored by short horizons) and generalization (accuracy on long-term dynamics, favored by long horizons).
Guiding Practice: It argues against the common practice of using single-step prediction or arbitrarily long horizons without justification. Instead, it suggests that the optimal training horizon is a function of the system's dynamics and computational resources.
Theoretical Unification: It bridges the gap between dynamical systems theory (Lyapunov exponents, attractors) and machine learning optimization (loss landscape geometry, gradient scaling), offering a mathematical explanation for phenomena like the vanishing/exploding gradient problem in the context of forecasting horizons.

The authors remain modest, acknowledging that their assumptions (determinism, full observability) are idealizations. They note that in highly stochastic or non-stationary real-world scenarios, the optimal horizon is further constrained by noise, and the theoretical bounds serve as approximations rather than exact prescriptions.