🤖 machine learning

Foundation Models and Fine-Tuning: Toward a New Generation of Models for Time Series Forecasting

This paper reviews the architectures, pre-training strategies, and optimization methods of foundation models for zero-shot time series forecasting and demonstrates that fine-tuning these models on specific datasets consistently improves forecasting accuracy over zero-shot baselines.

Original authors: Morad Laglil, Bertrand Pracca, Emilie Devijver, Eric Gaussier

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Morad Laglil, Bertrand Pracca, Emilie Devijver, Eric Gaussier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the future. Not the fate of the universe, but something much more practical: how much electricity a city will need tomorrow, how many umbrellas a shop should stock, or how traffic will flow through a busy intersection an hour from now. This is the world of time series forecasting. For decades, experts have used mathematical recipes to guess these numbers, but the world is messy, and old recipes often fail when the data gets weird or complicated.

Recently, a new kind of "super-learner" has arrived, inspired by the same technology that lets computers write poetry and chat like humans. These are called Foundation Models. Think of them as a master chef who has tasted every dish in the world. Instead of learning to cook just one specific meal (like a single restaurant's soup), this chef learns the general rules of flavor, heat, and ingredients from millions of different recipes. Once trained, this chef can walk into a kitchen they've never seen before and cook a decent meal without needing a specific recipe for that exact dish. This is called zero-shot learning: making a prediction on data the model has never seen before, just by using what it learned in its "training camp."

But here is the big question: Is it better to send this master chef into a new kitchen and let them work alone, or should we give them a quick, specific lesson on this kitchen's specific quirks before they start cooking? This is the heart of the new paper by Morad Laglil and his team. They want to know if taking these giant, pre-trained time-series models and giving them a little extra training on specific data (called fine-tuning) actually makes them better forecasters, or if it just confuses them.

The Great "Fine-Tuning" Experiment

The authors of this paper decided to put this idea to the test. They took two of the most powerful "master chef" models available—Chronos 2 and TTM-R3-PT—and put them through a rigorous training camp. They didn't just let them guess; they tried different ways of teaching them specific tasks.

They tested three main approaches:

Zero-Shot: The model tries to predict the future using only its general training, with no extra help.
Full Fine-Tuning: They took the entire model and retrained it from scratch on the new data, updating every single number inside the brain.
Parameter-Efficient Fine-Tuning (LoRA): Instead of rewriting the whole brain, they added a tiny, lightweight "adapter" layer. It's like giving the chef a specific cheat sheet for the new kitchen without changing their fundamental cooking skills.

They tested these methods on a massive collection of 15 different datasets, ranging from weather patterns and electricity usage to hospital admissions and sales figures. The goal was simple: see which method actually lowered the error rate and made better predictions.

What They Found: Size and Context Matter

The results were surprising and taught the researchers a valuable lesson about the size of the model and the data it faces.

For the giant model (Chronos 2):
This model is huge, with 120 million parameters. When the researchers tried to "Full Fine-Tune" it (rewriting the whole brain) on smaller datasets, it backfired. The model got confused and started to "overfit," which is like a student who memorizes the practice test answers so perfectly that they fail the real exam because they can't handle a slightly different question.

The Winner: The LoRA method (the tiny adapter) was the clear champion for Chronos 2 on medium and large datasets. By adding just a small, flexible layer, the model could adapt to the new data without forgetting its general knowledge. It improved accuracy, though the gains were modest (around 2-3%) on larger datasets, and it was crucial for avoiding the performance drops seen with full fine-tuning.
The Lesson: For massive models, you don't need to rewrite the whole book; you just need to add a few sticky notes. However, on very small datasets, even the giant model often performed best without any fine-tuning at all.

For the smaller model (TTM-R3-PT):
This model is much more compact, with only 1 million to 36 million parameters. It was already small enough that it didn't need a fancy adapter.

The Winner: Full Fine-Tuning worked best here. Because the model was smaller, it could handle updating all its own weights without getting confused. The "adapter" method (LoRA) actually didn't help much and sometimes made things worse.
The Lesson: For smaller models, a full workout is often better than a quick stretch.

The "One Size Does Not Fit All" Rule

The paper also discovered that the best strategy depends heavily on the type of data, how much of it you have, and the specific domain.

Small Datasets: If you only have a tiny amount of data (like a few days of weather records), Zero-Shot (letting the model guess on its own) is often the safest bet. Trying to fine-tune a giant model on too little data is like trying to teach a PhD student a new subject by showing them only three flashcards; they will likely get it wrong. In fact, for Chronos 2 on small datasets, fine-tuning actually degraded performance.
Big Datasets: When there is plenty of data, fine-tuning often beats guessing, but it is not a guaranteed win. The paper showed that for the smaller TTM model, zero-shot inference still outperformed fine-tuning in several specific configurations, even on larger datasets.
Different Domains: The models behaved differently depending on the topic. For example, in the "Nature" domain (like river levels or weather), the giant model was already so good at guessing that fine-tuning actually made it worse. But in "Transport" (like traffic), fine-tuning helped both models significantly.

The Bottom Line

The paper concludes that while foundation models are powerful tools that can predict the future without any specific training, fine-tuning is a powerful tool to make them even better—but only if you choose the right recipe for the right situation.

If you have a giant model, use the "adapter" (LoRA) method to tweak it gently, especially on large datasets. If you have a smaller model, feel free to give it a full training session. And if you don't have much data at all, or if the data is very specific (like nature patterns), it might be safer to just let the model do its best guess without interference.

This research suggests that we are moving toward a new generation of forecasting where we don't need to build a new model for every single problem. Instead, we can use one giant, smart model and give it a quick, tailored lesson to solve almost any time-based puzzle, from predicting the next stock price to planning the next city bus route. The future of prediction isn't just about having a bigger brain; it's about knowing how to teach it the right lessons.

Technical Summary: Foundation Models and Fine-Tuning for Time Series Forecasting

Problem Statement

Time series forecasting is a critical task across diverse domains such as energy, transportation, and finance. While traditional statistical methods (e.g., ARIMA) and modern deep learning models have advanced the field, they typically require training from scratch on specific target datasets. This approach is data-hungry and often impractical when target datasets are small or when rapid adaptation to new contexts is required.

Inspired by the success of foundation models in Natural Language Processing (NLP), Time Series Foundation Models (TSFMs) have emerged. These are large-scale models pre-trained on massive, diverse corpora to learn generalizable temporal representations. While TSFMs demonstrate strong zero-shot capabilities (predicting on unseen data without retraining), the literature has largely underexplored the potential of fine-tuning these models on specific target datasets to further enhance performance. This paper addresses the gap in understanding how to effectively adapt pre-trained TSFMs to specific downstream tasks, balancing the trade-offs between zero-shot generalization and dataset-specific specialization.

Methodology

1. Review of Architectures and Training Strategies

The authors provide a comprehensive review of the current landscape of TSFMs, categorizing them by architecture and training objectives (summarized in Table 1 of the paper):

Architectures: The review covers Transformer-based models (e.g., Chronos, Moirai, TimesFM, Timer-S1), Recurrent networks (TiRex based on sLSTM), and MLP-Mixer models (TTM).
Pre-training Strategies: Models utilize self-supervised learning on vast corpora (e.g., LOTSA, TimeBench, synthetic data). Objectives vary from autoregressive token prediction (Chronos) and masked patch prediction (Moirai) to quantile loss minimization (TimesFM, TTM).
Adaptation Mechanisms: The paper details two primary fine-tuning strategies:
- Full Fine-Tuning (FFT): Updating all model parameters on the target dataset.
- Parameter-Efficient Fine-Tuning (PEFT): Specifically Low-Rank Adaptation (LoRA), which freezes pre-trained weights and trains low-rank perturbation matrices ( $\Delta W = BA$ ), significantly reducing trainable parameters.
- Data Efficiency: The study also considers Few-Shot settings (using only 20% of target data) versus Full-Shot settings.

2. Empirical Evaluation Protocol

The authors conducted an extensive empirical study using the GIFT-Eval benchmark, which comprises 23 datasets (15 univariate, 8 multivariate) spanning 7 domains (e.g., Web/CloudOps, Energy, Transport, Healthcare) and 10 frequencies.

Models Selected: Two top-performing, publicly available models were chosen for fine-tuning: Chronos 2 (120M parameters, Encoder-only Transformer) and TTM-R3-PT (1M–36M parameters, MLP-Mixer).
Experimental Setup:
- Models were evaluated in Zero-Shot (pre-trained only), Full Fine-Tuning (FFT), LoRA-FullShot, and LoRA-FewShot configurations.
- Hyperparameters were fixed across all datasets to simulate realistic deployment constraints (no per-dataset tuning).
- Metrics: Performance was measured using MAPE (for point forecasting) and MWQL (Mean Weighted Quantile Loss, approximating CRPS for probabilistic forecasting), normalized against a Seasonal Naive baseline.

Key Results

The study reveals that the efficacy of fine-tuning is highly dependent on model scale, dataset size, and domain characteristics:

Impact of Dataset Size:
- Chronos 2 (Large Model): On small datasets, zero-shot inference outperforms all fine-tuning strategies. Full fine-tuning (FFT) severely degrades performance (overfitting), while LoRA offers marginal or negative gains. On medium and large datasets, LoRA-FullShot emerges as the superior strategy, improving upon zero-shot performance by 2–3%.
- TTM-R3-PT (Compact Model): Being smaller, TTM benefits more from direct weight updates. Full Fine-Tuning (FFT) provides modest improvements on large datasets, while LoRA variants offer little gain and occasionally degrade performance.
Domain-Specific Performance:
- Transport: Fine-tuning is generally beneficial. LoRA (specifically Few-Shot) often yields the best MAPE results, suggesting that lighter adaptation helps avoid overfitting to noisy, high-entropy dynamics.
- Nature & Energy: Zero-shot Chronos 2 is remarkably strong, particularly in the Nature domain, where full fine-tuning significantly inflates errors (e.g., +39% MAPE error on short-term forecasts).
- Web/CloudOps: LoRA-FullShot is the most effective strategy for Chronos 2, while zero-shot remains competitive for TTM.
Strategy Comparison:
- LoRA vs. FFT: For large models like Chronos 2, parameter-efficient adaptation (LoRA) is critical; full fine-tuning is unreliable and prone to overfitting. For compact models like TTM, full fine-tuning is often sufficient and effective.
- Data Efficiency: While LoRA-FewShot is competitive in specific scenarios (e.g., Energy domain for Chronos 2), LoRA-FullShot generally outperforms it when sufficient data is available.

Key Contributions

Comprehensive Review: The paper establishes a unified analytical framework for Time Series Foundation Models, detailing their architectures (Transformers, RNNs, MLPs), pre-training corpora, and optimization strategies.
Systematic Fine-Tuning Study: This work presents what the authors claim is the first large-scale empirical study of fine-tuning strategies for TSFMs. It systematically compares Zero-Shot, Full Fine-Tuning, and Parameter-Efficient Fine-Tuning (LoRA) across a diverse set of 95 unique configurations (dataset $\times$ frequency $\times$ horizon).
Practical Guidelines: The study provides actionable insights for practitioners:
- Use Zero-Shot for small datasets or when computational resources are limited.
- Use LoRA-FullShot for large foundation models (e.g., Chronos 2) on medium-to-large datasets to prevent overfitting.
- Use Full Fine-Tuning for compact models (e.g., TTM) where the parameter count is low enough to avoid overfitting risks.

Significance and Claims

The paper positions itself as a foundational step toward a "new generation" of time series forecasting models. Its primary significance lies in validating that fine-tuning is a viable and often necessary step to unlock the full potential of foundation models, but only when the adaptation strategy is matched to the model's scale and the data's volume.

The authors modestly note that while fine-tuning generally improves performance, it is not a universal panacea; in data-scarce scenarios, the generalization capabilities of the pre-trained model (Zero-Shot) remain superior. Furthermore, the paper highlights the current limitations of the field, including the scarcity of diverse, public time series data compared to NLP, and the lack of theoretical understanding regarding how these models compress and generalize temporal information. The work serves as a benchmark for future research into more robust and efficient adaptation mechanisms for time series foundation models.