Foundation vs. Specialized Models: Evaluating Catastrophic Forgetting in Continual Time Series Forecasting

Technical Summary: Foundation vs. Specialized Models in Continual Time Series Forecasting

Problem Statement
While Time Series Foundation Models (TSFMs) have demonstrated strong zero-shot and few-shot generalization capabilities, their behavior under continual fine-tuning remains poorly understood. Specifically, there is a lack of systematic evaluation regarding "catastrophic forgetting"—the tendency of models to lose previously acquired knowledge when sequentially fine-tuned on new tasks. This issue is critical for real-world applications where data distributions are non-stationary and evolve over time. The paper investigates whether the computational cost of large TSFMs is justified over smaller, specialized models when both are subjected to continual learning (CL) scenarios, and how effectively forgetting mitigation strategies can bridge the performance gap.

Methodology
The authors propose a systematic evaluation framework comparing TSFMs against a specialized, lightweight model across two distinct continual learning scenarios:

1. Models Evaluated:
   • TSFMs: TimesFM-2.0 (500M parameters, decoder-only), Chronos-2 (120M parameters, encoder-only), and Chronos-small (28M parameters, encoder-only).
   • Specialized Baseline: SamFormer (100K parameters), a lightweight Transformer utilizing Sharpness-Aware Minimization (SAM) and channel-wise attention.
2. Scenarios:
   • Synthetic ($S_{syn}$): Four multi-sinusoidal datasets ($D_1$ to $D_4$) with increasing complexity and non-harmonic periods, designed to induce strong spectral shifts and distribution changes.
   • Real-World ($S_{energy}$): Four energy forecasting datasets from Enedis (France), including residential consumption, photovoltaic production, a private household (unseen during pre-training), and wind production. These exhibit heterogeneity in seasonality, volatility, and external drivers.
3. Learning Protocols:
   • Sequential Training: Training on datasets $D_1 \to D_N$ without access to previous data.
   • Cumulative Training: Training on $\{D_1, \dots, D_t\}$ at stage $t$ (simulating infinite memory).
   • Independent/Offline Training: Baselines for dataset difficulty and upper-bound performance.
   • Dark Experience Replay (DER): A replay-based strategy combining experience replay with prediction distillation to mitigate forgetting.
4. Metrics:
   • RMSE: Root Mean Square Error for forecasting accuracy.
   • Backward Transfer (BWT): Measures forgetting by comparing performance on a dataset after training on subsequent tasks versus immediately after training on that dataset.
   • Cross-Series Transferability (CST): A novel, model-aware metric proposed to quantify how training on dataset $D_i$ impacts performance on $D_j$, defined as the ratio of RMSE after training on $D_i$ to RMSE from random initialization.

Key Contributions

1. Systematic CL Evaluation: The first comprehensive study of catastrophic forgetting in TSFMs (TimesFM-2.0, Chronos-2) compared to a specialized model (SamFormer) in both synthetic and real-world energy forecasting benchmarks.
2. Model-Aware Transferability Metric: Introduction of the CST metric, which outperforms standard similarity measures (like Dynamic Time Warping) in anticipating forgetting severity by accounting for the specific feature extraction capabilities of the model.
3. Mitigation Strategy Analysis: A rigorous assessment of DER in time series forecasting, demonstrating its efficacy across different model scales.

Results

• Inherent Forgetting: All models, regardless of scale, exhibited catastrophic forgetting under naive sequential training. However, larger foundation models (TimesFM-2.0) demonstrated greater inherent robustness, achieving lower average RMSE and BWT compared to smaller models in the synthetic scenario.
• Scenario Differences: Forgetting was more severe in the synthetic scenario ($S_{syn}$) than in the real-world scenario ($S_{energy}$). The authors attribute this to the synthetic tasks having abrupt spectral shifts, whereas real-world energy datasets, despite their noise, share more compatible latent temporal structures within the energy domain.
• Impact of DER: The application of Dark Experience Replay significantly mitigated forgetting across all architectures.
  • DER provided disproportionate gains to smaller models. For instance, on the synthetic scenario, SamFormer's BWT improved from 2.04 to 0.15 with DER.
  • By the end of the continual learning sequence, smaller models equipped with DER were able to match the performance of large TSFMs.
  • In some cases (e.g., Chronos-base on $S_{energy}$), DER achieved better average RMSE than cumulative training, suggesting a "curriculum training" effect where progressive complexity aids learning.
• Specialized vs. Foundation: While TSFMs generally outperformed SamFormer in zero-shot and naive sequential settings, the gap closed significantly when specialized models were augmented with effective mitigation strategies like DER.

Significance and Claims
The paper concludes that in realistic, non-stationary scenarios, the high computational cost of large foundation models may not be strictly necessary if smaller models are equipped with effective forgetting mitigation strategies. The findings suggest that:

• Catastrophic forgetting is a pervasive issue in time series CL, driven by distribution shifts and spectral incompatibilities.
• Model scale offers inherent robustness, but replay-based strategies (DER) can level the playing field, allowing parameter-efficient models to achieve competitive performance with foundation models.
• The proposed CST metric is a valuable tool for anticipating forgetting and designing more robust continual learning scenarios.

The authors maintain a modest stance, noting that their conclusions are specific to the evaluated models and scenarios, and that the choice between foundation and specialized models depends on the specific trade-off between computational resources and the availability of effective CL strategies.