Analyzing the retraining frequency of global forecasting models: towards more stable forecasting systems

Technical Summary: Analyzing the Retraining Frequency of Global Forecasting Models

1. Problem Statement

In large-scale operational environments (e.g., retail, energy, supply chain), forecasting models are frequently retrained as new data arrives. While the industry standard focuses on maximizing accuracy, this often ignores forecast stability—the consistency of predictions over time.

Frequent retraining can lead to "forecast volatility," where small updates in data cause significant shifts in predictions (vertical instability). This volatility disrupts business planning, increases costs (e.g., the bullwhip effect), and erodes trust in automated systems. There is a prevailing perception of an inherent trade-off: that increasing accuracy through frequent updates must necessarily come at the expense of stability. This paper investigates whether this trade-off is real for Global Forecasting Models (GFMs)—models trained across multiple time series simultaneously—and how retraining frequency affects this balance.

2. Methodology

The study employs a rigorous experimental design to compare the impact of retraining intervals on both point and probabilistic forecasts.

• Datasets: Three large-scale, heterogeneous datasets were used: M4 (diverse economic/financial series), M5 (Walmart retail demand), and VN1 (e-commerce weekly sales).
• Models:
  • Global Models (10 total): Five Machine Learning models (Linear Regression, Random Forest, XGBoost, LightGBM, CatBoost) and five Deep Learning models (MLP, LSTM, GRU, TCN, NBEATS, NHITS).
  • Local Models (Benchmarks): Automatic ARIMA and ETS.
• Retraining Scenarios: The authors tested a spectrum of strategies ranging from continuous retraining (updating at every new observation) to no retraining (using a single model for the entire test period), including various periodic intervals (e.g., weekly, monthly).
• Evaluation Metrics:
  • Accuracy: Root Mean Squared Scaled Error (RMSSE) for point forecasts; Scaled Multi-Quantile Loss (SMQL) for probabilistic forecasts.
  • Stability (Point): Symmetric Mean Absolute Percentage Change (sMAPC).
  • Stability (Probabilistic): The paper introduces a new, model-agnostic, and scale-free metric: Scaled Multi-Quantile Change (SMQC). This metric measures the average adjustment in predicted quantiles between consecutive forecast origins, normalized by a seasonal naive baseline to ensure comparability across different scales.
• Uncertainty Quantification: Since most models are not natively probabilistic, the authors used Conformal Inference to generate valid prediction intervals.

3. Key Contributions

1. First Comprehensive Study of GFM Stability: It is the first large-scale empirical analysis specifically targeting the temporal stability of global models across both point and probabilistic dimensions.
2. New Metric (SMQC): Proposes the Scaled Multi-Quantile Change, a distribution-free and scale-free metric designed to quantify the stability of probabilistic forecasts.
3. Challenge to Retraining Paradigms: It provides empirical evidence that the "more retraining = better" intuition is flawed for global models.
4. Practical Guidelines: Offers specific retraining intervals (e.g., 21–30 days for daily data) that optimize the balance between accuracy, stability, and computational cost.

4. Results

• Stability vs. Retraining Frequency: For global models, less frequent retraining improves (or maintains) stability. As the retraining interval increases, sMAPC and SMQC values decrease, meaning forecasts become more consistent.
• The Accuracy-Stability Trade-off:
  • In Local Models: A strict trade-off exists; increasing stability (by retraining less) significantly degrades accuracy.
  • In Global Models: The trade-off is significantly milder. GFMs can maintain high accuracy even with infrequent updates. In some cases, less frequent retraining actually improved accuracy (e.g., deep learning on M4).
• Point vs. Probabilistic Sensitivity: Probabilistic accuracy (uncertainty estimation) is more sensitive to outdated data than point accuracy. While point forecasts remain stable over long intervals, probabilistic forecasts show a slight, gradual decay in accuracy if updates are too infrequent.
• Optimal Windows: For daily data, a retraining period of 21–30 days is identified as a "sweet spot." For weekly data, updates can be postponed up to 8 weeks without significant performance loss.

5. Significance

The paper has profound implications for sustainable and efficient AI operations. By demonstrating that frequent retraining is often unnecessary for global models, the study suggests that organizations can:

• Reduce Operational Costs: Lowering retraining frequency can reduce computational costs and energy consumption by up to 75%.
• Enhance Decision-Making: More stable forecasts lead to smoother supply chain operations and more reliable resource allocation.
• Shift Research Focus: It encourages the forecasting community to move beyond "accuracy-only" benchmarks and adopt stability-aware evaluation frameworks.