Technical Summary: Understanding Key Features of Time Series Foundation Models from Epidemic Forecasting

Problem Statement

Accurate short-term forecasting of seasonal influenza is a critical public health need in the United States, informing vaccination timing, hospital staffing, and resource allocation. While the CDC FluSight and COVID-19 Forecast Hub initiatives have established concrete benchmarks for epidemic forecasting, the performance of modern time series foundation models (FMs) and large language model (LLM)-based approaches on operational infectious disease data remains insufficiently characterized.

Existing evaluations of time series FMs in epidemiology suffer from four primary limitations:

1. Data Scope: They often rely on single, country-level time series with limited history, failing to capture multi-season regional patterns essential for real-world planning.
2. Generalization: Spatial generalization (transfer across regions with different demographics and reporting practices) is rarely tested.
3. Experimental Design: Many studies use ad-hoc, long horizons (e.g., 24–60 weeks) that diverge from the CDC's operational 1–4-week targets and ignore data revisions (backfill).
4. Inconsistency: Heterogeneous data pipelines and baselines make it difficult to distinguish genuine modeling advances from preprocessing artifacts.

This paper addresses these gaps by conducting a systematic evaluation of regional influenza forecasting under realistic public health constraints, comparing classical neural networks, numerical transformer-based models, pretrained time series FMs, and LLM-based approaches.

Methodology

Data and Generalization Regimes

The study utilizes weekly time series data at the U.S. Health and Human Services (HHS) region level (10 regions) spanning approximately 20 years for Influenza-Like Illness (ILI) and three seasons for influenza-associated hospitalizations.

• Temporal Generalization: Models are trained on earlier segments of a specific region and evaluated on disjoint, later time ranges within the same region.
• Spatial Generalization: Models are trained on a subset of regions and evaluated on entirely unseen regions to test geographic transfer.
• Pretraining Sources: To assess cross-domain transfer, models are pretrained on diverse corpora including:
  • Epidemic Weekly Data: COVID-19 death counts.
  • TrafficL: High-frequency traffic lane occupancy.
  • M4: A large, heterogeneous benchmark of ~100,000 univariate series.
  • Hospitalization Data: Used as a mechanistically related signal.

Experimental Setup

• Horizons: 1–4 weeks ahead, aligned with CDC operational targets.
• Metrics: Mean Squared Error (MSE) and Normalized Nash–Sutcliffe Efficiency (NNSE).
• Models Evaluated: 17 deep forecasting models, including:
  • Pattern Models: PatchTST, iTransformer, TFT, TiDE, TCN, LSTM (iterative and direct).
  • Foundation Models: Chronos variants, TimesNet, TimeLLM.
  • Proposed Architecture: MultiFoundationCore (MFC), a Mixture-of-Experts (MoE) model that fuses multiple pretrained forecasters.
• Ablations: The study investigates the impact of pretraining sources, the use of auxiliary signals (hospitalization as input vs. pretraining), retraining frequency, and historical data length.

Key Contributions

1. Systematic Benchmarking: The authors provide the first systematic evaluation of time series foundation models for both ILI and hospitalization forecasting under explicit temporal and spatial generalization regimes using 1–4-week multi-horizon targets.
2. Architecture Comparison: A controlled comparison across model families reveals that Mixture-of-Experts (MoE) strategies, specifically the MultiFoundationCore (MFC) model, achieve the strongest overall performance. MFC fuses multiple pretrained forecasters, leveraging heterogeneous representations to provide complementary predictive information.
3. Pretraining Insights: The study demonstrates that pretraining provides the largest gains at longer horizons (weeks 3–4), particularly when the pretraining domain is mechanistically aligned with influenza dynamics (e.g., hospitalization data). Large generic corpora (M4) also offer significant benefits for long-range stability.
4. LLM Limitations: The paper finds that LLM-based time series methods (e.g., TimeLLM, Chronos) underperform relative to numerical forecasters in this setting. The authors attribute this to architectural mismatches where language-style tokenization fails to preserve the continuous numerical structure (amplitude, slope, phase) critical for short-horizon epidemic forecasting.
5. Auxiliary Signal Analysis: The research clarifies the utility of hospitalization data, showing it improves forecasts both as a pretraining source and as an auxiliary covariate, with pretraining yielding the most consistent gains across horizons.

Results

Temporal Evaluation (Within-Region)

• Top Performance: MultiFoundationCore achieved the best overall average performance (MSE = 0.382, NNSE = 0.864), outperforming all individual baselines.
• Numerical vs. LLM Models: Numerical transformer-based models (PatchTST, iTransformer) significantly outperformed LLM-based approaches. For instance, the best Chronos configuration averaged an MSE of 0.683, while PatchTST (without pretraining) achieved 0.439.
• Pretraining Impact: Pretraining consistently improved numerical models. PatchTST pretrained on Hospitalization data achieved an MSE of 0.412, and on M4 data 0.415, compared to 0.439 for the baseline. The benefits of pretraining were most pronounced at the 4-week horizon.
• Strategy: Direct multi-output forecasting models generally outperformed iterative models at longer horizons, as iterative approaches suffered from error accumulation.

Spatial Evaluation (Across-Region)

• Generalization: Models generally performed better under temporal splits than spatial splits, likely due to the limited diversity of the 10 HHS regions.
• Top Performer: In the spatial setting, PatchTST was the best-performing single model (Avg. MSE = 0.449, NNSE = 0.848), followed by TFT and LSTM.
• LLM Performance: LLM-based models (TimeLLM) continued to underperform in spatial generalization, reinforcing the difficulty of transferring language-style priors to unseen geographic contexts.

Auxiliary Signals and Retraining

• Hospitalization Data: Using hospitalization data as a pretraining source for ILI forecasting reduced MSE significantly (e.g., TFT MSE dropped from 0.428 to 0.346). Using it as an input stream also helped, particularly for shorter ILI histories.
• Retraining: More frequent retraining (every 10 steps vs. 200) yielded consistent, though diminishing, gains in accuracy, suggesting a practical balance exists between computational cost and predictive performance.
• Hospitalization Prediction: For the shorter, noisier hospitalization dataset, a compact TinyLSTM baseline performed surprisingly well, outperforming large foundation models when no auxiliary data was used. However, MultiFoundationCore still achieved the lowest overall error (MSE = 0.00224) when fusing ILI covariates.

Significance and Claims

The paper claims to provide actionable guidance for model selection and strategy in influenza surveillance:

• Model Selection: For operational 1–4-week forecasting, Mixture-of-Experts frameworks that fuse pretrained numerical experts are superior to single-model approaches.
• Pretraining Strategy: Pretraining is most beneficial for longer horizons and is most effective when the source domain is mechanistically related to the target (e.g., hospitalization for ILI).
• LLM Caution: Claims that reprogrammed LLMs decisively outperform specialized numerical models are not supported under realistic, short-horizon, multi-region epidemic settings. The continuous numerical nature of epidemic data requires inductive biases that language-style architectures currently lack.
• Data Utility: Long-term surveillance records and mechanistically linked auxiliary signals (like hospitalizations) are critical for robust multi-horizon forecasting.

The authors emphasize that their work is a systematic study of forecasting behavior rather than a fully operational real-time system. They release code, data, and pretrained checkpoints to serve as a reusable testbed for future research in epidemic forecasting with time series foundation models.