Technical Summary: EpiCastBench

Problem Statement

Epidemic forecasting has become a cornerstone of public health policy, yet the development of robust, generalizable forecasting methods is hindered by a lack of comprehensive, standardized benchmarks. Unlike fields such as computer vision (ImageNet) or natural language processing (GLUE), the epidemic forecasting domain lacks large-scale, standardized repositories. Existing datasets are often univariate, disease-specific, or geographically limited, failing to capture the diversity of transmission dynamics across different infectious diseases and environments. Furthermore, the absence of unified evaluation protocols makes it difficult to compare models fairly or assess their reproducibility across diverse epidemiological contexts.

Methodology

To address these gaps, the authors introduce EpiCastBench, a large-scale benchmarking framework designed for evaluating data-driven multivariate epidemic forecasting models.

Dataset Curation

The framework comprises 40 curated multivariate time series datasets representing incidence cases for eight infectious diseases: COVID-19, chickenpox, chikungunya, dengue, influenza, measles, tuberculosis, and Zika. These datasets span 27 distinct geographical regions, capturing significant spatial variability and epidemiological diversity. Key characteristics of the datasets include:

• Temporal Granularity: Daily, weekly, and monthly frequencies.
• Series Length: Ranging from 60 to 2,088 observations, covering small, medium, and large sample sizes.
• Dimensionality: Series counts vary from 4 to 107, including high-dimensional, low-sample-size scenarios (e.g., 31 provinces in China with only 60 time steps).
• Data Challenges: The datasets exhibit zero-inflation (prolonged periods of zero cases), missing values (imputed via Last Observation Carried Forward), and varying sparsity levels.

Global Feature Analysis

The authors utilized the tsfeatures package in R to analyze eight global time series features (trend, long memory, nonlinearity, non-stationarity, entropy, seasonality, spikiness, and curvature). This analysis revealed distinct temporal behaviors based on transmission mechanisms:

• Airborne diseases (e.g., COVID-19, measles) exhibit strong nonlinearity and non-stationarity.
• Vector-borne diseases (e.g., dengue, Zika) show pronounced spikiness and strong trends.
• Contact-based diseases (e.g., chickenpox) display high entropy and strong seasonality.
• Droplet-borne diseases (e.g., influenza) show balanced characteristics with higher curvature.

Experimental Setup

The study evaluates 15 forecasting models across three horizons (short-term, medium-term, and long-term) using a fixed-origin evaluation protocol. The models span four paradigms:

1. Statistical Baselines: Naive (Random Walk).
2. Machine Learning: DLinear, Random Forest, XGBoost, TSMixer, KAN.
3. Deep Learning: LSTM, DeepAR, TCN, N-BEATS, N-HiTS, Transformers, TiDE.
4. Foundation Models: Chronos-2 and TimesFM.

Evaluation was conducted using scale-independent metrics (MASE, SMAPE) and scale-dependent metrics (MAE, RMSE). Statistical significance was assessed using the Friedman test and Multiple Comparisons with the Best (MCB) test.

Key Results

The empirical evaluation yielded several critical findings regarding model performance across different dataset sizes, disease types, and forecasting horizons:

• Dominance of Foundation Models: Foundation models, particularly TimesFM and Chronos-2, consistently demonstrated superior performance, achieving the lowest median errors and the most stable interquartile ranges across most datasets and horizons. Their advantage is attributed to large-scale pretraining, which enables better generalization to unseen epidemic dynamics.
• Impact of Forecasting Horizon:
  • Long-term: Foundation models showed the most significant advantage, effectively capturing complex temporal dependencies and non-stationary dynamics.
  • Short-term: Classical machine learning models (e.g., KAN, Random Forest, XGBoost) remained competitive, sometimes outperforming deep learning models in highly sparse or small datasets.
• Zero-Inflation and Sample Size: The performance of foundation models is modulated by the interaction between zero-inflation and sample size. While they handle sparsity well in moderate-to-large datasets, their advantage diminishes in small, highly zero-inflated series (e.g., specific COVID-19 datasets in Chile), where ensemble methods occasionally perform better.
• Disease-Specific Dynamics:
  • Airborne/Droplet-borne: Foundation models excelled due to stable trends and seasonality.
  • Vector-borne: Performance was more variable; models like KAN and Random Forest occasionally outperformed foundation models in capturing abrupt local dynamics.
  • Contact-based: Models like TiDE and KAN showed strong performance in capturing smooth seasonal dynamics mixed with high variability.
• Statistical Significance: The MCB tests confirmed that the superior performance of foundation models is statistically significant across most horizons. However, the differences between the top-performing foundation models (TimesFM and Chronos-2) were often not statistically significant, indicating a robust "best-in-class" tier.

Significance and Contributions

The paper claims the following contributions to the field of epidemic forecasting:

1. Standardized Benchmark: EpiCastBench provides the first large-scale, unified benchmark for multivariate epidemic forecasting, filling a critical void in the availability of diverse, high-quality datasets comparable to those in other AI domains.
2. Reproducibility and Fairness: By establishing standardized preprocessing pipelines, evaluation settings, and statistical testing protocols, the framework enables consistent and reproducible comparisons of new forecasting methods against a diverse set of baselines.
3. Empirical Insights: The study offers a comprehensive characterization of how different model architectures perform under specific epidemiological conditions (e.g., zero-inflation, varying sample sizes, and transmission dynamics), guiding future model selection and development.
4. Open Resources: All datasets, code, and evaluation scripts are publicly available, fostering further research and the development of more generalizable epidemic forecasting methods.

The authors conclude that while foundation models currently offer the most robust performance, the specific dynamics of different diseases and data regimes necessitate continued exploration of hybrid and specialized approaches. They identify future work directions including the extension to spatiotemporal settings, the incorporation of probabilistic forecasting, and the integration of mechanistic epidemiological principles into data-driven models.