Technical Summary: PHYSIOME-ODE

Problem Statement

Current research on forecasting irregularly sampled multivariate time series (IMTS) with missing values suffers from a severe lack of diverse and challenging benchmarks. The field predominantly relies on a small set of four datasets (USHCN, PhysioNet-2012, MIMIC-III, MIMIC-IV) and a few toy examples. The authors identify a critical flaw in this evaluation landscape: on these existing datasets, a simple baseline model that predicts a constant value independent of time often outperforms or is competitive with sophisticated Ordinary Differential Equation (ODE)-based neural models. This unintuitive finding suggests that current datasets may be ill-suited for evaluating the true capabilities of ODE-based models, thereby hampering research progress. Furthermore, the existing datasets often lack necessary covariants (e.g., food intake for blood glucose prediction), making the forecasting task inherently impossible regardless of model complexity.

Methodology

To address these limitations, the authors propose Physiome-ODE, a new benchmark generated through a systematic methodology:

1. Data Generation from Biological ODEs:

   • The authors utilize the Physiome Model Repository (PMR), which contains 208 multivariate ODE models derived from biological research.
   • They treat these ODE systems as generative models. An ODE system $F$ with constants $a$ and initial values $x_0$ defines a continuous function $x(t)$.
   • To create IMTS instances, they apply a sampling process $s$ to generate irregular observation times and channels, and a noise model $p_{NOISE}$ to introduce observation noise.
   • Crucially, to ensure diversity and difficulty, they do not use fixed ODE instances. Instead, they vary the initial conditions ($x_0$), ODE constants ($a$), and the total duration ($T$) by sampling from distributions centered around literature values but controlled by spread parameters ($\sigma$).

2. Difficulty Metric: Joint Gradient Deviation (JGD):

   • To select the most challenging datasets, the authors introduce a metric called Joint Gradient Deviation (JGD).
   • JGD combines two measures:
     • Mean Gradient Deviation (MGD): Measures the heterogeneity of the gradient within a single function (time-normalized $L_2$ norm of the gradient around its mean).
     • Mean Point-wise Gradient Deviation (MPGD): Measures the heterogeneity of gradients across different functions in a distribution (standard deviation of gradients at each time point across the ensemble).
   • $JGD(p) = MPGD(p) \cdot E_{x \sim p}[MGD(x)]$.
   • The authors prove that MGD and MPGD can be approximated using finite samples and divided differences (Lemmas 1 and 2).
   • They optimize the spread parameters ($\sigma_{initial}, \sigma_{const}, \sigma_{dur}$) for each ODE model to maximize the JGD, thereby selecting the most difficult forecasting scenarios.

3. Benchmark Construction:

   • From the 208 models in the PMR, the top 50 models with the highest optimized JGD scores were selected.
   • For each selected model, 2,000 time series instances were generated.
   • Instances are created by sampling a random start time and selecting the next 100 steps, then randomly dropping 80% of observations to simulate irregular sampling.
   • Extreme values (outliers caused by unstable ODE states) are rejected to ensure training stability.

Key Contributions

1. Physiome-ODE Benchmark: The creation of the first large-scale, diverse benchmark for IMTS forecasting, consisting of 50 distinct datasets derived from real-world biological ODEs. It is an order of magnitude larger than the current evaluation setting.
2. Baseline Analysis: The introduction of a "constant forecast" baseline (GraFITi-C) which demonstrates that current state-of-the-art models often fail to outperform simple heuristics on existing datasets, highlighting the need for better evaluation protocols.
3. JGD Metric: The proposal of Joint Gradient Deviation as a simple, effective score to approximate the difficulty of time series forecasting tasks, allowing for the selection of challenging ODE instances.
4. Comprehensive Evaluation: A rigorous evaluation of five state-of-the-art IMTS models (GRU-ODE-Bayes, LinODEnet, CRU, Neural Flows, GraFITi) and the constant baseline across the new benchmark.

Experimental Results

The authors evaluated the models on the 50 Physiome-ODE datasets using Mean Squared Error (MSE) with a 5-fold cross-validation protocol (predicting the last 50% of the series).

• Model Performance: Unlike previous evaluations where simple baselines dominated, the results on Physiome-ODE show that ODE-based models can leverage their strengths. LinODEnet emerged as the overall best-performing model, winning the most datasets and achieving the highest average rank. CRU and GraFITi also performed competitively.
• Baseline Comparison: While complex models outperformed the constant baseline on many datasets, the benchmark includes difficult instances (e.g., DUP01, DOK01) where even the best models could not significantly outperform the constant baseline. This confirms the benchmark's ability to span a wide range of difficulties.
• Correlation: There is a strong positive correlation (Spearman coefficient of 0.77) between the JGD score and the best achievable MSE, validating JGD as a predictor of task difficulty.
• Channel Dependency: In supplementary experiments on regularly sampled versions of these datasets, the authors found that multivariate models (capturing channel dependencies) significantly outperformed univariate models, contrasting with findings on traditional benchmarks where ignoring channel correlations is sometimes beneficial.

Significance and Claims

The paper claims that Physiome-ODE represents a significant step forward in standardizing IMTS forecasting research. By moving away from biased, small-scale datasets toward a large, semi-synthetic benchmark grounded in biological physics, the authors expect to:

• Provide a more meaningful differentiation between competing forecasting models.
• Identify specific deficits in current architectures, particularly on high-difficulty datasets.
• Give a "fresh impulse" to research on irregular time series modeling by demonstrating that ODE-based methods can indeed excel when evaluated on appropriate, challenging data.

The authors emphasize that while the data is generated from ODEs (making it "semi-synthetic"), the underlying systems are based on decades of published biological research, ensuring a connection to real-world phenomena. They acknowledge that no single model is universally best across all datasets, reinforcing the need for diverse benchmarks rather than a single "winner-takes-all" evaluation.