Technical Summary: GenTS – A Comprehensive Benchmark Library for Generative Time Series Models

1. Problem Statement

While generative models have demonstrated significant potential in time series analysis (e.g., synthesis, forecasting, imputation), the current ecosystem lacks a unified, comprehensive benchmark library tailored specifically for them. Existing time series libraries (e.g., TSLib, GluonTS) are primarily engineered for discriminative models, featuring rigid workflows optimized for specific tasks like minimizing Mean Squared Error (MSE) for forecasting.

This structure is fundamentally incompatible with generative models, which often employ distinct and complex training paradigms (e.g., adversarial training, diffusion processes, differential equations) aimed at learning underlying data distributions rather than direct input-output mappings. Furthermore, existing libraries often lack coverage across:

• Diverse Tasks: Many focus on a single task (e.g., only synthesis or only forecasting) or ignore versatility.
• Model Diversity: Few libraries integrate the full spectrum of generative architectures (GANs, VAEs, Diffusions, Flows, Differential Equations).
• Evaluation: There is a scarcity of standardized, panoramic evaluation metrics that assess both statistical fidelity and downstream utility.

This gap hinders systematic assessment of new advancements and the comparison of different generative approaches.

2. Methodology: The GenTS Framework

To address these limitations, the authors propose GenTS, a comprehensive and extensible benchmark library built on PyTorch and PyTorch Lightning. The framework is designed with a modular architecture covering the full benchmarking pipeline:

A. Data Preprocessing

• Multi-Domain Datasets: GenTS includes over 15 datasets from six domains (Traffic, Energy, Financial, Medical, Weather, Physics), ranging from high-frequency (Hz) to daily resolutions.
• Synthetic Data: Includes simple physical trajectory simulations (Spiral2D, SineND) for rapid prototyping.
• BaseDataModule: A customizable class (inheriting LightningDataModule) that handles dataset splitting, dataloader instantiation, and task-specific preprocessing (e.g., simulating missing values for imputation or defining look-back windows for forecasting). It supports irregular time series and user-defined custom datasets.

B. Model Training

• Model Coverage: The library implements over 25 State-of-the-Art (SOTA) models and their variants, categorized into five types:
  • GANs: TimeGAN, RCGAN, COSCIGAN, etc.
  • VAEs: TimeVAE, KoVAE, TimeVQVAE, etc.
  • Diffusion: DiffusionTS, CSDI, ImagenTime, TMDM, etc.
  • Flows: MAF, FourierFlow, etc.
  • Differential Equations: LatentODE, LatentSDE, etc.
• Naive Baselines: Four "Vanilla" models (VanillaVAE, VanillaGAN, VanillaDDPM, VanillaMAF) are provided as starting points, utilizing simple MLPs or 1D-CNN backbones without task-specific adaptations.
• BaseModel: A template class (inheriting LightningModule) that standardizes training, validation, and sampling steps. It supports flexible callbacks, logging, and gradient accumulation. It also includes a unified sample method for generating time series under various conditions (unconditional, class-guided, forecasting, imputation).

C. Performance Evaluation

GenTS moves beyond simple MSE to provide a panoramic evaluation suite:

• Model-Free Metrics: Statistical measures including Wasserstein Distance, Continuous Ranked Probability Score (CRPS), and MSE to measure distributional closeness.
• Model-Based Metrics: Neural network-based evaluations including:
  • Predictive Score (PS): Evaluates the utility of synthetic data for training downstream forecasting models (Train on Synthetic, Test on Real).
  • Discriminative Score (DS): Measures how distinguishable synthetic data is from real data using a post-hoc classifier.
  • Context-FID (C-FID): Adapts Frechet Inception Distance using TS2Vec embeddings to measure local temporal context alignment.
• Visualization: Tools for t-SNE visualization of 2D distributions and plotting of forecasting/imputation results with prediction intervals.

3. Key Contributions

1. GenTS Library: The first comprehensive benchmark library designed specifically for generative time series models, offering standardized yet customizable workflows for data preparation, training, and evaluation across unconditional and conditional tasks.
2. Extensive Benchmarking: The authors conducted large-scale experiments across three major tasks—Synthesis (unconditional and class-guided), Forecasting, and Imputation—using diverse datasets and models.
3. Empirical Insights & Guidelines: Based on the benchmark results, the paper provides concrete suggestions for model selection in cross-task and task-specific contexts and identifies potential future research directions.

4. Experimental Results

The benchmarking experiments yielded several key findings:

• Time Series Synthesis:
  • Diffusion models (e.g., ImagenTime, DiffusionTS) generally demonstrated superiority, particularly in model-based metrics (PS and DS), indicating high fidelity and downstream utility.
  • GANs: While generally less stable, COSCIGAN stood out for its ability to fit complex data distributions, performing well on C-FID and Wasserstein Distance.
  • Class-Guided Synthesis: TimeVQVAE showed the best alignment with real class-labeled clusters. GANs generated categorized data but failed to align well with real clusters.
• Time Series Forecasting:
  • CSDI and TMDM frequently ranked as top performers for both deterministic and probabilistic forecasting.
  • TMDM showed absolute advantages on non-stationary datasets (e.g., Stocks) due to its specialized backbone.
  • Surprisingly, naive baselines (VanillaMAF, VanillaVAE) showed competitive performance on specific datasets (ETTh, Electricity), suggesting that architectural complexity is not always necessary.
• Time Series Imputation:
  • Diffusion-based models dominated this task with absolute superiority. CSDI consistently outperformed all other models, followed by ImagenTime and DiffusionTS.
  • Differential Equation-based models and naive baselines generally exhibited weaker performance, rarely achieving top-3 rankings.
• Efficiency:
  • GANs and VAEs: Computation burden is primarily in training; inference is fast.
  • Diffusion Models: Training speed is comparable to others, but inference is slower due to iterative denoising, making them suitable for offline deployment.
  • Differential Equations: Naturally burdened by larger computation graphs due to solving initial value problems.

5. Significance and Future Directions

The paper claims that GenTS fills a critical void in the generative time series community by providing a unified platform for systematic assessment. The authors emphasize that diffusion-based models are currently the most robust and advisable choice for general-purpose time series generation, particularly for conditional tasks.

The authors modestly suggest three potential future research directions:

1. Generative Time Series Foundation Models: Developing models trained on global data distributions to be inferred controllably for diverse downstream tasks.
2. Robust Evaluation: Addressing the sensitivity of model-based metrics (PS, DS) to the choice of auxiliary evaluator models by developing more holistic metrics.
3. Online Development: Creating adaptive paradigms for dynamic, real-world time series with complex distribution shifts, moving beyond static environments to reduce latency and computational burdens.

The authors conclude that while differential equation-based models underperformed in current benchmarks, their inherent inductive bias for continuous time suggests they remain a promising, under-explored area.