ConTSG-Bench: A Unified Benchmark for Conditional Time Series Generation

This paper introduces ConTSG-Bench, a unified benchmark featuring a large-scale, multi-modal dataset and comprehensive metrics to systematically evaluate and analyze the performance, limitations, and future directions of conditional time series generation models.

The Gist

Imagine you are a master chef. In the past, chefs could only cook "random" dishes. They would throw ingredients into a pot, stir, and hope for a delicious soup. This is like unconditional time series generation: creating data (like stock prices or weather patterns) that looks real but has no specific instructions.

But now, we want Conditional Time Series Generation. This is like a customer walking in and saying, "I want a soup that is spicy, has no carrots, and tastes like summer." The chef must create a dish that is not only delicious (realistic) but also follows those specific rules (the condition).

The paper "ConTSG-Bench" is about building a giant, standardized Taste Test to see which chefs (AI models) are actually good at following these complex orders.

Here is the breakdown of their work using simple analogies:

1. The Problem: A Messy Kitchen

Before this paper, the field of "Conditional Generation" was chaotic.

• Different Languages: Some chefs only understood "Class Labels" (e.g., "Make a soup"). Others understood "Attributes" (e.g., "Make a spicy soup"). A few could understand full sentences like "Make a spicy soup with no carrots."
• Different Ingredients: Every chef was tested on their own unique set of ingredients. You couldn't compare Chef A's spicy soup to Chef B's spicy soup because they were made with different vegetables.
• The Result: We didn't know who was actually the best chef, or if the chefs could handle complex, abstract instructions like "Make a soup that feels like a rainy Tuesday."

2. The Solution: ConTSG-Bench (The Ultimate Taste Test)

The authors built a unified kitchen called ConTSG-Bench. It's a massive, organized testing ground with three main features:

• The "All-in-One" Menu: They created datasets where the same data (the soup) is described in three different ways:

  1. A Label: "Spicy."
  2. A List of Attributes: "Spicy, No Carrots, Hot."
  3. A Sentence: "A hot, spicy soup without carrots."
     This allows them to test if a model can handle all these different ways of giving instructions.

• The "Literal vs. Abstract" Challenge: They tested two levels of difficulty:

  • Morphological (Literal): "Make the soup curve up, then down, then up." (Describing the shape).
  • Conceptual (Abstract): "Make a soup that feels like a storm." (Describing a high-level idea).
  • The Analogy: It's the difference between telling a robot to "move your arm up 5 inches" vs. "wave hello." The second one requires the robot to understand the concept of a wave and figure out the arm movement itself.

• The Scorecard: They didn't just ask, "Does it taste good?" They asked two questions:

  1. Fidelity: Is the soup actually soup? (Does it look like real data?)
  2. Adherence: Did you actually follow the order? (Is it spicy? Are there carrots?)
  • Key Finding: Many chefs made delicious soup, but it was the wrong flavor. Others made the right flavor, but it tasted like cardboard.

3. The Big Discoveries (What the Taste Test Revealed)

After testing 10 different "chefs" (AI models), they found some surprising things:

• The "Text" Chefs are the Most Talented (but inconsistent): Models that understand full sentences (like "Make a soup like a storm") have the highest potential. They can create the most complex dishes. However, they are also the most unpredictable. Some are geniuses; others are disasters.
• The "Fine-Grained" Struggle: This is the biggest failure. If you ask a chef, "Make the soup spicy in the first half, but mild in the second half," most models fail. They can't control the "local" details. They tend to make the whole bowl spicy or the whole bowl mild.
  • Analogy: It's like a painter who can paint a beautiful sunset, but if you ask them to paint a tiny, specific bird in the corner, they just paint a blob.
• The "New Combinations" Problem: If you train a chef on "Spicy Soup" and "Cold Soup," and then ask for "Spicy Cold Soup," many chefs freeze. They can't combine the rules they learned to make something new. They just memorized the old recipes.
• The "Usefulness" Test: Finally, they asked: "If we use this fake soup to train a new chef, will it help?" Sometimes, yes. Sometimes, the fake soup is so weird that it confuses the new chef.

4. Why Does This Matter?

Imagine you are a doctor. You have very few patient records (data scarcity). You want an AI to generate fake patient records that look real so you can train a diagnostic tool.

• If the AI generates records that look real but don't match the specific disease you are studying (bad adherence), your diagnostic tool will fail.
• If the AI can't handle complex instructions (like "generate a record for a patient with diabetes AND high blood pressure"), you can't simulate rare conditions.

ConTSG-Bench is the tool that tells us which AI models are ready for the real world and which ones are just playing pretend. It highlights that while we are getting better at making "fake data," we still need to teach these models how to follow precise, complex, and abstract instructions reliably.

Summary

• The Goal: Create a standard way to test AI that generates data based on instructions.
• The Tool: A massive benchmark with diverse instructions (labels, lists, sentences) and difficulty levels (shapes vs. concepts).
• The Verdict: Current AI is getting good at making things look real, but it's still terrible at following specific, detailed, or abstract instructions. We need better "chefs" before we can trust them with critical tasks like healthcare or climate modeling.

Technical Summary

1. Problem Statement

Conditional Time Series Generation (ConTSG) is critical for applications in healthcare, climate science, and finance, where synthetic data must adhere to specific user-defined conditions (e.g., "simulate an ECG with atrial fibrillation" or "generate traffic data with a sudden drop"). However, the field faces three major challenges:

• Fragmentation: Existing methods are isolated by their conditioning modalities (discrete labels, structured attributes, or natural language) and evaluated on incompatible datasets, preventing systematic comparison.
• Lack of Standardization: There is no unified framework to evaluate models across diverse condition types and semantic abstraction levels (e.g., direct morphological descriptions vs. high-level conceptual descriptions).
• Incomplete Evaluation: Prior benchmarks often focus solely on generation fidelity (how realistic the data looks) while neglecting condition adherence (how well the data matches the prompt), fine-grained local control, and compositional generalization (handling unseen attribute combinations).

2. Methodology: The ConTSG-Bench Framework

The authors introduce ConTSG-Bench, a unified benchmark designed to disentangle and evaluate ConTSG models across two primary axes:

1. Condition Modality: Class labels, structured attributes, and natural language text.
2. Semantic Abstraction Level:
   • Morphological: Direct descriptions of temporal structures (e.g., "upward trend with a peak").
   • Conceptual: High-level domain semantics requiring inference (e.g., "heart failure" or "stormy weather").

A. Dataset Construction

The benchmark comprises eight large-scale datasets spanning healthcare (PTB-XL), meteorology (Weather), energy (ETTm1), traffic, and network telemetry.

• Alignment Strategy: A key innovation is the creation of aligned conditions. For the same underlying time series, the benchmark provides a class label, a structured attribute vector, and a natural language description.
• LLM Pipeline: The authors use Large Language Models (LLMs) to generate morphological captions from raw time series, extract structured attribute schemas, and map these to class labels, ensuring consistency across modalities with minimal manual effort.
• Paired Conditions: For specific datasets (PTB-XL, Weather), they provide paired morphological and conceptual descriptions to enable controlled cross-abstraction comparisons.

B. Evaluation Protocol

The benchmark assesses models along five research questions (RQs) using a comprehensive suite of metrics:

1. Overall Benchmarking (RQ1): Evaluates Generation Fidelity (statistical realism) and Condition Adherence (alignment with the prompt) separately.
   • Metrics: Fréchet Inception Distance (FID), Precision/Recall, CTTP Score (contrastive alignment), and Joint FID.
2. Semantic Abstraction (RQ2): Compares performance on morphological vs. conceptual conditions using reference-based metrics (DTW, CRPS) to avoid bias from text embedding differences.
3. Fine-Grained Control (RQ3): Tests the ability to follow local constraints (e.g., "peak in segment 2, drop in segment 3") using segment-level classifiers and retrieval accuracy.
4. Compositional Generalization (RQ4): Evaluates performance on novel attribute combinations not seen during training, measuring robustness to out-of-distribution (OOD) conditions.
5. Practical Utility (RQ5): Measures if generated data can substitute for real data in downstream tasks (e.g., training classifiers) via a "Drop Rate" metric.

C. Evaluated Models

The benchmark evaluates 10 representative models covering all three modalities:

• Label-based: TimeVQVAE, TTS-CGAN.
• Attribute-based: TimeWeaver, TEdit, WaveStitch.
• Text-based: BRIDGE, VerbalTS, T2S, DiffuSETS, Text2Motion.

3. Key Contributions

1. Unified Benchmarking Framework: The first systematic evaluation protocol for ConTSG that covers diverse modalities and semantic levels, enabling fair cross-method comparison.
2. Multimodal Aligned Datasets: Construction of large-scale datasets where a single time series instance is paired with label, attribute, and text conditions, solving the data scarcity and alignment issues in prior work.
3. Comprehensive Analysis: An in-depth characterization of state-of-the-art models revealing critical bottlenecks, such as the trade-off between fidelity and adherence, and the difficulty of fine-grained control.
4. Open Source: All code, datasets, and evaluation pipelines are publicly released to facilitate reproducibility.

4. Experimental Results & Insights

The evaluation of 10 models across 8 datasets yielded several pivotal findings:

• Fidelity vs. Adherence are Orthogonal: High generation fidelity does not guarantee condition adherence. Some models generate realistic data that ignores the condition, while others follow the condition but produce implausible patterns.
• Text Conditioning is a Double-Edged Sword: Text-conditioned models (e.g., VerbalTS) achieve the highest performance ceiling but exhibit the largest variance across architectures. They offer rich expressiveness but require robust alignment mechanisms.
• Fine-Grained Control is a Major Bottleneck: Most models fail to achieve precise local control. On segment-level tasks, many generators perform no better than simple retrieval baselines, failing to preserve intended temporal orders or specific local patterns.
• Compositional Generalization is Weak: Models struggle with novel attribute combinations. Interestingly, models with strong condition adherence often show greater sensitivity to OOD combinations, suggesting they rely on memorized patterns rather than compositional understanding.
• Downstream Utility is Unpredictable: The utility of generated data for downstream classifiers varies significantly by dataset and cannot be reliably predicted by generation fidelity metrics alone. Some models even degrade downstream performance (Drop Rate > 1).

5. Significance and Future Directions

ConTSG-Bench establishes a new standard for the field, shifting the focus from simple unconditional generation to controllable, multimodal synthesis.

• Impact: It highlights that current methods lack the structural inductive biases necessary for complex real-world synthesis.
• Future Work: The authors suggest future research should focus on:
  • Architectures with compositional inductive biases to handle novel attribute combinations.
  • Segment-aware objectives to improve fine-grained local control.
  • Domain-agnostic generalization strategies to ensure robustness across diverse time series domains.

By providing a rigorous, unified framework, ConTSG-Bench aims to accelerate the development of reliable conditional time series generators for safety-critical applications.