CaTS-Bench: Can Language Models Describe Time Series?

1. Problem Statement

The paper addresses the challenge of Time Series Captioning (TSC), the task of generating natural language descriptions for time series data. While Large Language Models (LLMs) and Vision-Language Models (VLMs) have advanced in general text generation, they struggle with numeric reasoning, temporal alignment, and grounded multimodal interpretation specific to time series.

Existing benchmarks suffer from three critical limitations:

1. Synthetic/Template Bias: Most datasets rely on simplistic, template-based synthetic captions that lack the linguistic richness and nuance of human writing.
2. Lack of Context: They often omit crucial metadata (e.g., units, domain labels, historical context) required for grounded interpretation.
3. Data Scarcity: There is a severe lack of high-quality, human-annotated datasets to evaluate whether models can translate numeric trends into interpretable narratives.

2. Methodology

The authors introduce CaTS-Bench, a comprehensive, context-aware multimodal benchmark designed to evaluate TSC across 11 diverse real-world domains (e.g., climate, healthcare, finance, agriculture).

A. Data Construction

• Base Data: The benchmark utilizes 11 source datasets containing 287 million time steps. Data is processed into triplets: (1) raw numeric time series, (2) structured metadata (units, location, statistical primitives), and (3) visual line plots with randomized styles.
• Human-Rewritten (HR) Gold Standard: To avoid oracle bias, the authors curated 1,746 human-rewritten captions. These were generated by four distinct LLMs and then extensively edited and verified by humans to ensure factual accuracy, linguistic diversity, and rich narrative structure.
• Scalable Synthetic Pipeline: To address data scarcity for training, the authors developed a pipeline using an "Oracle" LLM (Gemini 2.0 Flash) grounded in strict metadata and precomputed statistics. This generated ~20,000 high-fidelity synthetic captions.
  • Validation: The synthetic data was validated via manual fact-checking (98.6% accuracy), human indistinguishability studies (41.1% detection rate, near random), and linguistic diversity analysis (low template repetition).

B. Task Definition

CaTS-Bench defines two primary evaluation tracks:

1. Time Series Captioning (TSC): Models receive a multimodal prompt (numeric series, metadata, line plot) and must generate a free-form narrative.
2. Diagnostic Q&A: A suite of 910 multiple-choice questions (MCQs) designed to isolate specific reasoning bottlenecks:
   • Time Series Matching: Identifying the correct series from distractors.
   • Caption Matching: Selecting the correct description for a series.
   • Plot Matching: Linking a caption to the correct visual plot.
   • Comparative Reasoning: Comparing statistical properties (variance, peaks) between series.
   • Temporal Matching: Exact index alignment (finding a value at a specific timestamp).

C. Evaluation Metrics

The paper introduces tailored metrics beyond standard NLP scores (BLEU, ROUGE):

• Numeric Fidelity Score: Measures the precision and recall of numeric values extracted from the caption against the ground truth.
• Statistical Inference Accuracy: Evaluates the correctness of inferred statistics (mean, min, max, std) with a 5% relative error tolerance.
• Visual Grounding Analysis: Uses attention map visualization to determine if models actually "look" at the plot or rely on textual priors.

3. Key Contributions

1. CaTS-Bench Benchmark: The first unified benchmark integrating numeric data, visual plots, rich metadata, and human-rewritten captions across 11 domains. It includes a gold-standard HR set and a diagnostic MCQ suite.
2. Scalable Augmentation Pipeline: A proven method for generating high-fidelity synthetic training data using metadata-grounded LLMs, which effectively bridges the data scarcity gap for TSC.
3. Diagnostic Framework: A rigorous evaluation suite revealing systematic failures in current VLMs regarding exact temporal indexing and multimodal integration.

4. Key Results

The authors evaluated leading proprietary (e.g., GPT-4o, Gemini 2.0 Flash) and open-source VLMs (e.g., LLaVA, Idefics, QwenVL).

• Performance Gaps:

  • Proprietary vs. Open-Source: Proprietary models generally outperform open-source counterparts in zero-shot settings.
  • Impact of Finetuning: Finetuning open-source models on the synthetic data yielded substantial gains, narrowing the gap with proprietary models, particularly in semantic and numeric metrics.
  • Human Baseline: Humans significantly outperform all models, especially in Plot Matching (where models perform near random chance) and Temporal Matching.

• Specific Failure Modes:

  • Visual Grounding: Models largely ignore the visual plot, relying instead on textual metadata and priors. Removing the plot image often had negligible impact on performance, while adding explicit numeric annotations to the plot improved results significantly.
  • Temporal Reasoning: Models struggle with exact index alignment. Accuracy degrades as the query moves further from the series start.
  • Hallucination: Models frequently hallucinate statistical values (e.g., incorrect means or standard deviations) even when they attempt to report them.
  • Multimodal Interference: In some cases, providing both text and image inputs degraded performance compared to image-only inputs, suggesting models struggle to fuse redundant modalities effectively.

• Statistical Inference:

  • Mean/Max: Relatively robust inference for mean and maximum values in stronger models.
  • Standard Deviation: A critical failure point; models consistently fail to infer standard deviation accurately, often hallucinating or omitting it entirely.

5. Significance and Conclusion

CaTS-Bench establishes that while LLMs/VLMs are capable of generating fluent text, they lack grounded numeric reasoning and true multimodal integration for time series data.

• Practical Utility: The benchmark highlights that automating time series interpretation requires more than just text generation; it demands precise numeric grounding and the ability to synthesize visual and textual cues.
• Future Directions: The paper suggests that current pretraining paradigms are ill-suited for structured visual reasoning over time series. Future work must focus on architectures that better integrate plot-based information with textual and numeric cues, moving beyond reliance on textual priors.
• Resource Release: The authors release the dataset, synthetic generation pipeline, and diagnostic suite to foster research in grounded, multimodal text generation for numeric domains.

In summary, CaTS-Bench serves as a critical diagnostic tool, exposing the "blind spots" of current AI in numeric domains and providing a foundation for developing more reliable, data-grounded generative models.