BEDTime: A Unified Benchmark for Automatically Describing Time Series

1. Problem Statement

Recent advancements in multi-modal models have claimed high performance on complex time series tasks, such as reasoning and cross-modal question answering. However, the authors argue that these claims often skip foundational evaluations. Specifically, there is a lack of rigorous benchmarks to determine if models can perform basic tasks like recognizing, differentiating, and generating natural language descriptions of univariate time series structural properties (e.g., trends, seasonality, anomalies).

Current limitations in the field include:

• Lack of Head-to-Head Comparisons: Models are often evaluated on isolated, specialized datasets without direct comparison against other architectures.
• Over-reliance on Complex Tasks: Benchmarks focus on high-level reasoning (e.g., forecasting, causal inference) rather than fundamental capabilities. If a model cannot describe a simple trend, it cannot be trusted on complex reasoning.
• Inadequate Evaluation of Modalities: There is insufficient understanding of whether Vision-Language Models (VLMs), Large Language Models (LLMs), or dedicated Time Series Language Models (TSLMs) are best suited for interpreting time series data.

2. Methodology: The BEDTime Benchmark

The authors introduce BEDTime, the first unified benchmark designed to assess models on three fundamental tasks using natural language descriptions of univariate time series.

A. Dataset Construction

BEDTime unifies five public datasets into a single framework, comprising 46,843 time series–description pairs (90.44% real-world, 9.56% synthetic). The datasets cover diverse domains:

• TRUCE-Stock: Real-world, noisy financial data with crowd-sourced descriptions.
• TRUCE-Synthetic: Short, simple synthetic data with crowd-sourced and template descriptions.
• SUSHI: Long, complex synthetic signals with prominent trends/seasonality and template descriptions.
• TaxoSynth: Variable-length taxonomic patterns with synthetic descriptions.
• NICU-HR: Real-world neonatal heart-rate signals with clinical event descriptions.

The data is reformatted to support three input modalities:

1. Text: Comma-separated numeric values.
2. Image: Matplotlib-rendered plots.
3. Vector: Raw numerical arrays (for TSLMs).

B. Proposed Tasks

The benchmark evaluates models on three specific capabilities:

1. Description Recognition (True/False): Given a time series and a description, determine if the description is consistent with the data.
2. Description Differentiation (Multiple Choice): Given a time series and four options (one correct, three distractors), select the correct description. Distractors are generated using four distinct strategies (Sentence-BERT, DTW, Euclidean Distance, LCSS) to ensure robustness.
3. Open Generation: Generate a natural language description of the time series' structural properties.

C. Evaluation Metrics

• Recognition/Differentiation: Accuracy and F1 scores.
• Generation:
  • Automated: Natural Language Inference (NLI) entailment (bidirectional) using DeBERTa and GPT-5.
  • Human: Expert evaluation based on six criteria: Coherence, Pattern Identification, Temporal Localization, Noise Filtering, Abstraction, and Linguistic Quality.

D. Experimental Setup

The authors evaluated 17 state-of-the-art models across three categories:

• LLMs: GPT-4o/5.4-mini, Gemini-2.0/3.0, Llama-3.1, Qwen2.5, Phi-3.5.
• VLMs: Vision variants of the above (e.g., GPT-4o-Vision, Qwen2.5-VL).
• TSLMs: ChatTime and ChatTS (dedicated time series foundation models).

3. Key Results

A. Model Performance Hierarchy

1. Vision-Language Models (VLMs) Dominate: VLMs consistently outperformed all other modalities across all datasets and tasks. They achieved near-perfect accuracy on synthetic data (e.g., SUSHI) and significantly higher accuracy on real-world data compared to text-only models. This suggests that visual representation of time series is crucial for understanding structural properties.
2. LLMs are Surprisingly Weak: Despite their popularity in time series analysis, text-only LLMs performed poorly, often lagging behind VLMs. Smaller open-weight models (e.g., Llama-3.1-8B, Phi-3.5) frequently fell below random baselines on noisy or long sequences.
3. TSLMs are Mixed: Dedicated TSLMs (ChatTS, ChatTime) generally outperformed comparably sized LLMs but still lagged behind VLMs.
   • ChatTS showed strong performance on differentiation tasks, rivaling larger LLMs.
   • ChatTime performed poorly (often below random) due to high refusal rates; it frequently failed to output discrete classification labels (e.g., "A" or "True"), instead returning raw numbers from the input.

B. Robustness to Perturbations

The study tested model resilience against realistic perturbations:

• Sequence Length: LLM performance degraded significantly as sequence length increased (due to attention diffusion and tokenization limits). TSLMs were more robust to length increases.
• Missing Data: All models suffered performance drops with missing values, but LLMs degraded sharply once >25% of data was masked.
• Noise: Performance declined as Gaussian noise increased. VLMs showed the highest robustness, while LLMs dropped below random baselines on noisy signals.
• Image Quality: VLM accuracy decreased monotonically as plot resolution (pixelation) degraded, though GPT vision variants remained robust to moderate degradation.
• Chain-of-Thought (CoT): CoT prompting improved LLM performance slightly (mean +3% accuracy), primarily by reducing refusal rates and mitigating positional bias, but it did not close the gap with VLMs.

C. Generation Quality

In open-ended generation tasks, VLMs produced descriptions with higher semantic alignment to ground truth (measured by NLI) and better human-rated scores for pattern identification and temporal localization compared to text-only models.

4. Key Contributions

1. BEDTime Benchmark: The first unified benchmark for automatically describing time series, unifying five datasets and three modalities (text, image, vector) into a single evaluation framework.
2. Comprehensive Evaluation: A head-to-head comparison of 17 models (LLMs, VLMs, TSLMs) revealing that visual encoders are currently the most effective modality for time series description, challenging the assumption that text-only LLMs are sufficient.
3. Robustness Analysis: Extensive testing demonstrating that current foundation models are fragile to real-world perturbations (noise, missing data, length), highlighting a critical gap between synthetic benchmarks and real-world deployment.
4. Open Resources: Full release of code, data, and evaluation protocols to facilitate reproducibility and future research.

5. Significance and Future Directions

• Paradigm Shift: The paper challenges the current trend of using text-only LLMs for time series tasks, suggesting that visual representations (plots) are essential for capturing structural nuances that numeric tokenization misses.
• Foundation for Reasoning: By establishing that models struggle with basic description tasks, the authors argue that complex reasoning capabilities (forecasting, anomaly detection) cannot be trusted until these foundational skills are mastered.
• Future Work: The authors call for:
  • Development of models that explicitly model structural features rather than relying on generic tokenization.
  • Improved robustness to noise and missing data.
  • Extension of benchmarks to multivariate time series.
  • Better instruction-following capabilities in dedicated TSLMs to avoid refusal behaviors.

In conclusion, BEDTime serves as a critical diagnostic tool, revealing that while multi-modal models (specifically VLMs) show promise, the field of time series language modeling is still in its infancy regarding fundamental understanding and robustness.