Technical Summary: TSAQA: Time Series Analysis Question And Answering Benchmark

Problem Statement

While Large Language Models (LLMs) have revolutionized natural language processing, their application to time series analysis has largely been confined to traditional numerical tasks such as forecasting, imputation, and anomaly detection. Existing benchmarks for Time Series Question Answering (TSQA) suffer from fragmented task coverage, limited modality, and non-standardized evaluation protocols. Furthermore, many existing datasets rely on open-ended answers that are difficult to evaluate objectively, or they focus on narrow domains. There is a critical need for a unified, large-scale benchmark that comprehensively evaluates the temporal analytical capabilities of language-capable models across diverse domains and task complexities, ranging from fundamental pattern recognition to advanced relational reasoning.

Methodology

Benchmark Construction

The authors introduce TSAQA, a unified benchmark comprising 210,000 high-quality samples curated from 13 distinct domains (including finance, healthcare, energy, transportation, and web traffic). The dataset is constructed from core time series repositories (e.g., Lotsa, Time-300B, UTSD), specialized anomaly detection benchmarks (e.g., ECG, SMD), and classification archives (UCR Archive).

The benchmark integrates 6 distinct tasks categorized into two groups:

1. Conventional Analysis:
   • Anomaly Detection: Identifying irregular patterns (True/False).
   • Classification: Categorizing time series based on semantic patterns (Multiple Choice).
2. Advanced Analysis:
   • Characterization: Inferring intrinsic properties like trend, seasonality, and dispersion.
   • Comparison: Analyzing relative similarities and differences between two time series.
   • Data Transformation: Understanding relationships between raw data and transformed data (Fourier, Wavelet, First-Order Differencing).
   • Temporal Relationship: Determining chronological dependencies among time series patches.

Question Formats

To ensure objective and reproducible evaluation, all tasks are cast into a standardized Question Answering (QA) format with three specific types:

• True-or-False (TF): Binary verification of a claim.
• Multiple-Choice (MC): Selecting the correct claim from options.
• Puzzling (PZ): A novel format where models are given the first patch of a sequence and a set of shuffled subsequent patches, requiring them to reconstruct the correct chronological order. This tests global causal reasoning and structural continuity.

Data Processing and Annotation

• Preprocessing: Time series are standardized via z-score normalization to prevent models from exploiting magnitude-based shortcuts across different domains. Input lengths vary between 32 and 256 points.
• Generation: For advanced tasks (Characterization, Comparison), QA pairs are generated using a multi-LLM consensus pipeline. GPT-4o generates initial questions and answers based on selected topics (e.g., trend, noise) and sub-topics. These are then validated and aggregated using a weighted voting scheme involving GPT-4.1, Gemini-2.5-Flash, and Claude-3.5-Sonnet to minimize single-model bias.
• Sampling: A hierarchical random sampling strategy ensures a balanced distribution across domains, datasets, and sequences.

Evaluation Protocol

The benchmark evaluates models using accuracy for TF and MC questions. For PZ questions, accuracy is calculated based on the proportion of correctly matched positions in the reconstructed sequence compared to the ground truth. The study evaluates:

• Commercial LLMs: GPT-4.1, GPT-4o, Claude-3.5-Sonnet, Gemini-2.5-Flash.
• Open-Source LLMs: Qwen3-8B/0.6B, LLaMA-3.1-8B/3.2-1B, Ministral-8B, Gemma3-1B.
• Time Series Foundation Models (TSFMs): ChatTime, ChatTS.
• Training: Open-source models are evaluated in zero-shot settings and after instruction tuning (SFT) using LoRA.

Key Results

Overall Performance

• Zero-Shot Limitations: Current LLMs struggle significantly with TSAQA. The best-performing commercial model, Gemini-2.5-Flash, achieves an average accuracy of 65.08%. Open-source models generally perform worse in zero-shot settings, with LLaMA-3.1-8B scoring 44.93%.
• Impact of Instruction Tuning: Supervised fine-tuning (SFT) yields substantial improvements. Notably, LLaMA-3.1-8B (SFT) reaches 85.26%, surpassing the zero-shot performance of commercial models. Gemma3-1B (SFT) even outperforms Gemini-2.5-Flash (69.70% vs. 65.08%), demonstrating that targeted training can bridge the gap between open-source and proprietary models.
• TSFMs: Language-capable time series foundation models (ChatTime, ChatTS) show mixed results, with ChatTime achieving 55.19%, indicating that general-purpose LLMs currently hold an advantage in reasoning over these specific benchmarks.

Task-Specific Findings

• Conventional vs. Advanced: Models perform relatively well on Anomaly Detection and Classification after tuning (e.g., LLaMA-3.1-8B reaches >91% on these). However, Advanced Analysis tasks remain challenging.
• The "Puzzling" Challenge: The PZ task proves to be the most difficult, with the best score (LLaMA-3.1-8B SFT) reaching only 67.68%.
• Input Length Correlation: Unlike most tasks where accuracy declines with longer input lengths, the Temporal Relationship (PZ) task shows a positive correlation between input length and accuracy. This suggests that models are effectively utilizing global context to deduce chronological order rather than relying on local pattern matching.
• Domain Difficulty: Zero-shot performance is lowest in Synthetic, IT, Robotics, and Web domains. Even after tuning, Sales and Web domains remain the most difficult, suggesting domain-specific complexities are not fully mitigated by general instruction tuning.

Error Analysis

• Smoothness Bias: In the PZ task, models exhibit a "smoothness bias," particularly in volatile domains like Web and Sales. They tend to select patch transitions that are smoother than the ground truth, failing to capture legitimate discontinuities and irregular dynamics.
• Transformation Limits: Models struggle significantly with Fourier Transform (FT) and Wavelet Transform (WT) tasks, which require global aggregation and complex arithmetic reasoning, whereas First-Order Differencing (FOD) is handled more effectively due to its reliance on local dependencies.

Significance and Contributions

The paper claims three primary contributions:

1. A Novel Large-Scale Benchmark: TSAQA provides a unified, standardized platform with 210k samples across 13 domains, 6 tasks, and 3 question types, addressing the fragmentation in existing TSQA datasets.
2. Comprehensive Construction and Statistics: The authors detail a rigorous data collection and annotation process, including a multi-LLM consensus pipeline and hierarchical sampling, ensuring transparency and reproducibility.
3. Extensive Evaluation and Analysis: The benchmark serves as a rigorous probe for the temporal reasoning capabilities of both general-purpose LLMs and specialized TSFMs. The results highlight that while instruction tuning improves performance, significant challenges remain, particularly in advanced analytical reasoning and handling complex temporal dynamics (e.g., volatility, global transformations).

The authors position TSAQA not merely as an evaluation tool but as a foundational resource for developing future language-capable time series reasoning models, potentially extending toward multivariate settings, multimodal inputs, and agentic systems. They acknowledge limitations, noting that the benchmark does not fully capture irregular sampling or exogenous drivers found in some real-world scenarios, and that the static nature of the dataset may not reflect evolving distributions.