Technical Summary: Towards Verifiable Agentic Data Science: Solving Irregular TSQA via Tool-Grounded Reasoning

1. Problem Statement

Real-world time series data is predominantly irregular: observations are asynchronous, missing values are often informative rather than random, and sampling frequencies vary across sensors and operational windows. Despite the rapid advancement of Large Language Models (LLMs) and AI agents in complex reasoning, existing Time Series Question Answering (TSQA) benchmarks (e.g., TSAQA, Time-MQA) largely assume regularly sampled inputs. This creates a fundamental evaluation gap, as current models are not rigorously tested on the semantic and structural challenges posed by irregular data.

Furthermore, existing methods for generating irregular time series often rely on synthetic transformations like random dropout (MCAR) or sparse masking, which fail to capture the domain semantics and specific causes of missingness found in real-world scenarios (e.g., ICU monitoring triggered by clinical necessity or industrial sensors logging only upon anomaly detection). The paper notes that while recent work (TIME-IMM) has established a taxonomy of irregularity, no prior pipeline has utilized this taxonomy to semantically ground the transformation from regular to irregular data for benchmarking purposes.

2. Methodology

The authors introduce IRTS-ToolBench, a benchmark designed to evaluate LLMs and AI agents on irregular univariate TSQA through tool-grounded reasoning. The methodology is built upon three core components:

A. LLM-Guided Irregular Transformation Pipeline

To bridge the gap between regular and irregular data, the authors propose a three-layer pipeline that transforms regular time series into semantically grounded irregular sequences:

1. Context Enrichment: An LLM generates a detailed context description for each time series, capturing domain, statistical features, and signal characteristics.
2. Taxonomy Selection: Based on the enriched context, an LLM selects the most appropriate irregularity type from a nine-type taxonomy (categorized as Trigger-Based, Constraint-Based, and Artifact-Based, following TIME-IMM) and outputs a transformation plan with anomaly annotations.
3. Parameter Generation: The LLM translates the plan into validated numerical parameters for transformation functions, producing irregular sequences with realistic missingness patterns and jittered timestamps.

B. Benchmark Construction

IRTS-ToolBench comprises 1,700 questions spanning 10 task types across 13 domains (e.g., finance, healthcare). The tasks are organized into a progressive reasoning hierarchy:

• Standard Reasoning: Anomaly Detection, Classification, Regular vs. Irregular Discrimination, and Regularity Recovery.
• Irregularity-Specific Reasoning: Characterization (temporal properties), Temporal Relationship (ordering/causality), and Irregularity Cause Attribution.
• Regularity–Irregularity Interface Reasoning: Missingness Reasoning (informative vs. non-informative gaps), Irregularity Severity Estimation, and Forecasting.

C. Tool Library and Golden Tool Sets

The benchmark includes a 30-tool library divided into two layers:

1. Irregularity Operators (7 tools): Handle specific irregularities (e.g., ALIGN ASOF, MISSING POLICY, RESAMPLE, GAP DETECT).
2. Advanced Analytical Tools (23 tools): Provide analysis primitives (e.g., TREND CLASSIFIER, SEASONALITY DETECTOR, FORECASTER, INTERPOLATE).

For each benchmark sample, a Golden Tool Set is constructed via a multi-LLM consensus protocol (using GPT-5.1, Claude Sonnet 4.5, and Gemini 2.5 Flash) to define the minimum set of tools required to answer a question correctly. This enables the evaluation of both direct LLM prompting and agentic tool-use frameworks.

3. Key Contributions

1. Novel Transformation Pipeline: A semantically grounded pipeline for converting regular time series into irregular ones, moving beyond random dropout to domain-aware irregularity generation.
2. Comprehensive Benchmark (IRTS-ToolBench): A dataset of 1,700 questions across 10 task types and 13 domains, specifically designed for irregular univariate TSQA.
3. Tool-Grounded Evaluation Framework: A 30-tool library and a protocol for constructing "Golden Tool Sets," allowing for the simultaneous evaluation of answer correctness and the reasoning behavior of agents regarding tool invocation.

4. Evaluation Results

The authors evaluated zero-shot performance on several commercial and open-source LLMs (including Claude-Opus-4.7, Qwen3.5-4B, Qwen3.6-27B, and DeepSeek-V4-Flash).

• Overall Performance: Qwen3.6-27B achieved the highest overall accuracy (78.59%), outperforming other models. Commercial models like Claude-Opus-4.7 showed consistent accuracy between 74% and 77%.
• Impact of Tool Calling: Tool calling provided significant improvements for tasks requiring explicit numerical analysis. For instance, Qwen3.6-27B improved from 96.80% to 99.60% on Anomaly Detection and reached 100% on Classification with tools. DeepSeek-V4-Flash showed massive gains on Irregularity Severity Estimation (31.33% to 98.67%) and Regularity Recovery (64.67% to 89.33%) when tools were enabled.
• Remaining Challenges: Despite tool augmentation, higher-level temporal reasoning tasks, such as Temporal Relationship reasoning and Regular-vs-Irregular Discrimination, remained difficult, with performance varying significantly across models.
• Human Baseline: Human reviewers achieved accuracies of 80% and 78%, indicating that while most questions are understandable, tasks like Regularity Recovery and Missingness Reasoning remain challenging even for humans.

5. Significance and Claims

The paper positions IRTS-ToolBench as a necessary step toward "verifiable agentic data science." Its significance lies in:

• Bridging the Evaluation Gap: It addresses the critical limitation that existing benchmarks ignore the irregular nature of real-world time series.
• Standardization: It provides standardized inputs and a reproducible evaluation protocol for researchers working on LLM-based irregular time series analysis.
• Behavioral Insight: By utilizing Golden Tool Sets, the benchmark allows researchers to study not just if a model gets the answer right, but how it reasons (i.e., whether it correctly identifies and invokes the necessary tools to handle irregularity).

The authors conclude that while current LLMs exhibit non-trivial reasoning abilities over irregular time series—especially when supported by context and tools—they still struggle with higher-level temporal reasoning and require strong scaffolding for reliable agentic tool use. They acknowledge limitations in their LLM-based generation pipeline, noting potential sensitivity to prompting and consensus quality, and suggest future work should focus on broader validation and incorporating more naturally irregular source data.