Technical Summary: TimeSage-MT

Problem Statement

Time series analysis is critical for decision-making in high-stakes domains such as finance, healthcare, and energy. While Large Language Model (LLM) agents demonstrate capabilities in natural language interaction and tool use, their ability to conduct reliable, multi-turn time series reasoning remains unverified. Existing benchmarks (e.g., Monash, M4, UCR/UEA) focus on isolated, single-step tasks like forecasting or anomaly detection with fixed objectives. They fail to evaluate the complex analytical workflows required in practice, where agents must inspect data quality, select methods, chain dependent computations, adapt to changing user goals, and ground conclusions in domain knowledge. Current evaluations often conflate shallow pattern matching or hallucinated numbers with genuine reasoning, lacking the granularity to diagnose failures in memory, uncertainty calibration, or domain-grounded decision-making.

Methodology

Benchmark Construction (TimeSage-MT)

The authors introduce TimeSage-MT, a multi-turn benchmark designed to evaluate agentic time series reasoning across 240 tasks and 2,680 dialogue turns. The benchmark spans 8 real-world domains (Healthcare, Energy, Manufacturing, Climate, Finance, Telecom, Transportation, Retail) and is organized into four complexity levels:

• L1 (Open Exploration): 3–10 turns for profiling and summarization.
• L2 (Multi-Skill Analysis): 5–12 turns requiring chained analytical steps.
• L3 (Grounded Synthesis): 8–15 turns integrating evidence into domain-specific reports.
• L4 (Full Decision Path): 12–20 turns culminating in structured, actionable decisions with calibrated uncertainty.

Data Generation Pipeline:
The benchmark is constructed via a reproducible, five-stage pipeline:

1. Data Selection: Curating 65 whitelisted real-world sources, with metadata tracking potential contamination from foundation model pretraining corpora.
2. Profiling: Generating an 11-metric profile (frequency, stationarity, missingness, etc.) and splitting data into 80% visible (for construction) and 20% held-out (for gold target extraction).
3. Task Assignment: Assigning tasks to levels and domains based on profile feasibility, ensuring balanced quotas.
4. Reasoning Path Construction: Creating a typed Directed Acyclic Graph (DAG) for each task, defining the sequence of analytical nodes, dependencies, and expected outputs.
5. Dialogue Generation: Using an LLM (Claude Opus 4.6) to generate multi-turn dialogues grounded in the visible data and reasoning path. Crucially, verifiable answer targets (numerical ground truth) are computed deterministically by running reference code on the held-out data, independent of the dialogue generation.

Quality Control: A rigorous four-stage verification process (P2–P5) includes 46 deterministic checks (data integrity, split validity, code execution), LLM cross-family reviews for bias and leakage, and human annotation via a specialized dashboard.

Evaluation Protocol

The evaluation employs two parallel scoring tracks:

1. Outcome Scoring: Measures the correctness of the final answer across five dimensions: Numerical Accuracy, Factual Verification, Analytical Quality, Decision Making, and Code Correctness. Numerical and code metrics are deterministic; analytical and decision quality use an LLM judge.
2. Capability Scoring: Diagnoses agent behavior against a 9-capability taxonomy (C1–C9):
   • Basic: Read (C1), Select (C2), Configure (C3).
   • Reasoning: Remember (C4), Chain (C5), Adapt (C6).
   • Action: Decide (C7), Calibrate (C8), Ground (C9).
     Most sub-checks are deterministic, with only C7 and C9 requiring LLM judgment.

Systems Evaluated

The benchmark evaluates six frontier LLMs (Claude Sonnet 4.6, GPT-5.3-Codex, GLM-5.1, Qwen-3.5-122B-A10B, MiniMax-M2.7, Gemini-3-Flash-Preview) under four system settings:

1. Direct Answering: No tools or code execution.
2. Code-Enabled Reasoning: Execution in a Python sandbox.
3. Skill-Guided Code Reasoning: Access to a curated library of 226 time series skills.
4. TimeSage: A structured agentic pipeline (Planner, Profiler, Executor, Evaluator, Reporter) utilizing the skill library and deterministic validation gates.

Key Results

Performance Trends

• Depth Sensitivity: All models perform best on L1 (Open Exploration) but suffer sharp performance drops (10–15 points) at L2 and beyond. This indicates that current agents struggle with multi-turn memory, dependency chaining, and complex decision workflows.
• Code vs. Reasoning: While Code Correctness is high (71–78%), other dimensions like Numerical Accuracy (15–18%) and Factual Verification are significantly lower. This reveals that generating runnable code does not guarantee trustworthy, data-grounded analysis.
• Capability Gaps: Models exhibit strong performance in Remembering (C4) and Chaining (C5) but show persistent weaknesses in Reading (C1) (misinterpreting data properties) and Configuring (C3) (setting incorrect parameters). Grounding (C9) and Calibration (C8) also remain challenging.

Impact of System Design

• Code-Enabled Reasoning: Provides substantial gains over Direct Answering, particularly in Numerical Accuracy and Decision Making, by anchoring responses to executable evidence.
• Skill Libraries: Benefit varies by model. Stronger models (e.g., GPT-5.3-Codex) show significant improvements in Analytical Quality and Decision Making when guided by skills. Weaker models (e.g., Qwen-3.5) show marginal gains, suggesting that skill libraries require a capable backbone to be effectively invoked and integrated.
• Structured Pipelines (TimeSage): Introduce a trade-off. For weaker backbones, the structured pipeline improves Numerical Accuracy and Factual Verification but can restrict expressiveness, lowering Decision Making scores. For stronger backlines, the additional orchestration layer can introduce planning conflicts, degrading overall performance.

Significance and Claims

The paper positions TimeSage-MT as the first multi-turn benchmark specifically for agentic time series reasoning. Its primary contribution is shifting the evaluation paradigm from single-step prediction accuracy to the correctness of the entire analytical trajectory.

The authors claim that TimeSage-MT exposes critical gaps in current agentic systems, specifically:

1. The inability to maintain reliable reasoning over long dialogues.
2. Failures in uncertainty calibration and domain-grounded decision-making.
3. The limitation of "code execution" alone without structured skill integration or robust validation.

The benchmark is presented not as a deployment certificate but as a diagnostic tool. It enables researchers to move beyond aggregate scores to identify specific failure modes (e.g., misreading data vs. failing to chain dependencies). The authors conclude that reliable time series agents require not just stronger LLMs, but better harness design that balances flexibility with rigorous numerical verification and domain grounding. The work provides a foundation for developing auditable, trustworthy AI tools for real-world time series applications.