HEARTS: Benchmarking LLM Reasoning on Health Time Series

The paper introduces HEARTS, a comprehensive benchmark comprising 16 real-world health datasets and 110 tasks across four reasoning capabilities, which reveals that current large language models significantly underperform specialized models in health time series analysis due to struggles with multi-step temporal reasoning and reliance on simple heuristics.

The Gist

Imagine you have a super-smart robot assistant (a Large Language Model, or LLM) that can write poetry, solve math problems, and chat like a human. Now, imagine you hand this robot a stack of medical charts filled with squiggly lines representing heartbeats, brain waves, and breathing patterns over days or even years. You ask it: "Based on these lines, is this patient sleeping well? Are they at risk of a heart attack tomorrow? Can you predict what their heart rate will be in an hour?"

This paper, titled HeaRTS, is essentially a giant "report card" for these robots to see how good they really are at reading these medical squiggly lines.

Here is the breakdown in simple terms:

1. The Problem: The "Generalist" vs. The "Specialist"

For a long time, AI researchers have been training these robots to be "generalists"—good at everything. But in the medical world, reading a heart monitor isn't like reading a book. It's like trying to understand a symphony by only looking at the sheet music for one instrument, or trying to predict the weather by looking at a single cloud.

Existing tests were too simple. They only asked about one type of signal (like just heartbeats) or used fake data. The authors realized, "We need a real-world gym where these robots have to lift heavy, complex weights."

2. The Solution: HeaRTS (The Ultimate Medical Time-Test)

The authors built HeaRTS (Health Reasoning over Time Series). Think of this as a massive, multi-level obstacle course designed specifically for medical data.

• The Course: It includes 16 different real-world datasets (like actual hospital records, wearable watch data, and sleep study logs).
• The Variety: It covers 12 different health areas (from sleep and metabolism to eye movements and coughing) and 20 different types of signals (from slow daily trends to super-fast electrical brain waves).
• The Tasks: They created 110 different challenges grouped into four levels of difficulty:
  1. Perception: "What is the average heart rate?" (Reading the data).
  2. Inference: "Did the patient have a seizure at 3 AM?" (Finding patterns).
  3. Generation: "Fill in this missing 10 minutes of data" or "Predict the next hour of breathing." (Creating new data).
  4. Deduction: "Based on this week's data, will this patient have a stroke next month?" (Connecting the dots over time).

3. The Results: The Robots Struggle

The authors tested 14 of the smartest AI models (like GPT-4, Claude, Gemini, etc.) on over 20,000 test cases. The results were surprising and a bit disappointing:

• The "Smart" Robot isn't a "Doctor": Even the most advanced AI models performed much worse than specialized medical software designed just for one specific job. It's like asking a brilliant philosopher to perform heart surgery; they know a lot, but they lack the specific tools and training for the job.
• General Smarts Don't Help: A model's score on general reasoning tests (like math or logic puzzles) had almost no connection to how well it did on medical time-series tasks. Being good at chess doesn't mean you're good at reading an EKG.
• The "Cheat Code" Failure: The AI models often didn't actually "reason" through the complex time patterns. Instead, they relied on simple tricks (heuristics).
  • Analogy: If you ask a human to predict the stock market, they might look at the trend. If you ask this AI, it often just draws a straight line or copies the last few seconds of data and adds some noise, hoping it looks right. It's like a student guessing the answer by looking at the shape of the question mark rather than doing the math.
• The More Data, The Worse They Do: As the data got longer (years instead of minutes) or faster (thousands of samples per second), the AI's performance dropped. They get overwhelmed by the sheer volume of "squiggles."

4. The "Living" Benchmark

The coolest part of this paper is that HeaRTS isn't a static test. It's a "Living Ecosystem."

• Think of it like a video game that keeps getting new levels. As AI gets better, the researchers will add harder medical datasets and new types of tasks.
• They built a system where other scientists can easily plug in their own new AI models or new medical data to see how they stack up.

The Big Takeaway

The paper concludes that while AI is amazing at writing text and code, it is currently terrible at understanding the complex, flowing story of human biology over time.

It's not just about making the AI "smarter" or "bigger." We need to teach them how to think like doctors and physiologists, not just like text predictors. HeaRTS provides the map and the measuring stick to help us get there.

In short: We built a giant, realistic medical exam for AI. The AI took the test, and while it passed the easy questions, it failed the hard, real-world stuff. Now, we have a clear roadmap to help it learn how to actually save lives.

Technical Summary

Here is a detailed technical summary of the paper "HeaRTS: Benchmarking LLM Reasoning on Health Time Series".

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong reasoning capabilities in text, code, and mathematics, their ability to perform genuine reasoning over health time series remains unproven. Existing benchmarks suffer from three critical limitations:

• Narrow Scope: They cover only a small subset of health modalities (e.g., mostly ECG) and tasks, failing to reflect the diversity of real-world physiological data.
• Lack of Realism: Many benchmarks rely on synthetic signals that lack clinical noise, context, and complex temporal dependencies found in real patient data.
• Evaluation Gap: Current evaluations often reduce complex reasoning to simple multiple-choice questions, failing to test hierarchical reasoning capabilities such as long-horizon synthesis, cross-channel correlation, and causal deduction.

The core question addressed is: Do LLMs perform genuine multi-step reasoning on health time series, or do they merely exploit surface patterns and domain priors?

2. Methodology: The HeaRTS Benchmark

The authors introduce HeaRTS (Health Reasoning over Time Series), a unified, "living" benchmark designed to evaluate LLMs across diverse health domains.

A. Data Diversity and Scale

HeaRTS integrates 16 real-world datasets spanning 12 health domains (e.g., Metabolic, Surgery, Sleep, Emotion, Ophthalmology) and 20 signal modalities (ranging from standard bioelectrical signals like ECG/EEG to high-frequency audio and IMU data).

• Temporal Resolution: Sampling frequencies range from daily aggregates to 48 kHz.
• Sequence Length: Inputs vary from 60 steps to over 1 million steps.
• Time Span: Data covers seconds to years.
• Total Samples: Over 20,226 test cases.

B. Hierarchical Task Taxonomy

Instead of a single QA format, HeaRTS defines 110 tasks grouped into four cognitive categories to test different levels of reasoning:

1. Perception: Basic signal understanding (e.g., statistical calculation, feature extraction like bandpower).
2. Inference: Analysis at varying granularities (e.g., event localization, physiological classification, subject profiling).
3. Generation: Temporal synthesis (e.g., future forecasting, signal imputation, cross-modal translation).
4. Deduction: High-level reasoning requiring temporal directionality and longitudinal integration (e.g., temporal ordering, trajectory analysis, counterfactual deduction).

C. Evaluation Framework

• Agent-Based Execution: To handle long context windows and complex data, the authors use the CodeAct framework, allowing LLMs to write and execute Python code to read and analyze data files rather than ingesting raw sequences directly into the prompt.
• Metrics: Tasks are scored using Accuracy, IoU (Intersection over Union), or sMAPE (symmetric Mean Absolute Percentage Error), normalized to a 0–1 scale.
• Baselines: Includes a "Naive Baseline" (random guessing or simple heuristics) and comparisons against specialized State-of-the-Art (SOTA) machine learning models.

3. Key Contributions

1. First Diverse Health Time-Series Benchmark: HeaRTS is the first benchmark to span 12 domains and 20 modalities with the broadest coverage of sequence length, frequency, and time span.
2. Unified Hierarchical Evaluation: It moves beyond multiple-choice questions to a multi-modal evaluation setting (JSON outputs, code execution) covering four distinct cognitive levels.
3. Living Ecosystem: The benchmark is designed as a dynamic, community-driven platform with standardized APIs for adding new datasets, tasks, and agent architectures, ensuring it evolves with the field.
4. Comprehensive Empirical Analysis: The paper provides a rigorous evaluation of 14 SOTA LLMs (including GPT-4.1, GPT-5, Claude 4.5, Gemini 2.5, Qwen3, etc.) against the benchmark.

4. Key Results and Findings

The evaluation of 14 LLMs on 20K+ samples revealed significant gaps in current capabilities:

• Finding 1: LLMs Lag Behind Specialized Models.

  • LLMs substantially underperform compared to specialized time-series models (SOTA ML). While SOTA models concentrate in a high-performance regime, LLMs are shifted significantly lower, even in their best-case scenarios.
  • Performance on HeaRTS is weakly correlated with general intelligence indexes (e.g., MMLU, GPQA), indicating that general reasoning skills do not translate to health time-series reasoning.

• Finding 2: Reliance on Low-Complexity Heuristics.

  • LLMs struggle with deep temporal reasoning. In generation tasks (forecasting/imputation), they often default to simplistic strategies like copy-pasting, linear interpolation, or polynomial fitting rather than modeling stochastic dynamics.
  • In inference tasks, they rely heavily on explicit rules or priors (e.g., "hypotension is <65") and fail when fine-grained distinctions or domain-specific feature extraction are required without explicit guidance.

• Finding 3: Performance Degrades with Temporal Complexity.

  • There is a strong inverse correlation between performance and data complexity. As sequence length increases and sampling frequency rises, LLM performance drops significantly.
  • This "cognitive burden" suggests current models cannot maintain reasoning efficacy over long horizons or high-density signals.

• Finding 4: Model-Agnostic Difficulty Patterns.

  • Models within the same architectural family (e.g., GPT-4.1 vs. GPT-5, Qwen3 vs. Qwen3 Coder) show highly consistent behavioral patterns. Tasks difficult for one variant are difficult for all, suggesting the difficulty is intrinsic to the data domain rather than the model architecture.
  • Scaling alone is insufficient: Increasing model size or recency does not guarantee improved proficiency in this domain.

• Finding 5: Input Format and Information Overload.

  • Input format (File vs. Text vs. Image) has a moderate effect, with visual ingestion generally underperforming text-based file access.
  • More data is not better: Providing auxiliary signals (multimodal context) often fails to improve performance and can even degrade it due to "informational distraction" or "contextual neglect."

5. Significance and Future Directions

• Standardization: HeaRTS provides a standardized testbed to measure the specific gaps in LLM reasoning for healthcare, moving beyond narrow analytics to hierarchical reasoning.
• Research Direction: The results highlight that current LLMs lack the domain-specific priors and computational primitives needed for robust physiological reasoning. Future work must focus on integrating domain-specific tools, verified clinical rules, and modality-specific architectures rather than relying solely on scaling general-purpose models.
• Community Impact: By releasing the dataset, code, and a "living" framework, the authors enable the community to collaboratively develop next-generation LLM agents capable of handling the complexity of real-world health signals.

In conclusion, HeaRTS demonstrates that while LLMs show promise in basic signal perception, they currently lack the deep, multi-step temporal reasoning required for complex clinical decision-making, necessitating a shift from general-purpose scaling to specialized, tool-augmented reasoning agents.