Technical Summary: Holtercare-Bench and Holtercare-23K

Problem Statement

While Multimodal Large Language Models (MLLMs) have revolutionized medical AI, current research is heavily skewed toward static spatial modalities (e.g., radiology images) or short-term signals. In cardiology, standard datasets like PTB-XL and MIMIC-IV-ECG capture only seconds of activity, failing to address the clinical gold standard: continuous dynamic Holter monitoring. This modality presents unique challenges, including ultra-long sequences (13–24 hours), hundreds of thousands of heartbeats, and fleeting pathological events (e.g., brief ventricular tachycardia) that require millisecond-level temporal sensitivity. Existing benchmarks lack the capacity to evaluate complex temporal reasoning, causal clinical logic, and global summarization required for long-term ECG analysis.

Methodology

1. Dataset Construction: Holtercare-23K

To bridge the data gap, the authors introduce Holtercare-23K, a large-scale multimodal dataset derived from 788 real-world clinical Holter records (recorded in 2026), each lasting 13–24 hours.

• Tri-Modal Alignment: To overcome the modality mismatch where language models cannot process raw voltage arrays, the authors developed HolterAgent, an automated data engine that converts raw signals into three modalities:
  • Signal: High-quality, denoised electrophysiological data (3 leads).
  • Video: Dynamic ECG video streams generated via sliding windows (e.g., 10-second durations).
  • Text: Structured clinical records including de-identified patient demographics (age, gender) and Electronic Medical Records (EMRs) with symptoms.
• Annotation System: The dataset features a rigorous three-level annotation system:
  • Beat-Level: Precise timestamps for 18 heartbeat categories and non-beat events.
  • Rhythm-Level: Continuous event descriptions (e.g., ST-segment changes) with timestamps.
  • Report-Level: Global summaries verified by professional cardiologists, containing statistics (total beats, HR extremes, burden percentages) and diagnostic conclusions.
• Scale: The dataset comprises 22,980 QA pairs across three distinct task types.

2. Benchmark Design: Holtercare-Bench

Based on the dataset, the authors propose Holtercare-Bench, a comprehensive evaluation framework mirroring the diagnostic workflow of cardiologists. It consists of 12 fine-grained tasks divided into three tiers:

• Closed-QA (5 Tasks): Discrete decision-making tasks evaluated by strict matching accuracy. Tasks include Presence (existence of rhythm), Event Counting, Event Timing (millisecond precision), HR Extremum Timing, and Diagnosis.
• Open-QA (5 Tasks): Free-text generation tasks evaluated using BLEU, ROUGE-L, F1-Bio, and a custom normalized metric (ScoreMAE) for numerical reasoning. Tasks include Event Counting, HR Extremum Counting, Event Timing, Diagnosis, and Evidence Reasoning (generating causal rationales).
• Report Generation (2 Tasks): Global abstraction tasks evaluated via text metrics and an LLM-as-a-judge framework (ScoreGPT) based on GPT-5-mini. Tasks include Statistical Overview (extracting global metrics) and General Summary (clinical report generation).

3. Experimental Setup

The authors evaluated 13 mainstream models, including generalist models (e.g., GPT-5-mini, Qwen3-VL-8B) and medical-specific models (e.g., LLaVA-Med, HealthGPT).

• Zero-Shot Evaluation: Models were tested without fine-tuning to establish baselines.
• Fine-Tuning: Two representative open-source models (Phi-4-mini-3.8B for text, Qwen3-VL-8B for video) were instruction-tuned on the Holtercare-23K training set to assess the dataset's efficacy.

Key Results

Zero-Shot Limitations

Zero-shot evaluations revealed a significant performance gap. While advanced generalist models showed reasonable foundational comprehension in Presence detection (e.g., GPT-5-mini at 76.31%), they struggled severely with fine-grained temporal problems and ultra-long context reasoning.

• Temporal Sensitivity: Models failed to maintain temporal consistency over long sequences, resulting in low accuracy on Event Timing and HR Extremum Timing.
• Reasoning & Summarization: Medical-specific models and generalists alike produced low entity coverage and poor ROUGE-L scores in Report Generation, indicating an inability to synthesize global statistics or perform causal reasoning without domain-specific alignment.

Impact of Fine-Tuning

Fine-tuning on Holtercare-23K yielded substantial improvements, demonstrating the dataset's potential to unlock long-context capabilities:

• Qwen3-VL-8BFT (Video): The fine-tuned model achieved 99.70% accuracy on the Event Timing sub-task and 95.28% on Diagnosis. The Mean Absolute Error (MAE) for Event Counting dropped from 7.34 to 0.63.
• Phi-4-mini-3.8BFT (Text): The fine-tuned model showed massive leaps across all metrics, with Event Timing accuracy rising to 63.41% and Diagnosis to 49.61%.
• Report Generation: Fine-tuned models achieved state-of-the-art F1-Bio and ROUGE-L scores, successfully synthesizing ultra-long sequences into coherent clinical reports.

Significance and Claims

The paper claims that Holtercare-Bench and Holtercare-23K provide a foundational benchmark for long-term medical MLLMs. The work highlights that:

1. Current Limitations: Existing MLLMs lack the high-quality data and specific benchmarks necessary to learn complex clinical reasoning over ultra-long pathological sequences.
2. Data Efficacy: High-quality, tri-modal alignment (signal-video-text) is critical for activating a model's temporal perception and clinical reasoning capabilities.
3. Future Direction: The benchmark establishes a strong baseline and defines performance frontiers, serving as a necessary step toward developing native dynamic ECG MLLMs capable of expert-level diagnostic analysis.

The authors emphasize that while current models struggle with precise temporal localization and causal reasoning in long sequences, the proposed dataset successfully demonstrates that fine-tuning can bridge the gap toward expert-level performance.