ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

This paper introduces the ECG-Reasoning-Benchmark, a multi-turn evaluation framework revealing that while state-of-the-art multimodal large language models possess medical knowledge, they critically fail to perform genuine step-by-step visual reasoning when interpreting ECG signals, often relying on superficial cues rather than grounding diagnoses in actual visual evidence.

The Gist

Imagine you are hiring a new doctor to interpret heart charts (ECGs). You have two candidates:

1. The "Textbook Reciter": This doctor has memorized every medical textbook. If you ask, "What does a heart attack look like?" they can recite the definition perfectly: "It involves ST-segment elevation in leads II, III, and aVF." They sound incredibly smart and confident.
2. The "Detective": This doctor looks at the actual squiggly lines on the chart, measures them with a ruler, checks the specific leads, and then says, "Yes, I see the elevation here, so it's a heart attack."

The Problem:
For the last few years, we've been training AI models (specifically Multimodal Large Language Models) to be doctors. We've fed them millions of heart charts and their corresponding reports. The AI has become the "Textbook Reciter." It can generate a beautiful, fluent report that sounds exactly like a real doctor wrote it.

But here's the catch: The AI isn't actually looking at the chart. It's just guessing the diagnosis based on the pattern of the text it was trained on. It's like a student who memorized the answer key but never learned how to solve the math problems. If you give them a slightly different problem, they fail because they don't understand the logic, they just know the words.

The Solution: The "ECG Reasoning Benchmark"
The authors of this paper built a new, super-tough test called ECG-Reasoning-Benchmark. Instead of just asking the AI, "What is the diagnosis?" (which is easy to fake), they turned the test into a 20-question interrogation.

Think of it like a Sherlock Holmes mystery where the AI has to prove its case step-by-step:

1. The Clue Hunt: "To diagnose a heart block, what specific thing should we look for?" (The AI must pick the right rule).
2. The Visual Check: "Okay, is that thing actually present in this specific chart?" (The AI must look at the image).
3. The Evidence: "Point to the exact spot on the chart where you see it." (The AI must highlight the specific wave).
4. The Measurement: "How long is that wave? Is it 120ms or 150ms?" (The AI must measure it).
5. The Verdict: "Based only on what you just found, is the diagnosis confirmed?"

The Shocking Results
The researchers ran this test on the smartest AI models available (including models from Google, OpenAI, and specialized medical AIs). The results were a wake-up call:

• The "Reciters" Failed: Even the most advanced models could only complete the full chain of reasoning about 6% of the time.
• The "Knowledge" Trap: The models knew the rules (Step 1). They knew what to look for. But the moment they had to actually look at the chart and find the evidence (Steps 2, 3, and 4), they collapsed.
• The Hallucination: When the AI couldn't find the evidence, it didn't say, "I can't see it." Instead, it made up a story, claiming to see waves that weren't there, just to keep the conversation flowing.

The Big Takeaway
This paper is a reality check for the medical AI industry. It proves that sounding smart is not the same as being smart.

Currently, our AI models are like actors who can recite a script perfectly but don't understand the plot. They are great at generating text, but terrible at the actual work of a doctor: observing, measuring, and reasoning based on physical evidence.

What needs to happen?
We can't just train AI to write better reports. We need to teach them to look. We need to build systems that force the AI to point to the evidence, measure the lines, and admit when it doesn't know, rather than just guessing the most likely-sounding answer. Until we fix this "reasoning gap," we can't fully trust AI to make life-or-death medical decisions.

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have shown promise in automated electrocardiogram (ECG) interpretation, a critical gap exists between their ability to generate fluent diagnostic reports and their capacity for genuine clinical reasoning.

• The "Black Box" & Hallucination Risk: Current MLLMs often rely on superficial visual cues or memorized text patterns rather than analyzing the raw physiological signal. They frequently "hallucinate" medically plausible explanations that recite textbook definitions without grounding them in the actual ECG evidence.
• Flawed Evaluation Paradigms: Existing benchmarks predominantly use the "LLM-as-a-Judge" framework, where a reference LLM evaluates the output of a target model. Since these reference answers are often synthetically generated without direct signal analysis, this method only measures linguistic fluency and style mimicry, failing to verify if the model's logic is grounded in the actual ECG data.
• Lack of Ground Truth: Public ECG datasets typically provide high-level diagnostic labels but lack the granular, step-by-step annotations (wave boundaries, interval measurements, specific lead findings) required to verify a model's reasoning chain.

2. Methodology

The authors propose ECG-Reasoning-Benchmark, a rigorous evaluation framework designed to test the step-by-step logical deduction of MLLMs.

A. Automated ECG Analysis Pipeline (Ground Truth Generation)

To overcome the lack of granular annotations, the authors developed an automated pipeline to generate verifiable ground truth directly from raw 12-lead signals:

1. Wave Detection & Segmentation: Uses a U-Net3+ architecture to detect P, QRS, and T waves. It employs context-aware post-processing (template matching, physiological constraints, and multi-lead consensus) to recover missed waves (e.g., non-conducted P waves) and ensure biological validity.
2. Feature Extraction & Quantification: Calculates precise temporal (duration, intervals like PR, QT) and amplitude measurements, as well as morphological patterns (e.g., qR, rS, RSR').
3. Finding Extraction: Maps quantitative measurements to discrete clinical findings (e.g., "Prolonged PR interval" if >200ms) using established clinical criteria.
4. Diagnosis Derivation: Applies hierarchical logic diagrams (validated by board-certified specialists) to derive final diagnoses from the extracted findings.

B. The Benchmark Framework

The benchmark consists of 6,400+ samples covering 17 core ECG diagnoses (e.g., AV Blocks, Bundle Branch Blocks, Myocardial Infarctions) derived from PTB-XL and MIMIC-IV-ECG datasets.

• Evaluation Workflow: Instead of a single-turn question, the evaluation uses a 4-step recursive verification loop for each clinical finding required for a diagnosis:
  1. Criterion Selection: The model must identify the correct diagnostic criterion (e.g., "Is QRS prolonged?").
  2. Finding Identification: The model must confirm the presence of the finding in the signal.
  3. ECG Grounding: The model must explicitly locate the evidence:
     • Lead Grounding: Identify specific leads (e.g., V5, V6).
     • Wave Grounding: Identify the time segment (e.g., 1.45s – 1.61s).
     • Measurement Grounding: Select the correct value range.
  4. Diagnostic Decision: The model decides if the diagnosis is confirmed or if further findings are needed.
• Metrics:
  • Completion (%): The strict proportion of samples where the model answers every step correctly in the chain.
  • Depth (0-4): The average number of steps successfully completed before failure.
  • GT-Reasoning-Based Diagnosis Accuracy (GT-RDA): Measures performance when the model is given the correct reasoning history, testing if it can utilize valid evidence.

3. Key Contributions

1. Novel Evaluation Paradigm: Shifts the standard from subjective "LLM-as-a-Judge" scoring to an objective, step-by-step verification of clinical reasoning grounded in physical signal evidence.
2. Automated Ground Truth Pipeline: A robust tool that extracts explicit diagnostic features from raw signals, providing a transparent and traceable chain of evidence for 17 core diagnoses.
3. Comprehensive Benchmark: A dataset of 6,400+ multi-turn samples with balanced positive/negative cases across diverse logical paths, ensuring models cannot exploit superficial shortcuts.

4. Results

The authors evaluated a diverse suite of models, including ECG-specific models (PULSE, GEM, ECG-R1), medical-domain models (Hulu-Med, MedGemma), and general-purpose MLLMs (Qwen3-VL, Llama-3.2-Vision, Gemini, GPT-5).

• Critical Failure in Reasoning Chains: The Completion metric was abysmal across all models, with the highest score being only ~6.26% (Gemini-3-Flash). Most models failed to maintain a complete logical chain.
• The "Grounding" Bottleneck: While models performed well on Criterion Selection (Step 1, identifying what to look for), they failed catastrophically at Finding Identification and ECG Grounding (Steps 2 & 3). They could retrieve medical knowledge but could not link it to the visual evidence in the signal.
• The "Pattern Matching" Flaw in ECG-Specific Models:
  • ECG-specific models (e.g., ECG-R1, PULSE) showed high Initial Diagnosis Accuracy (IDA) (~80-85%), suggesting they can guess the label correctly.
  • However, when provided with the Ground Truth Reasoning (GT-RDA), their accuracy collapsed (dropping to ~22-35%). This indicates they rely on superficial pattern matching between global signal features and labels, rather than valid deductive reasoning.
  • Conversely, general medical models (e.g., Hulu-Med) had low IDA (50%) but achieved **99% accuracy** when given the correct reasoning path, proving they possess the latent capacity to reason if guided correctly.
• Depth Analysis: Models rarely progressed beyond Depth 2.0, confirming that the inability to ground findings in the signal is the primary bottleneck.

5. Significance

• Exposing the Illusion of Competence: The study demonstrates that high diagnostic accuracy or fluent text generation in current MLLMs does not equate to clinical reasoning capability. Models are currently "bypassing" actual visual interpretation.
• Redefining Medical AI Standards: The paper argues that for AI to be trusted in high-stakes healthcare, it must move beyond "black box" predictions and "LLM-as-a-Judge" evaluations. It necessitates reasoning-centric training and evaluation that enforces grounding in raw physiological data.
• Future Directions: The findings highlight the urgent need for training paradigms that force models to perform explicit visual grounding and logical deduction, rather than simply mimicking the linguistic style of synthetic medical reports.

In conclusion, ECG-Reasoning-Benchmark reveals a profound gap in current medical AI: while models know what to diagnose, they fundamentally lack the ability to prove it through step-by-step visual evidence, a critical requirement for safe clinical deployment.