Technical Summary: ECG-LLM

Problem Statement

Cardiovascular diseases remain the leading cause of death globally, yet front-line triage in primary care is often constrained by limited access to advanced diagnostic imaging such as echocardiography (ECHO) or cardiac magnetic resonance (CMR). While the 12-lead electrocardiogram (ECG) is a standard, inexpensive initial test, existing AI systems for ECG interpretation are typically limited to fixed diagnostic labels or automated report generation. These systems lack the flexibility to support patient-specific clinical reasoning or to infer complex cardiac phenotypes (e.g., ventricular volumes, wall thickness) that are not directly visible on the ECG waveform but are critical for clinical decision-making. Furthermore, current multimodal approaches often require separate models for specific tasks or rely on contrastive learning that produces representations rather than direct clinical answers.

Methodology

The authors introduce ECG-LLM, a foundation model designed to perform language-mediated, multimodal cardiac reasoning using only the raw 12-lead ECG signal as input.

Architecture

ECG-LLM follows a vision-language model (VLM) architecture adapted for time-series data:

• ECG Encoder: A self-supervised ViT-MAE (Masked Autoencoder) encoder pretrained on 12-lead ECGs using a masked reconstruction objective. This encoder processes raw waveform patches into latent tokens without relying on paired text or imaging data during pretraining.
• Projector: A lightweight two-layer MLP that maps the ECG encoder tokens into the embedding space of the language model.
• Language Model: An autoregressive LLM (Llama-3.1-8B-Instruct) that generates free-text answers. During training, the LLM backbone is frozen, and only LoRA (Low-Rank Adaptation) adapters and the projector are trained to minimize next-token prediction loss.

Multimodal-to-Language Supervision

The core innovation is a multimodal-to-language supervision strategy. Instead of training separate models for specific phenotypes, the authors construct a unified supervision space using natural language.

1. Data Sources: The model is trained on 679,112 ECG studies from four cohorts: UK Biobank (linked to CMR), MIMIC-IV-ECG, PTB-XL, and EchoNext (linked to ECHO).
2. QA Generation: For each ECG study, structured metadata (ECG measurements, diagnostic reports, CMR/ECHO phenotypes, demographics) are compiled. A locally hosted LLM (GPT-OSS-20B) generates over 5 million clinically grounded question-answer (QA) pairs. These pairs simulate front-line clinical queries (e.g., "What is the ventricular function?") and provide answers derived from the available imaging and clinical data.
3. Training Objective: The model is trained to answer these questions conditioned only on the ECG waveform and the question text. At inference, no imaging data or external retrieval modules are used; the model must infer imaging-derived phenotypes solely from the ECG representation.

Key Contributions

1. Multimodal-to-Language Supervision: A strategy that shifts ECG-language training from reproducing machine-generated reports to answering flexible, question-driven queries about both ECG-derived and imaging-related cardiac phenotypes.
2. Large-Scale Instruction-Tuning Resource: A pipeline generating 5+ million clinically realistic, GP-style QA pairs from unified ECG, CMR, and ECHO data, creating a resource for ECG-conditioned reasoning.
3. ECG-LLM Model: An open-source, autoregressive model that processes raw 12-lead ECGs to answer free-text cardiac questions without rendered images, task-specific classification heads, or inference-time imaging.
4. Systematic Evaluation: Comprehensive benchmarking across UK Biobank, MIMIC-IV-ECG, PTB-XL, and EchoNext, covering ECG interpretation, imaging phenotype inference, and standard QA benchmarks.

Results

1. ECG- and CMR-Derived Phenotype Inference

Evaluated on 1,000 UK Biobank ECGs, ECG-LLM successfully answered questions regarding conventional ECG measurements (heart rate, intervals, axes) and CMR-derived phenotypes (volumes, ejection fraction, wall thickness).

• ECG Phenotypes: The model achieved strong macro-F1 scores for directly visible properties (e.g., heart rate, QRS duration).
• CMR Phenotypes: The model demonstrated the ability to infer structural signals not directly measured on ECGs, such as left ventricular (LV) wall thickness and ventricular volumes, performing significantly above majority-class baselines. Confusion matrices indicated that while rare classes were challenging, the model captured coarse structural signals.

2. Echocardiographic Phenotype Inference (EchoNext)

On the EchoNext dataset (5,442 paired ECG-ECHO studies), the model was evaluated on binary classification of 11 echocardiographic phenotypes via a "Yes/No" free-text protocol.

• Performance: ECG-LLM achieved the highest F1-scores for 10 of 11 phenotypes compared to the Columbia Mini-Model and a supervised ViT baseline.
• Key Findings: Strongest performance was observed for increased LV wall thickness (F1: 0.77), moderate-or-greater aortic stenosis (F1: 0.67), and right-ventricular systolic dysfunction (F1: 0.61). Performance was lower for rare or weakly expressed findings (e.g., pericardial effusion, pulmonary regurgitation), though still non-random.

3. ECG-QA Benchmark

Evaluated on the ECG-QA benchmark (PTB-XL and MIMIC-IV-ECG), ECG-LLM outperformed existing baselines (Fusion, MedViLL, M3AE) on targeted tasks:

• Single-Verify: 76.71% exact match on PTB-XL (vs. 74.60% for M3AE).
• Single-Choose: 68.32% exact match on PTB-XL (vs. 57.10% for M3AE).
• The model showed strong performance across diverse concepts including conduction blocks, arrhythmias, and ischemic changes, though exact match on complex multi-label "Single-Query" tasks remained a challenge due to the strictness of the metric.

4. Diagnostic Report Generation

Using an LLM-as-a-Judge protocol (GPT-OSS-20B) to evaluate free-text diagnostic summaries, ECG-LLM matched or exceeded the performance of PULSE-7B (an ECG-image baseline) and significantly outperformed general medical VLMs (MedGemma-4B) and the raw-signal baseline ECG-Chat (without its retrieval modules). This confirmed the model's ability to map raw ECG morphology to conventional diagnostic language.

Significance and Claims

The paper claims that ECG-LLM represents a shift from fixed-label prediction to flexible, question-driven cardiovascular reasoning. By using natural language as a unified supervision space, the model learns to associate raw ECG waveforms with a broad spectrum of cardiac knowledge, including imaging-derived phenotypes.

The authors emphasize that this approach allows a single ECG to serve as a flexible interface for front-line triage, enabling clinicians to ask targeted questions about a patient's broader cardiovascular status (e.g., "Is there evidence of reduced ejection fraction?") even when imaging is delayed or unavailable. The work positions ECG-LLM not as an autonomous diagnostic system, but as a clinician-facing decision support tool that can provide a "second opinion" and guide the necessity of further imaging. The authors acknowledge limitations, including the risk of hallucination, the discretization of continuous variables, and the need for human clinical evaluation before deployment.