🤖 AI

Position: Evaluation of ECG Representations Must Be Fixed

This position paper argues that current ECG representation learning benchmarks are too narrow and flawed, demonstrating that proper evaluation practices reveal random encoders often match state-of-the-art pre-training models while highlighting the urgent need to expand assessment to include structural heart disease and patient-level forecasting.

Original authors: Zachary Berger, Daniel Prakah-Asante, John Guttag, Collin M. Stultz

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zachary Berger, Daniel Prakah-Asante, John Guttag, Collin M. Stultz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the field of Artificial Intelligence (AI) trying to learn how to read heart signals (ECGs) is like a group of students preparing for a very specific, high-stakes exam. For years, everyone has been studying from the same three textbooks (datasets called PTB-XL, CPSC2018, and CSN). These textbooks are filled with questions about arrhythmias (irregular heartbeats) and waveform shapes (what the squiggly lines look like).

The authors of this paper, Zachary Berger and his team, are raising their hands to say: "Wait a minute. The way we are grading these students is broken, and it's giving us the wrong idea about who is actually smart."

Here is the breakdown of their argument using simple analogies:

1. The "One-Size-Fits-All" Exam is Flawed

Currently, the AI community grades these models using a single score called Macro-AUROC.

The Analogy: Imagine a teacher grading a student who took a test with 100 questions. 90 of the questions are about "Apples," and 10 are about "Rare Exotic Fruits." The teacher averages the score so that every question counts the same.
The Problem: If the student gets all the "Apple" questions right but fails the "Exotic Fruit" questions, they get a decent average. But in the real world (the hospital), the doctor might only care about the "Exotic Fruits" (rare heart conditions). By averaging everything together, the current system hides the fact that the model might be terrible at the things that actually matter.
The Noise: Furthermore, some of those "Exotic Fruit" questions have only one or two examples in the whole test. If the model gets that one question right or wrong by pure luck, it swings the entire grade up or down, making the results unreliable.

2. The "Surprise" Baseline: The Random Guess

The most shocking finding in the paper is about the "baselines." In science, you always compare a new, fancy invention against a simple, basic version to see if the invention actually adds value.

The Analogy: Imagine a race where everyone is running on a track. The fancy runners are wearing high-tech, pre-trained shoes (complex AI models). The "baseline" runner is wearing a pair of randomly generated, untrained shoes (a model with random weights that hasn't learned anything yet).
The Result: The authors found that on many tasks, the runner with the random shoes finished just as fast as the runners with the fancy shoes. Sometimes, the random runner even won!
The Takeaway: This suggests that for many heart tasks, the signal is so obvious that you don't need a super-complex, pre-trained brain to find it. A simple, random starting point is often good enough. This means the "fancy" models might not be as special as we thought.

3. The Paper's Prescription: A Better Syllabus

The authors argue that to fix this, the field needs to change how it tests these models. They propose four main rules:

Expand the Curriculum: Don't just test on arrhythmias. The heart signal also holds clues about structural heart disease (like a weak heart muscle), hemodynamics (blood pressure and flow), and future patient outcomes (will this person get sick in a year?). The current exams ignore these important topics.
Stop Hiding the Details: Instead of reporting just one big average score, report the score for each specific task. If a model is great at detecting heart attacks but terrible at detecting valve issues, we need to know that, not just see a "75%" average.
Measure Uncertainty: Because some heart conditions are rare, the results can be shaky. The authors say we must report "confidence intervals" (a range of possible scores) to show how sure we are. If the range is huge, the result isn't trustworthy.
The "Random Shoe" Check: Every time someone claims their new AI model is the best, they must prove it beats the "random shoe" baseline. If it doesn't, the fancy model isn't actually doing anything useful.

Summary

The paper is a call to action. It says the current way of evaluating AI for heart signals is like grading a student only on their ability to recite the alphabet while ignoring their ability to write a story. It reveals that many "advanced" AI models are barely better than a random guess, and that we need to start testing them on a wider, more clinically relevant set of tasks with stricter, more honest grading rules.

The Bottom Line: The field needs to stop chasing high average scores on narrow tests and start proving that these models can actually do the complex, real-world jobs doctors need them to do.

Technical Summary: Position: Evaluation of ECG Representations Must Be Fixed

Problem Statement

The field of 12-lead electrocardiogram (ECG) representation learning has converged on a limited set of benchmarking practices that the authors argue are insufficient for ensuring reliable progress and clinical alignment. Current evaluation relies heavily on three public multi-label datasets (PTB-XL, CPSC2018, and CSN) which focus predominantly on arrhythmia and waveform morphology classification. The authors contend that this narrow scope ignores the ECG's capacity to encode broader clinical information, such as structural heart disease, hemodynamic states, and patient-level forecasting.

Furthermore, the prevailing evaluation protocol suffers from methodological flaws:

Metric Limitations: The field relies almost exclusively on macro-AUROC (unweighted mean of per-label AUROCs). This metric obscures performance on individual clinically relevant tasks and amplifies noise from severely imbalanced labels (where many classes have very few positive examples).
Lack of Rigor: Uncertainty is rarely quantified (e.g., via confidence intervals), and statistical significance testing is often absent, making it difficult to distinguish genuine improvements from sampling noise.
Baseline Deficiencies: Many studies fail to compare learned representations against simple baselines, such as randomly initialized encoders.

Methodology

The authors conducted an empirical evaluation to test the robustness of current conclusions regarding ECG representation quality.

Models Evaluated:
Five representative pre-training approaches spanning different paradigms (contrastive learning, reconstruction, and multi-modal supervision) were evaluated:

CLOCS: Contrastive learning across space, time, and patients.
KED: Foundation model with clinical knowledge distillation.
HeartLang: ECG language model using reconstruction and masked-token prediction.
MERL: Multi-modal ECG-text alignment.
D-BETA: Multi-modal ECG-text with reconstruction regularization.
Baseline: A randomly initialized 1D ResNet-18 encoder.

Datasets and Tasks:
The models were evaluated across six settings, categorized into three task families:

Standard Benchmarks: PTB-XL (SUPER, SUB, FORM, RHYTHM subsets), CPSC2018, and CSN (focusing on arrhythmia and waveform).
Structural Disease: EchoNext (ECG paired with echocardiography to predict structural abnormalities like valve disease and ejection fraction).
Physiological & Forecasting:
- Hemodynamic Inference: Predicting mean pulmonary capillary wedge pressure (mPCWP) and mean pulmonary artery pressure (mPA) from private MGH data.
- Patient Forecasting: Predicting the development of heart failure (LVEF < 40%) within one year from private MGH data.

Evaluation Protocol:
The authors implemented a rigorous evaluation framework adhering to proposed best practices:

Linear Probing: All encoders were frozen, and $\ell_2$ -regularized logistic regression classifiers were trained on top of the embeddings.
Metrics: Reporting of both AUROC and AUPRC (Area Under the Precision-Recall Curve) for individual tasks, alongside macro-AUROC.
Uncertainty Quantification: 95% confidence intervals were calculated using paired bootstrapping (1,000 resamples).
Statistical Testing: Pairwise permutation tests were used to determine if performance differences were statistically significant.
Data Filtering: Labels with insufficient positive examples in the test set (e.g., <10) were excluded from quantitative evaluation to reduce noise.

Key Contributions

1. Critique of Current Benchmarking Practices

The paper demonstrates that the reliance on macro-AUROC and the inclusion of low-support labels distort method rankings. When labels with few positive examples are removed, the relative performance of methods changes significantly. Furthermore, the lack of uncertainty quantification in prior literature means that many reported "state-of-the-art" gains are statistically indistinguishable from noise.

2. The Random Encoder Baseline

A central and surprising finding is that a randomly initialized encoder with linear evaluation often matches or exceeds the performance of state-of-the-art pre-trained models on many tasks. Specifically:

On standard benchmarks (PTB-XL, CPSC2018, CSN), the random baseline frequently outperforms CLOCS and HeartLang.
On structural disease tasks (EchoNext), the random baseline is competitive with pre-trained methods, particularly at low data regimes (1% training data).
This suggests that for many ECG tasks, the clinically relevant signal is sufficiently apparent that random convolutions do not distort it, and the "learning" may be occurring primarily in the linear probe rather than the pre-training phase.

3. Proposed Evaluation Best Practices

The authors outline a set of recommendations for the community:

Expand Task Scope: Include structural disease, hemodynamics, and patient forecasting, not just arrhythmia.
Report Task-Specific Metrics: Prioritize per-task AUROC and AUPRC over aggregate macro-AUROC.
Quantify Uncertainty: Always report confidence intervals and use statistical tests for comparisons.
Filter Low-Support Labels: Exclude tasks with insufficient positive examples from headline metrics.
Adopt Random Baselines: Use a randomly initialized encoder as a mandatory baseline to assess the true utility of pre-training.

Results

Ranking Instability: When uncertainty is accounted for, the "winning" method on standard benchmarks is not consistent. For instance, KED and D-BETA often appear dominant via macro-AUROC, but their superiority vanishes when statistical significance is considered.
Task Heterogeneity: Performance varies widely across task types. While KED, MERL, and D-BETA show gains on specific arrhythmia tasks (e.g., ST/T changes), they do not universally dominate.
Structural Disease: On EchoNext, CLOCS and HeartLang consistently underperform compared to other methods and the random baseline. MERL and KED show stronger performance on structural endpoints.
Forecasting: For 1-year heart failure prediction, MERL and KED outperform the random baseline, though the random baseline still significantly outperforms CLOCS and HeartLang.
Robustness: The competitiveness of the random baseline was verified across a grid of 36 different configurations (varying sampling rates, filtering, normalization, and backbone architectures), confirming the finding is not an artifact of a specific setup.

Significance and Claims

The paper argues that the current trajectory of ECG representation learning is misleading due to flawed evaluation protocols. The authors claim that:

Current rankings are unreliable: Many conclusions about which pre-training method is "best" are driven by noise, low-support labels, and a lack of statistical rigor rather than genuine representational superiority.
Pre-training gains may be overstated: The fact that a random encoder is often competitive suggests that the marginal utility of complex pre-training objectives may be smaller than currently believed, or that the signal in ECGs is more linearly separable than anticipated.
The field must evolve: To ensure representations are clinically useful, the community must adopt broader, more rigorous benchmarks that reflect the full scope of ECG utility (structural, hemodynamic, and prognostic) and adhere to strict statistical reporting standards.

The authors position this work as a necessary correction to ensure that future methodological development in ECG representation learning is aligned with clinically meaningful objectives and reproducible scientific standards.