Benchmarking ECG FMs: A Reality Check Across Clinical Tasks

This paper benchmarks eight ECG foundation models across 26 clinical tasks, revealing that architectural choices like ECG-CPC's compact state-space design often outweigh massive scale in performance and label efficiency, while highlighting significant remaining gaps in cardiac structure and outcome prediction.

The Gist

Imagine you are trying to teach a computer to read an electrocardiogram (ECG)—the squiggly lines on a heart monitor that tell doctors if a heart is healthy or in trouble. For years, researchers have been building "specialist" computers, each trained to spot just one specific problem, like a heart attack or an irregular rhythm.

But what if we could build a universal heart expert? A computer that learns from millions of heartbeats first, and then can be quickly taught to solve any heart-related puzzle, from diagnosing diseases to predicting if a patient will need the ICU. This is the promise of Foundation Models (FMs).

However, the field is a bit like the Wild West. Everyone is claiming their model is the best, but they are testing them on different, tiny puzzles. This paper is a massive "reality check" to see who actually wins.

Here is the story of their findings, explained simply:

1. The Great Race: 8 Contenders, 26 Challenges

The researchers gathered 8 different AI models (the contenders) and put them through a grueling obstacle course of 26 different medical tasks.

• The Tasks: Some were easy (identifying if a patient is male or female), some were hard (predicting if a patient will die within 30 days), and some were in the middle (diagnosing specific heart conditions).
• The Test: They tested the models in two ways:
  • The "Frozen" Test: The model is a finished product. You can't change its brain, you just add a simple "answer sheet" on top.
  • The "Fine-Tuning" Test: You are allowed to tweak the model's brain slightly to learn the specific task better.

2. The Big Surprise: Size Doesn't Matter (Much)

In the world of AI, there is a popular belief that "Bigger is Better." The idea is that if you build a massive, complex brain with billions of parameters, it will automatically be smarter.

The paper's biggest shocker: This isn't true for heart data.

• The Heavyweights: Several models were massive Transformers (like the ones that power advanced chatbots). They were huge, complex, and required supercomputers to train.
• The Lightweight Champion: The winner was a model called ECG-CPC. It was tiny—orders of magnitude smaller than the giants. It was like bringing a sleek sports car to a race against heavy cargo trucks.
• The Result: ECG-CPC won or tied in 5 out of 7 categories. It proved that having the right architecture (the internal design) is more important than just having a big brain. The "sports car" design was perfectly suited for the "road" of heart signals, while the "cargo trucks" were over-engineered and clumsy.

3. The "Label Efficiency" Superpower

Imagine you are teaching a child to recognize dogs.

• The Old Way (Supervised Baseline): You show them 1,000 pictures of dogs and say, "This is a dog."
• The Foundation Model Way: You show them 1,000 pictures of dogs, but the model has already studied 10 million pictures of animals, weather, and landscapes.

The paper found that these pre-trained models are 3.3 to 9 times more efficient.

• If a standard model needs 1,000 examples to learn a task, the Foundation Model might only need 100 to 300.
• Why this matters: In medicine, data is often scarce. Finding 1,000 patients with a rare disease is hard. Finding 100 is easy. These models can learn effectively even when data is thin, making them perfect for rare diseases.

4. Different Brains, Same Results

The researchers looked inside the "brains" of the models using a special X-ray called CKA (Centered Kernel Alignment).

• They expected the winning models to look similar inside.
• The Twist: They didn't. The models that got the best scores had completely different internal structures.
• The Analogy: It's like two chefs making the exact same delicious cake. One uses a stand mixer and a specific recipe; the other uses a whisk and a different method. They arrive at the same tasty result, but the path they took was totally different. This means there isn't just one way to build a great heart AI; there are multiple paths to success.

5. The "Gap" in the Road

While the news is mostly good, the paper also points out where the road is still bumpy.

• The models are great at diagnosing heart rhythm issues (the "what is wrong with the beat" questions).
• They are not yet great at predicting complex outcomes like "Will this patient need surgery?" or "What is their exact heart structure?"
• It's like the AI is a brilliant mechanic who can hear a strange noise in the engine, but it's still learning how to predict if the car will break down next week.

The Takeaway

This paper tells us that we don't need to build bigger and bigger "monsters" to solve heart problems. Instead, we need smarter, more efficient designs (like the lightweight ECG-CPC).

These models are like universal translators for the heart. They learn the "language" of heartbeats once, and then they can speak fluently in many different medical dialects. While they aren't perfect yet, they are a massive leap forward, offering hope for better, faster, and more accessible heart care for everyone.

Technical Summary

Here is a detailed technical summary of the paper "Benchmarking ECG FMs: A Reality Check Across Clinical Tasks."

1. Problem Statement

Despite the growing availability of large-scale Electrocardiogram (ECG) datasets, machine learning for ECG interpretation remains fragmented. Current approaches are often limited to narrow tasks or specific datasets, and the evaluation of Foundation Models (FMs) in this domain is inadequate.

• Key Gaps: Existing benchmarks often compare FMs against weak baselines or focus on single task categories, preventing generalizable conclusions.
• Unanswered Questions:
  • Do larger models consistently outperform smaller, task-specific ones?
  • Which architectures (Transformers, CNNs, State-Space Models) yield the most transferable representations?
  • How do different pretraining strategies affect label efficiency across diverse clinical domains?
  • Is massive scale the primary driver of performance, or do architectural inductive biases matter more?

2. Methodology

The authors conducted a comprehensive benchmarking study involving 8 ECG Foundation Models and 2 supervised baselines across 12 public datasets and 26 clinically relevant tasks.

Models Evaluated

The study compared diverse architectures and pretraining paradigms:

• Transformers: ECG-JEPA (JEPA), ST-MEM (Masked Autoencoder), HuBERT-ECG, ECG-FM.
• CNNs: ECGFounder, MERL, ECGFM-KED.
• Structured State-Space Models (SSMs): ECG-CPC (proposed by authors), S4 (supervised baseline).
• Baselines: Net1D (CNN) and S4 (SSM) trained from scratch.

Benchmarking Pipeline

• Task Categories: The 26 tasks were grouped into seven categories:
  1. Adult ECG Interpretation (diagnosis).
  2. Pediatric ECG Interpretation.
  3. Cardiac Structure & Function (echocardiogram prediction).
  4. Cardiac Outcomes (ICD-10 discharge diagnoses).
  5. Non-Cardiac Outcomes.
  6. Acute Care Predictions (deterioration, mortality, ICU admission).
  7. Patient Characteristics (sex, age, biometrics, lab values).
• Evaluation Modes:
  • Full Fine-tuning: Training the entire model with a linear head.
  • Frozen Evaluation: Using the pretrained encoder as a fixed feature extractor with a learnable query-attention head.
  • Linear Evaluation: Freezing the encoder and training only a linear classification/regression head.
• Metrics: Macro-averaged AUROC for classification and Mean Absolute Error (MAE) for regression. Statistical significance was determined via bootstrapping.
• Scaling Analysis: Investigated label efficiency by training on subsets of data (from 250 to 1000+ samples) to measure how quickly models converge compared to supervised baselines.
• Representation Analysis: Used Centered Kernel Alignment (CKA) to analyze the internal similarity of layer representations, investigating whether high-performing models learn similar or distinct features.

3. Key Contributions

1. Comprehensive Benchmark: The first large-scale, systematic evaluation of ECG FMs covering 8 models, 12 datasets, and 26 tasks across classification and regression.
2. Introduction of ECG-CPC: The authors propose ECG-CPC, a lightweight SSM trained with Contrastive Predictive Coding (CPC) on the HEEDB dataset. Despite having only 3.8M parameters (orders of magnitude smaller than Transformer-based FMs), it achieves state-of-the-art performance.
3. Architecture vs. Scale Insight: Demonstrated that architecture matters more than scale. Compact SSMs outperformed massive Transformers in 5 out of 7 task categories.
4. Label Efficiency Quantification: Showed that FMs improve label efficiency by 3.3x to 9x compared to supervised baselines, though this varies significantly by architecture (e.g., ECG-JEPA is most efficient in low-data regimes, while ECG-CPC excels in high-data regimes).
5. Representation Divergence: Revealed that models with similar downstream performance learn markedly different internal structures, suggesting multiple viable paths to effective ECG representation.

4. Key Results

Performance Across Domains

• Adult ECG Interpretation: ECGFounder, ECG-JEPA, and ECG-CPC consistently outperformed strong supervised baselines (S4).
• Pediatric ECG: ECG-JEPA dominated, despite lacking pediatric pretraining data.
• Cardiac Structure & Function: ECG-CPC ranked first, followed by ECGFounder.
• Outcomes (Cardiac/Non-Cardiac/Acute Care): ECG-CPC dominated these categories, often matching or exceeding the supervised baseline where other FMs failed.
• Patient Characteristics: ECG-CPC ranked first in 5 of 6 tasks.

Architecture Efficiency

• SSM Superiority: The SSM-based ECG-CPC outperformed much larger Transformer and CNN models despite having significantly fewer parameters. This challenges the assumption that FM quality scales primarily with parameter count.
• Frozen vs. Fine-tuned: Strong models (ECG-CPC, ECG-JEPA, ECGFounder) maintained high performance even in frozen evaluation modes, proving their utility as feature extractors.
• Scaling Behavior:
  • ECG-JEPA: Steep learning curve (fast learning with few samples) but lower performance ceiling.
  • ECG-CPC: Slower initial learning but reaches a higher performance ceiling with more data.

Representation Analysis (CKA)

• ECG-CPC: Exhibited a clear, structured evolution of representations from local CNN features to sequential SSM features, with distinct specialization in the final layers.
• Transformers (JEPA/ST-MEM): Showed high redundancy in middle layers (Blk1-Blk10 were nearly identical), suggesting potential inefficiency or "representation collapse."
• CNNs (ECGFounder): Showed smooth gradients but high similarity in mid-network layers, indicating potential overparameterization.

5. Significance and Implications

• Redefining FM Requirements: The success of the small-scale ECG-CPC suggests that architectural inductive biases (specifically those of SSMs regarding long-range dependencies and spectral filtering in physiological signals) are more critical than massive parameter counts for ECG tasks.
• Practical Deployment: SSMs offer a balanced trade-off, providing superior predictive performance with moderate computational costs compared to Transformers, making them highly attractive for real-world clinical deployment.
• Label Efficiency: The study provides actionable guidance for practitioners: use ECG-JEPA for scenarios with extremely limited data (<1,000 samples) and ECG-CPC for larger datasets where maximal performance is required.
• Evaluation Standards: The paper argues that task performance alone is insufficient for evaluating FMs; representation analysis (like CKA) is necessary to understand model efficiency and internal mechanisms.
• Open Science: The authors released code, model weights, and the benchmarking framework to foster reproducibility and further research in ECG foundation models.

In conclusion, this paper serves as a "reality check," demonstrating that while FMs hold great promise for ECG analysis, the field must move beyond the "bigger is better" paradigm. Instead, it should focus on architectural suitability for physiological signals, with Structured State-Space Models emerging as a highly efficient and effective solution.