💻 computer science

How Do Electrocardiogram Models Scale?

This paper systematically investigates scaling laws for electrocardiogram (ECG) models by training 120 models across varying sizes and paradigms, revealing that while self-supervised learning offers superior data and transfer efficiency, the optimal path to effective ECG foundation models depends on strategically aligning architecture (ResNet vs. Transformer) with the pre-training paradigm rather than relying solely on brute-force scaling.

Original authors: Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawei Li, Fabio Bonassi, Ming Jin, Stefan Gustafsson, Johan Sundström, Thomas B. Schön, Antônio H. Ribeiro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Does Bigger Always Mean Better?

Imagine you are trying to teach a robot to read an electrocardiogram (ECG)—the squiggly line that shows your heart's rhythm. In the world of AI, there is a popular belief called "scaling laws." It's the idea that if you make the robot's brain (the model) bigger and feed it more textbooks (the data), it will automatically get smarter.

However, for heart signals, this rule has been confusing. Sometimes a tiny robot works better than a giant one. Sometimes a robot trained on 1 million heartbeats fails, while one trained on 10,000 succeeds. The authors of this paper wanted to figure out why this happens and find the perfect recipe for building the best heart-AI.

The Experiment: A Massive Kitchen Test

To solve this, the researchers didn't just cook one meal; they cooked over 120 different versions of AI models.

The Ingredients: They used a huge public dataset called CODE, which contains 2.3 million heart recordings from 1.6 million patients.
The Tools: They tested two main types of "brains":
1. ResNets: These are like specialized chefs who are great at spotting local patterns (like a specific bump or dip in the heart line). They are efficient and good at recognizing shapes.
2. Transformers: These are like storytellers. They are great at understanding long-range connections and context, but they are often "greedy" and need massive amounts of food to learn.
The Cooking Styles: They tried two ways to teach these chefs:
1. Supervised Learning (SL): The teacher gives the student a textbook with the answers already written (labeled data). "This line means a heart attack."
2. Self-Supervised Learning (SSL): The teacher gives the student a puzzle with missing pieces and says, "Figure out the missing part based on the rest." The student learns the structure of the heart rhythm without being told specific diagnoses.

The Findings: What Actually Works?

1. The "Bottleneck" Problem

When they tested the models on data they had seen before (In-Distribution):

The Supervised Chef (SL) hit a wall. No matter how much they grew the chef's brain, performance stopped improving once they ran out of textbooks. They were data-bottlenecked. It's like having a genius student but only giving them one textbook; they can't get smarter without more books.
The Self-Supervised Student (SSL) kept growing. These models didn't hit a wall. Whether they got bigger or had more data, they kept getting better. They were hungry for both size and data.

2. The "Out-of-Distribution" Challenge (The Real Test)

The real test was seeing how well these models handled new, unseen heart conditions (Out-of-Distribution). This is like asking a chef who only cooked Italian food to suddenly make sushi.

The ResNet Advantage: The specialized chefs (ResNets) were much more efficient. To get the same improvement in performance, a Transformer needed to be 1.3 to 2.5 times larger than a ResNet. The ResNet was like a compact, high-performance sports car, while the Transformer was a heavy truck that needed more fuel to go the same distance.
The SSL Advantage: The self-supervised students were 16 times more efficient at learning from new data than the supervised students. They could generalize better because they learned the language of the heart, not just the specific answers.
The Transfer Efficiency: When moving to completely new tasks (like predicting patient sex or detecting structural heart disease), the SSL models were up to 7.6 times better at transferring their knowledge than the SL models.

3. The "Sweet Spot"

The paper found that the best results didn't come from just "brute-forcing" size.

Small to Medium Sizes: ResNets trained with Self-Supervised Learning (SSL) were the clear winners. They were the most efficient and accurate.
Huge Sizes: If you go to extremely massive sizes (hundreds of millions of parameters), the Transformers (SSL) finally catch up and overtake the ResNets. But for most practical purposes, the ResNet-SSL combo is the most efficient path.

The "Recipe" for Success

The authors conclude that building a great ECG foundation model isn't about just making the model bigger or buying more data. It's about matching the right tool to the right teaching method.

Don't just add data to a Supervised model: If you use a standard "textbook" approach, adding more data eventually stops helping.
Use Self-Supervised Learning: Let the AI learn the patterns of the heart on its own first. This makes it much better at handling new, unseen medical problems.
Pick the Right Architecture: For most sizes, a ResNet (the specialized chef) is more efficient than a Transformer (the storyteller). Only when you have massive resources should you switch to the Transformer.

Summary Analogy

Imagine you are training a detective to solve heart crimes.

Supervised Learning is giving the detective a list of 10,000 solved cases. They memorize the list well, but if a new type of crime appears, they are stuck.
Self-Supervised Learning is giving the detective a library of all crime scene photos and asking them to find the missing pieces. They learn how crimes happen.
ResNets are detectives with a magnifying glass (great for details).
Transformers are detectives with a wide-angle lens (great for big pictures).

The paper says: Give the detective with the magnifying glass (ResNet) the "find the missing pieces" training (SSL). This combination creates the most efficient, adaptable, and powerful detective for heart signals, without needing to build a detective so big they can't fit in the room.

Technical Summary: How Do Electrocardiogram Models Scale?

Problem Statement

While scaling laws have successfully established a framework for foundation models in natural language processing (NLP), their applicability to electrocardiogram (ECG) models remains poorly characterized. Recent empirical observations in the ECG domain are inconsistent: some studies suggest that increasing model or dataset size yields diminishing or non-monotonic returns, while others show that compact models can outperform significantly larger pre-trained counterparts. Existing literature often fails to disentangle the coupled effects of architectural inductive biases (e.g., CNNs vs. Transformers) and pre-training paradigms (Supervised Learning [SL] vs. Self-Supervised Learning [SSL]). Consequently, the roles of architecture, paradigm, and expected improvements with scale remain ambiguous, leaving practitioners without clear guidelines on how to allocate compute resources for effective ECG foundation models.

Methodology

To address these gaps, the authors conducted a systematic investigation involving the pre-training of over 120 models ranging from 20,000 to 200 million parameters. The study utilized the CODE dataset (2.3 million records, 1.6 million patients) as the primary pre-training source and the MIMIC-IV dataset to study label scaling.

The experimental design decoupled two primary variables:

Architectures: Two representative families were compared:
- ResNet: A 1D convolutional architecture adapted for ECG signals, known for local inductive biases aligning with signal morphology.
- Transformer: A vanilla Transformer backbone (specifically HeartLang) adapted for ECG, capable of capturing long-range temporal dependencies.
Pre-training Paradigms:
- Supervised Learning (SL): Trained on expert-annotated labels (e.g., rhythm disorders).
- Self-Supervised Learning (SSL): Implemented via Contrastive Multi-Segment Coding (CMSC) for ResNets and Masked Autoencoding for Transformers.

The authors evaluated these configurations across:

In-Distribution (ID): A held-out 10% subset of the CODE dataset.
Out-of-Distribution (OOD): Eleven diverse clinical tasks including arrhythmia detection (CPSC2018, CSN), structural heart disease detection (EchoNext), acute care prediction (MIMIC-ICU, Mortality), and patient characteristics (Sex).

The analysis relied on neural scaling laws (modeling loss as a power law of model size $N$ and data size $D$ ) and loss-to-loss scaling laws (predicting OOD loss from ID loss).

Key Contributions

Systematic Disentanglement: The study is the first to jointly analyze the scaling behaviors of ECG models by isolating the effects of architecture (ResNet vs. Transformer) and paradigm (SL vs. SSL) across a wide range of scales.
Identification of Bottlenecks: The authors characterize specific bottlenecks for different configurations, revealing that SL models are primarily data-bottlenecked in-distribution, whereas SSL models scale robustly across both dimensions.
Efficiency Metrics: The paper quantifies the efficiency gains of different configurations, providing concrete multipliers for parameter efficiency (ResNet vs. Transformer) and data efficiency (SSL vs. SL).
Compute-Optimal Allocation: The study derives optimal strategies for allocating a fixed compute budget between model size and dataset size for each architecture-paradigm combination.

Key Results

1. Scaling Behaviors and Bottlenecks

In-Distribution (ID): SL models exhibit a sharp saturation in parameter scaling (bottlenecked by dataset size), whereas SSL models show no immediate evidence of saturation, following irreducible-loss-free power laws.
Out-of-Distribution (OOD):
- Architectural Bias: ResNets are 1.3 to 2.5 times more parameter-efficient than Transformers. To match the loss reduction of a 10-fold increase in ResNet parameters, Transformers require significantly larger scaling (19x for SL, 288x for SSL).
- Paradigm Bias: SSL is up to 16 times more data-efficient than SL. SSL consistently improves OOD data scaling efficiency across both architectures.

2. Transfer Efficiency (Loss-to-Loss)

SSL models demonstrate superior transfer efficiency compared to SL. The transfer exponent ( $\kappa$ ), which governs how ID improvements translate to OOD gains, is significantly higher for SSL.
For unseen clinical tasks (e.g., structural heart disease detection, sex prediction), SSL achieves up to 7.6 times higher transfer efficiency than SL.
SL models often operate near their ID loss floor, meaning further ID improvements yield diminishing returns for OOD generalization.

3. Absolute Performance and Scale

ResNet Dominance: Across observed scales, ResNet-based models generally achieve the lowest absolute OOD loss, highlighting a strong architectural inductive bias for ECG signals.
Transformer-SSL at Scale: While ResNets dominate at smaller to medium scales, Transformer-SSL models eventually overtake ResNet-SSL at very large model sizes (approaching 200M parameters), suggesting Transformers require significant scale to overcome their initial parameter inefficiency.
Label Scaling: Increasing the number of supervised labels improves ID performance but yields highly variable OOD gains depending on label composition. Simply scaling supervised labels is suboptimal compared to the benefits of SSL.

4. Compute-Optimal Allocation

SL Models: Under optimal allocation, both ResNet-SL and Transformer-SL direct >70% of marginal compute to data rather than model size.
SSL Models: Transformer-SSL inverts this pattern, directing ~73% of marginal compute to model size, favoring larger models over larger datasets.

Significance and Claims

The paper claims that the path to effective ECG foundation models lies not in "brute-force" scaling of a single configuration, but in the strategic alignment of architecture and paradigm.

Resolution of Conflicting Literature: The study resolves previous inconsistencies in the literature by showing that non-monotonic scaling behaviors arise from the specific coupling of architecture and paradigm. For instance, the failure of small models to scale in some studies may be attributed to using SL with architectures that are data-bottlenecked, while the success of compact models in others may stem from the inherent efficiency of ResNets in low-data regimes.
Practical Guidance: The findings suggest that for tasks with limited labeled data or requiring generalization to unseen clinical conditions, Self-Supervised Learning is the superior paradigm. Furthermore, while Transformers are powerful at massive scales, ResNets offer superior parameter efficiency for most practical scales.
Broader Implications: The authors note that these insights regarding non-monotonic scaling and the importance of disentangling architectural and paradigmatic biases likely extend to other biosignal modalities, such as EEG and PPG.

The study concludes that effective scaling in the ECG domain requires recognizing that "one size does not fit all"; the optimal strategy depends heavily on the available compute budget, the specific clinical task (ID vs. OOD), and the choice of pre-training objective.