💻 computer science

Beyond full fine-tuning: enhancing generalizability in ECG foundation models' downstream adaptation

This study demonstrates that parameter-efficient fine-tuning (PEFT) strategies, such as partial fine-tuning and BitFit, outperform full fine-tuning in electrocardiogram foundation models by significantly enhancing generalizability and reducing overfitting, particularly on out-of-domain data and resource-constrained scenarios.

Original authors: Giuliana Monachino, Beatrice Zanchi, Georgiy Farina, Francesca Dalia Faraci

Published 2026-07-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giuliana Monachino, Beatrice Zanchi, Georgiy Farina, Francesca Dalia Faraci

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot chef who has spent years tasting millions of different dishes from every corner of the world. This robot, a "Foundation Model," has learned the general rules of cooking: how heat affects meat, how spices blend, and how dough rises. It's a culinary genius, but it's never actually cooked a specific dish for you yet. Now, you want this robot to make your favorite specific meal, like a perfect lasagna. The old way of doing this was to send the robot back to culinary school to re-learn everything from scratch, just for lasagna. But that takes forever, uses up all the electricity, and sometimes the robot gets so obsessed with your specific recipe that it forgets how to cook anything else, or worse, it memorizes your recipe so perfectly that it fails if you change the brand of cheese.

This paper is about a smarter way to teach the robot. Instead of making the whole robot re-learn everything, the researchers tried a technique called "Parameter-Efficient Fine-Tuning" (PEFT). Think of this as giving the robot a special, tiny cheat sheet or just letting it adjust a few specific knobs on its control panel, while leaving the rest of its massive brain frozen in place. The big question they wanted to answer was: Can we tweak just a few parts of this super-smart robot to make it great at a new task, without making it forget its general cooking skills or overheat the kitchen? They were especially curious if this method would help the robot handle ingredients it had never seen before, like a new type of pasta from a different country.

The Experiment: Tweaking the Knobs, Not the Whole Brain

The researchers took a massive AI model trained on hundreds of thousands of heart signals (ECGs) and tried to teach it to spot different heart problems. They compared the old "full fine-tuning" method—where the model re-learns every single connection in its brain—to several "PEFT" methods. These methods were like different ways of giving the robot a nudge:

Partial Fine-Tuning: Unfreezing just the last few layers of the model (like letting the robot adjust its final plating skills).
BitFit: Only adjusting the tiny "bias" numbers (like tweaking the salt shaker).
LayerNorm Tuning: Only adjusting the normalization layers (like calibrating the oven temperature).
Linear Probing: Freezing the whole brain and only training the final decision-making head (like telling the robot, "You know everything, just tell me if this is a heart attack or not").

They tested these methods on five different heart tasks, ranging from spotting a specific irregular heartbeat (Atrial Fibrillation) to diagnosing 22 different heart conditions. They also tested them with huge datasets, medium-sized ones, and tiny ones (down to just 1% of the data) to see if the method held up when data was scarce.

The Findings: Less is More (and Smarter)

The results were surprisingly clear. The researchers found that the "full fine-tuning" method, which tries to change everything, often made the model overfit. This is like a student who memorizes the practice test answers perfectly but fails the real exam because the questions are slightly different. The full fine-tuning model would get great scores on the training data but would struggle when faced with new, unseen heart signals.

In contrast, the PEFT methods, which only tweaked a small fraction of the model's parameters, did something magical. They not only saved a massive amount of computing power (making the process faster and cheaper) but also made the model more generalizable.

On familiar data: The PEFT models performed just as well as the full fine-tuning models, and often better, especially when the dataset was small. They didn't get confused or over-memorize the training examples.
On new data: This is where the magic really happened. When the researchers tested the models on heart signals from completely different sources (out-of-domain data) that the model had never seen before, the PEFT models significantly outperformed the full fine-tuning models. By not changing the whole brain, the models kept their "general knowledge" of how heart signals work, allowing them to adapt better to new situations.

Interestingly, the best method depended on the job. For simple tasks like spotting one specific heart rhythm, the simplest tweaks (like BitFit or LayerNorm) worked great. But for complex tasks, like diagnosing 22 different heart conditions, the researchers needed to unlock a few more layers (Partial Fine-Tuning) to give the model enough flexibility to learn the complexity. However, even in these complex cases, PEFT was still better at preventing overfitting than the full fine-tuning approach.

The Takeaway

The paper suggests that for adapting huge AI models to specific medical tasks, especially when data is limited, we don't need to overhaul the entire system. By using efficient techniques that only adjust a small number of parameters, we can create models that are not only faster to train but also more robust and reliable when they encounter real-world data that looks slightly different from their training. It turns out that sometimes, knowing when not to change everything is the key to making a model truly smart.

Technical Summary: Beyond Full Fine-Tuning in ECG Foundation Models

Problem Statement

Foundation Models (FMs) have demonstrated significant potential in electrocardiogram (ECG) analysis by learning general representations from vast unlabeled datasets via self-supervised learning (SSL). However, adapting these large-scale models to specific downstream tasks presents two critical challenges:

Computational Burden: Full fine-tuning (updating all model weights) requires substantial computational resources and memory, which is often prohibitive.
Overfitting and Poor Generalization: Given the high complexity of FMs (often comprising billions of parameters) and the frequent scarcity of labeled data for specific downstream tasks (especially for rare conditions), full fine-tuning risks overfitting. This limits the model's ability to generalize to unseen in-domain data and, more critically, to out-of-domain data where distribution shifts occur.

Current adaptation strategies in the ECG domain predominantly rely on full fine-tuning or linear probing (training only the classification head). Parameter-Efficient Fine-Tuning (PEFT) techniques, while established in Natural Language Processing (NLP) and Computer Vision, remain underexplored in biosignals and time-series analysis.

Methodology

The study proposes evaluating PEFT techniques as a superior alternative to full fine-tuning for adapting ECG Foundation Models. The research utilizes a pre-trained ECG FM (based on a CNN feature extractor and a transformer encoder, trained via contrastive learning) and compares several adaptation strategies across five downstream scenarios:

Binary Classification: Atrial Fibrillation (AF) detection (CinC2017, PTB-XL) and Life-Threatening Arrhythmia (LTA) detection (VFDB).
Multi-Class/Multi-Label Classification: Cardiac diagnosis with 5 superclasses and 22 classes (PTB-XL).

Adaptation Strategies Evaluated:

Full Fine-Tuning (Baseline): Updates all model parameters.
Partial Fine-Tuning: Unfreezes only the last $N$ transformer blocks (optimized per scenario, up to 6 blocks).
BitFit: Updates only the bias terms of the model.
LayerNorm Tuning: Updates only the LayerNorm parameters within the attention modules.
Linear Probing: Freezes the entire backbone and trains only the classification head.
Supervised Training from Scratch: Trained end-to-end with random initialization (as a reference).

Experimental Design:

Data Scaling: Experiments were conducted using 100%, 10%, and 1% of the training subjects to simulate data scarcity.
Evaluation Metrics: Performance was measured using AUROC (binary) and Macro-AUROC (multi-class).
Generalization Tests:
- In-Domain: Evaluation on held-out test sets from the same dataset used for fine-tuning.
- Out-of-Domain: Direct inference on completely unseen datasets (AFDB, GEORGIA, CUDB) not used in pre-training or fine-tuning.
Statistical Analysis: A two-sided Mann-Whitney U test with five different random seeds was used to determine statistical significance against the full fine-tuning baseline.

Key Results

1. Effectiveness of PEFT on In-Domain Data

Binary Tasks: For AF and LTA detection, most PEFT techniques (Partial FT, BitFit, LayerNorm) achieved performance comparable to full fine-tuning, despite updating a significantly smaller number of parameters.
Complex Tasks: For multi-class diagnosis (5 and 22 classes), techniques with very few trainable parameters (BitFit, LayerNorm, Linear Probing) struggled to match full fine-tuning performance, likely due to the high complexity of the task. However, Partial Fine-Tuning successfully matched full fine-tuning performance in most multi-class scenarios by unlocking a sufficient number of transformer blocks.
Data Scarcity: In low-data regimes (1% training size), supervised training from scratch suffered dramatic performance drops, whereas PEFT methods maintained robust performance, often outperforming full fine-tuning.

2. Generalization and Overfitting

Reduced Overfitting: PEFT methods consistently demonstrated a smaller generalization gap (difference between training and test performance) compared to full fine-tuning. This indicates that limiting trainable parameters reduces the model's tendency to memorize the training data.
In-Domain Superiority: In several AF detection scenarios (e.g., CinC2017-10%, PTB-XL-100%), PEFT methods achieved significantly higher test set performance than full fine-tuning.

3. Out-of-Domain Generalization

Enhanced Robustness: The primary advantage of PEFT was observed on out-of-domain datasets. PEFT models frequently outperformed full fine-tuning on unseen data (e.g., AFDB, GEORGIA, CUDB).
Preservation of Pre-trained Knowledge: By freezing the majority of the model, PEFT preserves the general representations learned during large-scale pre-training, making the models more robust to distribution shifts.
Counter-Intuitive Findings: In some cases, PEFT models that performed slightly worse than full fine-tuning on in-domain data (e.g., PTB-XL-1% for AF) performed comparably or better on out-of-domain data.

Key Contributions

The paper claims four primary contributions:

Validation of PEFT in ECG: Demonstrating that PEFT techniques can adapt ECG Foundation Models to downstream tasks with performance at least comparable to full fine-tuning.
Enhanced Robustness: Showing that PEFT reduces overfitting and improves generalization on unseen in-domain data.
Superior Out-of-Domain Performance: Proving that the generalization benefits of PEFT are even more pronounced on out-of-domain data, where full fine-tuning often fails to generalize.
Task-Dependent Strategy: Establishing that the optimal PEFT technique depends on task complexity and dataset characteristics. Simple tasks benefit from highly efficient methods (BitFit, LayerNorm), while complex, heterogeneous tasks require Partial Fine-Tuning to unlock sufficient model capacity.

Significance and Claims

The authors conclude that Parameter-Efficient Fine-Tuning (PEFT) represents a valuable alternative to full fine-tuning for ECG Foundation Models. The study argues that full fine-tuning is not always the optimal strategy, particularly regarding model generalization.

The significance of this work lies in its potential to facilitate the real-world deployment of ECG AI tools. By reducing computational costs and, more importantly, enhancing robustness to distribution shifts (a common issue in clinical settings where training and deployment data differ), PEFT addresses critical barriers to adoption. The paper emphasizes that selecting an appropriate PEFT strategy based on the specific downstream task and data characteristics is crucial for maximizing model utility and reliability.