🤖 machine learning

LSTrans: Efficient Knowledge Transfer for Lightweight and Automated ECG Classification

The paper proposes LSTrans, a lightweight hybrid model combining a specialized 1D convolutional backbone with a Transformer encoder and knowledge distillation techniques to achieve efficient, high-sensitivity ECG classification suitable for resource-constrained wearable devices.

Original authors: Yi Zhao, Jiajun Gao, Chenyang Xu, Yuxi Zhou, Hao Wang

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yi Zhao, Jiajun Gao, Chenyang Xu, Yuxi Zhou, Hao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your heart is a busy orchestra, and an Electrocardiogram (ECG) is the sheet music recording every beat. Doctors need to read this music instantly to spot trouble, but the "super-smart" computer brains (deep learning models) that are really good at this are usually too heavy and hungry for power to fit on a tiny wearable device like a smartwatch. They are like a massive supercomputer trying to run on a wristwatch battery—it just doesn't work.

The authors of this paper, Yi Zhao and their team, built a new solution called LSTrans. Think of LSTrans as a "lightweight detective" designed specifically to fit on those small devices without losing its sharp eyes.

The Detective's Toolkit: A Hybrid Approach

Instead of using just one type of tool, LSTrans uses a clever mix. It has a 1D-LSNet backbone, which is like a specialized set of magnifying glasses that look at the raw heartbeat signal directly.

The "Even" Layers: These act like a high-speed camera zooming in on tiny, sudden glitches in the music (microscopic morphological variations), like a skipped beat or a weird wave shape. They use special filters to catch these sharp details without getting confused by background noise.
The "Odd" Layers: These act like a wide-angle lens, stepping back to see the big picture of the rhythm over time (macroscopic trends). They use a "Large-Kernel Perception" module to understand the long-range story of the heartbeat, connecting dots that are far apart.

To make sure this detective doesn't miss the big picture, they added a Transformer encoder at the end. If the magnifying glasses are the eyes, the Transformer is the brain that connects all the clues to understand the full story of the heart's rhythm.

The "Teacher-Student" Trick

Here is the really clever part: How do you teach a tiny, lightweight model to be as smart as a giant, heavy one? The authors used Knowledge Distillation.
Imagine a master chef (the "Teacher" model) who has spent years learning to cook perfect dishes. Instead of trying to shrink the master chef into a tiny kitchen, the chef writes a detailed cookbook and teaches a young apprentice (the "Student" model, which is LSTrans) how to cook.

Homogeneous Distillation: The teacher and student are built the same way, just different sizes. It's like a senior apprentice teaching a junior one.
Heterogeneous Distillation: The teacher is a completely different, massive architecture (like a giant ResNet). The student learns from this "heavy" teacher's wisdom, effectively stealing the teacher's experience to become smart without being heavy.

The paper found that the Heterogeneous approach (learning from the giant, different teacher) worked best. It suggests that borrowing the "structural priors" (the deep, over-parameterized habits) from a massive model helps the tiny student spot rare, tricky heart problems that it might otherwise miss.

The "Low-Rank" Shortcut

To keep the model from getting too heavy during training, they used a technique called Low-Rank Adaptation (LoRA).
Think of the model as a giant library of books. Usually, to update the library, you'd have to rewrite every single book. That takes forever and uses a lot of paper. LoRA is like sticking a few sticky notes in the margins of the books instead of rewriting them. The authors found that for the "local" parts of the model (the magnifying glasses), they only needed a tiny number of sticky notes (a rank of 4), but for the "global" parts (the brain), they needed a few more (a rank of 16). This kept the model small and prevented it from "forgetting" what it already knew.

The Results: Fast, Light, and Accurate

The team tested LSTrans on three major datasets (G12EC, PTB-XL, and Chapman-Shaoxing) using 12-lead ECG signals.

Performance: The "Hetero" version of LSTrans achieved an average AUC of 0.949 and an Fβ=2 score of 0.716. The paper notes this is competitive with much larger models, like the massive ECG-Founder (which has 30.68 million parameters), while LSTrans only uses about 3.19 million parameters for the student.
Efficiency: This is where it shines. Compared to the heavy ECG-Founder, LSTrans reduced the peak memory usage by about 36.4% and ran nearly 3 times faster. Compared to another lightweight method called H-Tuning, it cut memory usage by 41.09% and sped up training by 4.77 times.
Safety: In medical terms, they care deeply about not missing a diagnosis (false negatives). By using a specific metric called Fβ=2 (which weighs "missing a case" as twice as bad as a false alarm), LSTrans showed significant improvements, reducing errors by roughly 11% compared to some other lightweight methods on specific datasets.

What They Say It's NOT

The paper is very clear about what doesn't work well for this specific goal:

No 2D Images: They argue against converting the 1D heartbeat signals into 2D images (like spectrograms). They say this can cause "phase alignment errors" and adds unnecessary computational weight. Their model works directly on the raw 1D signal.
No Simple Shrinking: Just making a big model smaller (simple compression) often leads to missing subtle heart anomalies. Their complex "interleaved" design is necessary to keep the sensitivity high.
No "Magic" Inference Data: The authors admit they haven't measured the speed on actual wearable hardware yet. Their speed numbers are based on training on a powerful NVIDIA RTX 4090 GPU. They suggest that real-world edge devices might have different challenges with their dynamic convolution methods.

The Bottom Line

The authors suggest that LSTrans offers a "competitive balance" between being smart enough to catch heart issues and being light enough to run on future wearables. They proved this through extensive experiments on standard datasets, showing that by mixing a specialized 1D backbone with a Transformer and using smart teaching methods (distillation) and shortcuts (LoRA), you can get near-expert performance without the heavy baggage. However, they note that real-world testing on actual wearable patches is the next step to confirm it works outside the lab.

Technical Summary: LSTrans

Problem Statement

The deployment of deep learning models for automated electrocardiogram (ECG) classification on resource-constrained wearable devices and Internet of Healthcare Things (IoHT) nodes faces significant barriers. Standard deep networks impose high computational costs and memory footprints that exceed the strict limits of edge hardware. While existing solutions attempt to address this, they often suffer from specific trade-offs:

Complex Architectures: Multi-branch designs (e.g., ECGTransForm, MTA-Net) capture necessary spatial-temporal features but incur heavy computational overhead.
Limited Scope: Lightweight networks (e.g., CrossStateECG-Lite) often fail to support multi-lead, multi-label clinical diagnoses, restricting themselves to single-lead biometric settings.
Conversion Artifacts: Methods converting 1D ECG signals to 2D spectrograms (e.g., HMT-KD) introduce phase alignment errors and additional computational costs.
Compression Risks: Direct model compression often degrades diagnostic recall, leading to critical false-negative diagnoses on subtle morphological anomalies.

Consequently, there is a need for a model that balances high diagnostic sensitivity (particularly for rare anomalies) with minimal memory usage and training latency.

Methodology

The authors propose LSTrans, a lightweight hybrid model designed to capture both macroscopic rhythmic trends and microscopic morphological variations efficiently. The framework integrates a specialized 1D convolutional backbone with a Transformer encoder, optimized via Low-Rank Adaptation (LoRA) and Knowledge Distillation (KD).

1. 1D-LSNet Backbone

Instead of converting signals to 2D, LSTrans processes raw 1D ECG signals directly through a hierarchical temporal abstraction. The backbone features an interleaved architecture alternating between even-indexed and odd-indexed layers to balance dual-scale feature capture:

Even-Indexed Layers (Local Micro-Morphology): These layers utilize a Reparameterized Module (RepVGGDW) combined with Squeeze-and-Excitation (SE). The RepVGGDW captures multi-scale temporal patterns (e.g., ST-segment elevations, Q-waves) and is folded into a single kernel during inference for zero latency. The SE block dynamically recalibrates channel weights to suppress background artifacts and highlight diagnostically relevant leads.
Odd-Indexed Layers (Global Context & Noise Filtering): These layers employ a Large-Kernel Perception (LKP) and Small-Kernel Aggregation (SKA) module. LKP generates dynamic weights from the temporal context to adapt to non-stationary signal variations. SKA applies these weights to local high-resolution windows to filter baseline noise and emphasize critical peaks. In the final stage (Stage 3), a 1D multi-head self-attention mechanism is introduced to model long-range dependencies (e.g., irregular intervals).

2. Transformer Integration and LoRA

The output of the 1D-LSNet is fed into a Transformer encoder to model long-term heart rhythm dependencies. To ensure parameter efficiency, the authors employ Low-Rank Adaptation (LoRA):

Strategic Rank Allocation: A minimal rank ( $r_c=4$ ) is assigned to convolutional layers (which extract translation-invariant motifs like QRS complexes), acting as a regularizer against high-frequency noise. A higher rank ( $r_t=16$ ) is assigned to Transformer mixers to preserve the high intrinsic dimensionality of global temporal dynamics.
Freezing Strategy: The backbone remains largely frozen to preserve pre-trained morphological representations, with only low-rank adapters updated during fine-tuning. This prevents catastrophic forgetting and stabilizes the optimization trajectory.

3. Knowledge Distillation Framework

To transfer diagnostic expertise from high-capacity teacher models to the lightweight student, two paradigms are explored:

Homogeneous Distillation (KD-Homo): Uses a teacher with the same interleaved backbone structure, enforcing self-refinement within the same structural manifold.
Heterogeneous Distillation (KD-Hetero): Utilizes a heavy residual network and Transformer teacher. This allows the student to inherit diverse, over-parameterized structural priors, effectively emulating the representational capacity of a deep hybrid model.

The training objective combines binary cross-entropy loss with a distillation loss, scaled by a temperature factor $T$ and a balancing hyperparameter $\alpha$ .

Key Contributions

Refactored 1D-LSNet: The authors refactor the 2D heteroscale paradigm into a specialized 1D backbone that processes raw ECG signals directly, maximizing efficiency while retaining critical morphological representations.
Interleaved Architecture: An alternating block design couples dynamic large-kernel filters for global context with multi-scale, lead-attentive units for local morphology. This prevents signal degradation during downsampling and minimizes false-negative errors.
Diverse Distillation Strategies: The paper systematically analyzes both homogeneous and heterogeneous distillation, demonstrating that structured teacher-student transfer effectively restores diagnostic sensitivity under constrained resources.

Experimental Results

Experiments were conducted on three 12-lead ECG datasets: G12EC, PTB-XL, and Chapman-Shaoxing, following a patient-disjoint 10-fold cross-validation protocol.

Performance: The LSTrans (Hetero+LoRA) variant achieved an average AUC of 0.949 and an average $F_{\beta=2}$ score of 0.716. It outperformed specialized lightweight baselines like H-Tuning and CE-SSL, showing relative improvements of approximately 14.6% and 29.5% in $F_{\beta=2}$ scores, respectively.
Comparison with Foundation Models: Despite having significantly fewer parameters (3.19M student vs. 30.68M for ECG-Founder), LSTrans matched the performance of the large-scale foundation model, narrowing the absolute gap in macro $G_{\beta=2}$ to only 0.036 points on PTB-XL.
Efficiency: LSTrans demonstrated substantial efficiency gains:
- Reduced peak GPU memory footprint by approximately 36.4% compared to ECG-Founder and 41.09% compared to H-Tuning.
- Accelerated average training runtime by nearly 3 times compared to ECG-Founder and 4.77 times compared to H-Tuning.

Significance and Limitations

The paper claims that LSTrans offers a competitive balance between diagnostic sensitivity and resource efficiency, making it suitable for deployment on edge devices where memory and power are limited. By combining a specialized 1D backbone with efficient adaptation and distillation, the framework reduces peak memory footprints and training latency without compromising the ability to detect subtle cardiac anomalies.

The authors acknowledge several limitations:

Efficiency evaluations are based on training-phase metrics; actual inference latency on non-GPU edge hardware remains unmeasured.
Runtime weight generation in dynamic convolutions may incur control-flow overhead on resource-constrained platforms.
Generalization remains dependent on large-scale pretraining.

Future work is proposed to focus on optimizing dynamic operators, measuring real-world edge latencies, and conducting clinical validation on wearable ECG patches.