EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillation

Technical Summary: EVL-ECG

Problem Statement
High-fidelity Electrocardiogram (ECG) interpretation increasingly relies on massive Vision-Language Models (VLMs). However, deploying these foundation models in clinical edge-care settings is hindered by extreme computational demands. While Knowledge Distillation (KD) offers a compression solution, existing methods struggle when transferring knowledge from large VLM teachers to efficient student models. Specifically, traditional KD approaches fail to address two critical barriers in cross-architecture distillation:

1. Severe Vocabulary Mismatch: Differences in tokenizers between the teacher and student models.
2. Unbalanced Visual Tokens: Discrepancies in visual encoders leading to mismatched token counts and representations.
   Standard methods often treat these mismatches in isolation, failing to capture the complex spatio-temporal dependencies inherent in ECG signals, such as the global structural relationships across leads and the fine-grained morphological features of waveforms.

Methodology: EVL-ECG Framework
The authors propose EVL-ECG, a Multi-Aspect Heterogeneous Knowledge Distillation framework designed to bridge the gap between massive teachers and efficient students. The framework integrates three ECG-aware innovations:

1. Optimal Transport-based Visual Feature Matching:
   To preserve the global spatial topology of ECG images (e.g., the standard 12-lead grid), the method treats teacher and student visual tokens as empirical distributions. It employs Optimal Transport (OT) regularization to solve for an optimal transport plan that minimizes the entropically regularized Sinkhorn distance. This ensures the student model aligns its visual feature map with the teacher's distribution, preventing spatial misalignment of critical diagnostic regions (e.g., confusing precordial leads with limb leads) despite mismatched token representations.

2. Multi-Head Cross-Attention (MHCA) Alignment:
   To address the sequence length disparity between the teacher's dense, high-resolution representations and the student's compact latent space, the framework utilizes Multi-Head Cross-Attention. The student's hidden states act as queries, while the teacher's states serve as keys and values. This allows the student to adaptively isolate and aggregate diagnostically relevant features (e.g., localized ectopic beats, ST-segment deviations) without enforcing strict positional matching. Theoretically, this mechanism is interpreted as an Entropic Barycentric Projection under an OT framework, aligning distributed cardiac patterns rather than individual tokens.

3. Geometric Intra-Architecture Relation Matching:
   Recognizing that ECG diagnosis relies on structural topology and temporal relationships (e.g., P-wave to QRS intervals), this module distills the teacher's internal diagnostic logic. It defines two relation potentials:

   • Distance-wise potential: Captures magnitude and interval-based signal relationships via pairwise Euclidean distance.
   • Angle-wise potential: Captures directional morphology and electrical axis via pairwise cosine similarity.
     The distillation loss aligns these structural potentials between the projected teacher and student representations, ensuring the student inherits the teacher's geometric reasoning regarding the ECG signal's morphology.

The total objective function combines these components with a standard cross-entropy loss, weighted by hyperparameters to balance reconstruction and relational constraints.

Key Contributions

1. Unified Heterogeneous KD Framework: EVL-ECG resolves tokenizer and visual-token mismatches simultaneously through MHCA alignment and OT regularization, ensuring the preservation of critical morphological features in ECG images.
2. Geometric Relation Matching: The introduction of a module that distills internal diagnostic logic via distance- and angle-wise potentials, enabling the student to inherit sophisticated reasoning patterns for complex cardiac analysis while maintaining a resource-friendly size.
3. Efficient Foundation Model: The establishment of an efficient 2B-parameter ECG foundation model suitable for resource-constrained clinical environments.

Experimental Results
EVL-ECG was evaluated across multiple ECG benchmarks (PTB-XL Super, CODE-15%, ECG-QA, CPSC-2018, CSN, G12EC, MMMU-ECG) using metrics including Macro AUC, Macro F1, Hamming Loss, and Accuracy.

• Performance: The model outperforms existing state-of-the-art distillation methods (e.g., ULD, MultiLevelOT, DSKD, EM-KD) and proprietary VLMs (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet).
• Specific Gains: Improvements of up to 2.4% AUC and 1.1% clinical accuracy were observed over existing baselines. For instance, on PTB-XL-Super, EVL-ECG achieved a 75.2 AUC compared to GPT-4o's 55.6.
• Robustness: The model demonstrated improved modeling of inter-lead dependencies and enhanced sensitivity to subtle morphological variations, such as ST-segment deviations and irregular QRS complexes.
• Efficiency: In terms of computational footprint, EVL-ECG requires 8.54 seconds per step and 56.4 GB of VRAM, offering a competitive trade-off compared to heavier distillation frameworks like EM-KD.

Significance and Claims
The paper claims that EVL-ECG successfully bridges the gap between high-fidelity, massive foundation models and the practical constraints of clinical edge deployment. By specifically addressing the heterogeneity of architectures and the unique spatio-temporal nature of ECG signals, the framework enables the deployment of sophisticated reasoning patterns in resource-constrained medical environments. The authors emphasize that their approach preserves clinically meaningful structures—such as inter-lead consistency and waveform geometry—rather than merely optimizing aggregate prediction scores.

The authors maintain a modest stance regarding clinical deployment, noting that EVL-ECG is a decision-support model rather than a standalone diagnostic system. They acknowledge limitations regarding real-world noise (baseline wander, artifacts) and the need for external validation on diverse, multi-ethnic datasets to mitigate potential biases. Future work is proposed to enhance robustness against clinical noise and extend the paradigm to capture multi-lead temporal dependencies more deeply.