Echo2ECG: Enhancing ECG Representations with Cardiac Morphology from Multi-View Echos

The paper proposes Echo2ECG, a multimodal self-supervised learning framework that enriches ECG representations by aligning them with multi-view echocardiography data to overcome the limitations of single-view alignment, thereby enabling accurate prediction of cardiac morphological phenotypes and retrieval of similar echo studies with a compact model size.

The Gist

Imagine your heart is a complex, bustling city. To understand how this city is running, doctors have two main tools:

1. The ECG (Electrocardiogram): This is like listening to the city's power grid. It's cheap, fast, and everywhere. It tells you if the electricity is flowing smoothly or if there's a short circuit (like an arrhythmia). However, it can't tell you if the city's buildings are crumbling or if the roads are too narrow. It hears the noise of the electricity but can't see the shape of the city.
2. The Echo (Echocardiogram): This is like sending a drone to take high-definition photos of the city from every angle. It shows you the size of the heart chambers, how well the "pump" is working, and the shape of the valves. But, drones are expensive, require a skilled pilot (a sonographer), and aren't available in every small town.

The Problem

Doctors have long wanted to use the cheap, easy ECG to predict the detailed structural problems usually only seen by the expensive Echo.

Previous AI attempts tried to teach the ECG to "see" the heart by showing it pictures from the Echo. But they made a critical mistake: they only showed the AI one single photo from the Echo (like looking at the city only from the north side).

The ECG, however, listens to the entire heart's electrical activity from all sides. Trying to match a "global" electrical song to a "local" single photo is like trying to guess the layout of a whole house just by looking at one brick. The AI got confused because the information didn't match up.

The Solution: Echo2ECG

The researchers behind this paper, Echo2ECG, fixed this by changing the training method.

Instead of showing the AI just one photo, they showed it the entire drone flight video (all the different views of the heart). They taught the AI to listen to the ECG's "power grid song" and match it to the complete 3D shape of the heart city.

Think of it like this:

• Old Method: You try to learn what a whole apple looks like by only looking at a single slice of it.
• Echo2ECG: You look at the whole apple, peel it, slice it, and see it from every angle, then you learn to recognize that apple just by hearing the sound of someone biting into it.

How It Works (The Magic Trick)

The team used a "self-supervised" approach. They didn't need a human to write a report for every single pair of ECG and Echo. Instead, they let the AI learn on its own:

1. The Teacher: A powerful AI that already knows how to read Echo images (the drone photos) was frozen (kept as a reference).
2. The Student: A smaller AI that reads ECGs (the power grid).
3. The Lesson: The system showed the "Student" an ECG and the "Teacher" the corresponding full set of Echo views. The Student had to learn to predict the Teacher's understanding of the heart's shape.

Over time, the Student (the ECG AI) learned to "hallucinate" the heart's structure just by listening to the electricity.

The Results: Why It's a Big Deal

The paper tested this new "Student" on two main tasks:

1. Diagnosis: Can it tell if the heart's pump is weak (Low Ejection Fraction) or if there is structural disease?
2. Retrieval: If you give the AI an ECG, can it find the matching Echo video from a database?

The findings were impressive:

• Better than the Giants: Even though their model is tiny (18 times smaller than the biggest competing models), it beat all the other "giants" at diagnosing heart structure.
• Data Efficient: It learned so well that it could diagnose heart issues using just 0.1% of the training data that other models needed. It's like a student who can pass the final exam after reading just one page of the textbook, while others need to read the whole library.
• The "18x Smaller" Advantage: Because the model is so small, it can run on cheap laptops or even mobile phones, making advanced heart screening accessible to remote villages where big computers don't exist.

The Bottom Line

Echo2ECG is a breakthrough because it finally taught the cheap, simple heart monitor (ECG) to understand the complex 3D shape of the heart by listening to the "full story" of the ultrasound, not just a single snapshot.

This means that in the future, a simple, cheap, 10-second ECG test could potentially tell you not just if your heart rhythm is off, but also if your heart muscle is weak or your valves are damaged—without you ever needing to go to a hospital for an expensive ultrasound. It's a giant leap toward making high-quality heart care available to everyone, everywhere.

Technical Summary

1. Problem Statement

Electrocardiography (ECG) is a ubiquitous, low-cost tool for diagnosing electrical heart abnormalities (e.g., atrial fibrillation) but lacks the ability to directly quantify cardiac morphological phenotypes (structural features) such as Left Ventricular Ejection Fraction (LVEF). Traditionally, these measurements require Echocardiography (Echo), which is often limited by cost, expertise, and accessibility.

Existing AI approaches attempting to predict morphology from ECG face two main limitations:

1. Representational Mismatch: Current self-supervised methods (e.g., EchoingECG) align ECGs with single-view Echo studies (e.g., Apical 4-Chamber). However, a single Echo view captures only a local anatomical snapshot, whereas an ECG reflects global electrical activity. This creates a misalignment between the global signal and partial structural data.
2. Dependency on Text: Many multimodal models rely on paired text reports (discharge summaries) to bridge modalities, limiting scalability to datasets where clinical notes are available.

2. Methodology: Echo2ECG

The authors propose Echo2ECG, a multimodal self-supervised learning framework designed to distill comprehensive cardiac morphological information from multi-view Echo studies into ECG representations.

Core Architecture

The framework consists of four main components:

1. ECG Encoder: A 12-layer tiny Transformer initialized from OTiS (a time-series foundation model pre-trained on MIMIC-IV-ECG). It maps 12-lead ECG signals to a mean token embedding.
2. Echo Encoder: A MViTv2 (Multiview Transformer) initialized from EchoPrime. It processes individual Echo views (up to 128 per study) and outputs [CLS] token embeddings. This encoder is frozen during training.
3. Echo View Aggregation Module: A critical innovation where view-level embeddings from a single study are projected and aggregated via attention pooling into a single study-level embedding. This allows the model to weigh the relevance of different views (e.g., A4C vs. PSAX) to form a complete anatomical picture.
4. Projection Layers: Modality-specific layers that project ECG and Echo embeddings into a shared 512-dimensional latent space.

Training Objective

The model utilizes Contrastive Learning (inspired by CLIP) to align the global ECG embedding with the aggregated multi-view Echo study embedding.

• Loss Function: A bidirectional contrastive loss ($ECG \to Echo$ and $Echo \to ECG$) pulls matched pairs together and pushes unmatched pairs apart within a batch.
• Optimization: Only the ECG encoder, the Echo view aggregator, and projection layers are trainable. The framework has 12.5M trainable parameters and is trained for 50 epochs.
• Data: Pre-trained on 11,859 paired ECG-Echo studies (acquired within 7 days of each other) from a private institution.

3. Key Contributions

1. Multi-View Alignment: Unlike prior work that aligns ECGs to single Echo views, Echo2ECG aligns ECGs to complete multi-view Echo studies, resolving the representational mismatch between global electrical signals and partial anatomical snapshots.
2. Text-Free Self-Supervision: The framework eliminates the need for paired text reports, relying strictly on the ECG-Echo relationship, thereby increasing scalability.
3. Lightweight Efficiency: Despite being 18× smaller than the largest baseline (EchoingECG at 126.6M params), Echo2ECG (7.1M params) achieves superior performance.
4. Robust Feature Extraction: The resulting ECG encoder serves as a powerful, out-of-the-box feature extractor for downstream tasks requiring morphological data.

4. Experimental Results

The model was evaluated on two clinically relevant tasks across three datasets (Internal, EchoNext, UK Biobank):

A. Structural Cardiac Phenotype Classification

• LVEF Classification: Echo2ECG achieved the highest AUROC on the Internal dataset (0.785) and EchoNext (0.723), outperforming the second-best models by 5.2% and 2.7% respectively. On the UK Biobank, it ranked second only to PTACL (which uses Cardiac MRI, a superior modality to Echo).
• Structural Heart Disease (SHD) Classification:
  • Low-Data Regime: A k-Nearest Neighbors (kNN) classifier trained on just 1% of the data using Echo2ECG features outperformed most baselines trained on 100% of the data.
  • Extreme Low-Data: At 0.1% training data, Echo2ECG outperformed all competing models (including EchoingECG) trained on the same amount of data.

B. Cross-Modal Retrieval

The model was tested on retrieving Echo studies with similar morphological characteristics (LVEDV, LVESV, SV, EF) using ECG queries.

• Echo2ECG (Multi-view) achieved the highest Precision@1 (0.387 for LVEDV) and lowest Mean Rank (MnR).
• It significantly outperformed single-view alignment and the EchoingECG baseline (which performed worse than random retrieval in some metrics), confirming that multi-view alignment is essential for encoding structural information.

C. Ablation Studies

• Multi-view vs. Single-view: Multi-view alignment consistently outperformed single-view alignment.
• Pooling Strategy: Attention pooling for aggregating Echo views was more effective than mean pooling or using the [CLS] token alone, proving that learnable aggregation of multi-view data is crucial.

5. Significance and Conclusion

Echo2ECG demonstrates that morphological information can be effectively distilled from comprehensive imaging (multi-view Echo) into a lightweight ECG representation without relying on text data.

• Clinical Impact: It enables the use of low-cost, widely available ECGs for early screening of structural heart diseases (like reduced LVEF), potentially reducing the need for immediate, expensive Echo scans in initial screenings.
• Technical Impact: It establishes a new paradigm for multimodal medical learning by addressing the "global vs. local" representational mismatch through multi-view aggregation, setting a new state-of-the-art for ECG feature extraction with significantly fewer parameters than existing large models.

Limitations: The study does not account for beat-level or phase-level synchronization between ECG and Echo (temporal alignment), and it still requires paired ECG-Echo studies for pre-training, which may be scarce at massive scales.