🧬 biology

CardioJEPA: A Framework for Holistic Cardiac Representations Through Predictive Multimodal Alignment with ECG-Preservation

CardioJEPA is a novel predictive alignment framework that leverages frozen cardiac foundation models to effectively integrate structural insights from cardiac MRI into ECG representations while preserving electrophysiological information, thereby outperforming existing contrastive methods across diverse diagnostic and structural tasks.

Original authors: Daniel Uyterlinde, Laura Alvarez-Florez, Samuel Ruipérez-Campillo, Lukas P. A. Arts, Folkert W. Asselbergs, Erik J. Bekkers, Ivana Išgum, Fleur V. Y. Tjong

Published 2026-11-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Uyterlinde, Laura Alvarez-Florez, Samuel Ruipérez-Campillo, Lukas P. A. Arts, Folkert W. Asselbergs, Erik J. Bekkers, Ivana Išgum, Fleur V. Y. Tjong

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

The heart is a complex machine that speaks two distinct languages. One is electrical, a rapid series of signals that tell the muscle when to squeeze; the other is structural, the physical shape and size of the chambers that pump blood. For over a century, doctors have relied on the electrocardiogram, or ECG, to listen to the electrical language. It is a cheap, quick, and ubiquitous tool that can reveal rhythm problems and conduction delays. However, the ECG only offers an indirect view of the heart's physical structure. To see the heart's shape, volume, and motion in detail, physicians need cardiac magnetic resonance imaging, or CMR. This technology provides a rich, three-dimensional picture of the organ's anatomy and function, but it is expensive, requires specialized equipment, and is not available to every patient. The central challenge in modern cardiology is whether the detailed structural knowledge found in an MRI scan can be translated into the electrical signals of an ECG, allowing doctors to infer the heart's physical condition from a simple, widely available test.

A team of researchers has developed a new method called CardioJEPA to bridge this gap. Their work focuses on teaching a computer model to understand the relationship between the electrical signals of an ECG and the detailed images of a CMR scan. The goal is not to replace the MRI, but to create a smarter ECG analysis tool that carries the structural insights of an MRI within its own data. By aligning these two different types of medical data, the researchers aim to build a representation of the heart that captures both its electrical activity and its physical form, all derived from the ECG alone. This approach relies on the idea that the heart's structure influences its electrical activity, meaning that some of the information hidden in an MRI scan is already present, though perhaps faintly, in the ECG signal.

The researchers began with a significant hurdle: the two types of data they wanted to connect are fundamentally different in nature. An ECG records a changing electrical signal over time, while a CMR scan captures a series of images showing the heart's anatomy and its movement. Previous attempts to link these two modalities often used a method that treated the data like a game of "find the match." In those approaches, the computer was trained to push apart representations of different patients while pulling together those of the same patient. While this worked to some extent, the researchers found that this method had a hidden cost. By forcing the computer to focus only on what the ECG and the MRI had in common, the process tended to strip away the unique electrical details that make an ECG valuable in the first place. It was as if, in trying to make the two languages speak the same words, the system forgot the unique grammar of the electrical signal.

To solve this, the team designed a new framework that avoids forcing the two data types to become identical. Instead of using a matching game, they built a predictive system. Imagine a student who has studied a textbook and is then asked to predict the next chapter based on a summary. In this system, the computer looks at the ECG signal and tries to predict what the corresponding MRI image would look like, without ever seeing the actual image during the prediction phase. Crucially, the researchers kept the "teacher" model—the one that understands the MRI images—completely frozen and unchanged. This ensured that the system was learning to map the electrical signal onto a stable, high-quality understanding of heart structure, rather than trying to change the understanding of the structure itself.

A key innovation in this work was how they prepared the MRI data for the system to learn from. The researchers realized that a standard MRI scan contains two main types of information: the static shape of the heart and the dynamic motion of it beating. If these two are mixed together, the static shape tends to dominate, drowning out the subtle details of the motion. To fix this, they separated the two components. They took the information about the heart's shape and the information about its movement, processed them separately, and then combined them into a single, balanced target. This allowed the system to learn about both the heart's structure and its function with equal clarity, ensuring that the ECG model could learn to recognize not just the size of the heart, but also how well it was pumping.

Another critical step was ensuring that the electrical information unique to the ECG was not lost during this learning process. The researchers added a specific safeguard that kept the new model anchored to its original, pre-trained knowledge of electrical signals. This prevented the system from discarding vital details about heart rhythm and conduction in its rush to learn about heart structure. The result was a model that successfully integrated structural knowledge without sacrificing the electrical precision that makes the ECG a powerful diagnostic tool.

The team tested their new framework on a massive dataset of over 63,000 paired ECG and MRI recordings from the UK Biobank. The results showed that their approach outperformed all previous methods for linking these two types of data. The new model was better at predicting structural features like the volume of the heart's chambers and the thickness of its walls. More importantly, it retained nearly all of the original electrical information, preserving the ability to detect rhythm abnormalities and conduction delays. When tested on external groups of patients with different heart conditions, including those with Chagas cardiomyopathy, a parasitic disease that affects the heart, the model performed with high accuracy without needing any specific training for those diseases. This suggests that the model had learned a broad, general understanding of heart health that could be applied to new situations.

The study demonstrates that it is possible to enrich a simple, widely available test with the deep structural insights of advanced imaging without losing the unique strengths of the original tool. By using a predictive approach that respects the distinct nature of both electrical and structural data, the researchers have created a more holistic view of the heart. This work suggests that in the future, a standard ECG could provide doctors with a much clearer picture of the heart's physical condition, potentially allowing for earlier detection of structural heart disease in settings where MRI scans are not available. The findings indicate that the electrical signals of the heart contain a wealth of information about its structure, and with the right tools, that information can be unlocked to improve patient care.

Technical Summary of CardioJEPA: A Framework for Holistic Cardiac Representations

Problem Statement
The electrocardiogram (ECG) is the primary cardiac diagnostic tool but offers only an indirect reflection of cardiac structure and function. Conversely, Cardiac Magnetic Resonance (CMR) imaging provides detailed anatomical and functional data but is costly and less accessible. While large-scale self-supervised pretraining has produced strong unimodal foundation models for both ECG and CMR, aligning these modalities to transfer imaging-derived structural knowledge into the ECG latent space remains challenging.

Existing approaches predominantly rely on contrastive learning (e.g., InfoNCE objectives) to align ECG and CMR representations. However, this formulation presents two critical limitations:

Instance Discrimination: Contrastive methods treat different patients as negative samples, potentially separating physiologically similar phenotypes.
Information Loss: Maximizing mutual information bounds drives the ECG representation toward a "minimal sufficient statistic" of the CMR. This process risks suppressing clinically relevant electrophysiological information (e.g., rhythm, conduction delays) that is unique to the ECG and has no direct counterpart in CMR, thereby degrading the ECG's independent diagnostic value.

Methodology: CardioJEPA Framework
The authors propose CardioJEPA, a predictive alignment framework built upon frozen cardiac foundation models. The framework addresses the alignment challenge through three methodological components:

Pretrained Encoders:
- ECG Encoder: Initialized from ECG-FM, a CNN-Transformer hybrid pretrained on 1.5 million 12-lead ECGs.
- CMR Target Encoder: Initialized from CineMA, a multi-view masked autoencoder trained on UK Biobank CMR data. This encoder remains frozen during alignment to prevent representation collapse.
Isolated-PCA CMR Target Construction:
- Standard CMR embeddings (end-diastolic and end-systolic) are nearly identical (cosine similarity > 0.996), causing the alignment loss to be dominated by static anatomy and providing a weak signal for cardiac motion.
- To address this, the authors construct an isolated-PCA target. They apply Principal Component Analysis (PCA) independently to the static anatomy (end-diastolic embedding) and the motion vector (difference between end-systolic and end-diastolic embeddings).
- These two streams are z-scored and concatenated into a 512-dimensional vector. This separates static anatomical variance from functional motion, ensuring the target contains balanced information for transfer.
Predictive Alignment and Preservation:
- Predictive Objective: Instead of contrastive instance discrimination, CardioJEPA uses a Joint-Embedding Predictive Architecture (JEPA) approach. A linear predictor maps the ECG representation to the frozen CMR target using a mean-squared error (MSE) loss ( $L_{JEPA}$ ). This avoids repulsive forces between similar patients.
- Preservation Term: To prevent the suppression of ECG-unique information ( $U_{ecg}$ ), a distillation loss ( $L_{distill}$ ) is introduced. This term anchors the trainable ECG representation to its initial, pretrained state using a cosine similarity penalty. This ensures the encoder retains the electrophysiological information encoded by the foundation model while learning structural features.

The total loss function is defined as:
$L_{total} = L_{JEPA} + \alpha L_{distill}$

Key Results
The framework was evaluated on 63,193 paired ECG–CMR recordings from the UK Biobank and tested on external cohorts.

Structural and Electrical Performance: CardioJEPA outperformed previous ECG–CMR alignment methods (ECCL, MMCL, PTACL) across structural (e.g., LVEF, LVM) and electrical (e.g., PR interval, QRS duration) tasks. Notably, the unaligned ECG foundation model already surpassed prior alignment methods on structural tasks, but CardioJEPA further improved structural performance ( $\tilde{S} = 0.637$ ) while retaining nearly all pre-alignment electrical information ( $\tilde{U}_{ecg} = 0.699$ vs. baseline $0.709$).
Preservation of Unique Information: Without explicit preservation, predictive alignment retained ~2.8% more ECG-unique information than contrastive learning. With the distillation term, the cost of alignment was reduced to a negligible 1.0% deviation from the unaligned baseline.
Latent Space Geometry: Predictive alignment resulted in a significantly smaller modality gap (0.190) compared to contrastive alignment (0.454), creating a more coherent, overlapping latent manifold without the "cone" compression typical of contrastive methods.
Out-of-Distribution Generalization:
- Structural Heart Disease: On the EchoNext dataset, the frozen CardioJEPA representation (with only a linear probe) outperformed end-to-end trained state-of-the-art models (EchoNext-mini and full EchoNext) in detecting structural heart disease (AUROC 0.850 vs. 0.820).
- Chagas Cardiomyopathy: In the 2025 PhysioNet Challenge setting, CardioJEPA improved upon the frozen backbone of the challenge-winning system (AUROC 0.859 vs. 0.838) and an end-to-end CNN (AUROC 0.852 vs. 0.80), despite never seeing Chagas data during training.

Significance and Claims
The paper claims that CardioJEPA demonstrates that incorporating structural information from CMR does not require sacrificing ECG-specific electrophysiological information.

Theoretical Contribution: The work validates that predictive latent alignment, combined with explicit preservation, avoids the "minimal sufficient statistic" collapse inherent in contrastive learning, allowing the ECG representation to mirror the geometry of the CMR target without discarding modality-unique data.
Practical Impact: The resulting representations are "holistic," capturing both imaging-derived structural knowledge and intrinsic electrical characteristics. This enables the inference of cardiac characteristics that typically require imaging (e.g., structural heart disease, Chagas cardiomyopathy) from ECG alone, even in external cohorts and disease populations where the model was not specifically trained.
Modality Gap: The framework substantially reduces the modality gap between ECG and CMR, suggesting that ECG models enriched with imaging-derived structure can extend cardiac assessment capabilities in settings where CMR is unavailable.

The authors acknowledge limitations, including the reliance on linear probing (which may underestimate nonlinear capacity) and the restriction to cine CMR, noting that future work could explore richer temporal representations and tissue-characteristic sequences.

Technical Summary of CardioJEPA: A Framework for Holistic Cardiac Representations

More like this