⚡ electrical engineering

Global and Local Contrastive Learning for Joint Representations from Cardiac MRI and ECG

This paper introduces PTACL, a multimodal contrastive learning framework that leverages both global patient-level and local temporal-level alignment to integrate structural insights from Cardiac MRI into ECG representations, thereby significantly improving cardiac phenotype retrieval and functional parameter prediction without adding new learnable parameters.

Original authors: Alexander Selivanov, Philip Müller, Özgün Turgut, Nil Stolt-Ansó, Daniel Rückert

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Selivanov, Philip Müller, Özgün Turgut, Nil Stolt-Ansó, Daniel Rückert

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand a complex machine, like a car engine, but you only have two very different tools to look at it. The first tool is a stethoscope that listens to the engine's rhythm and electrical sparks; it's cheap, easy to carry, and tells you if the engine is running smoothly or sputtering. The second tool is a high-tech, expensive 3D scanner that can see the exact shape of the pistons, how much fuel is in the tank, and how hard the engine is pushing. In the world of medicine, the "stethoscope" is the Electrocardiogram (ECG), a common test that tracks the heart's electrical activity. The "3D scanner" is the Cardiac Magnetic Resonance (CMR) scan, which gives doctors a detailed, gold-standard view of the heart's size, shape, and pumping power.

The problem is that the ECG is everywhere, but it's a bit "blind" to the heart's physical structure. It can hear the rhythm but can't tell you exactly how big the heart chambers are or how much blood they pump. The CMR scan sees everything perfectly, but it's too expensive and slow to use on everyone. Scientists have been trying to teach computers to use the cheap ECG to guess the expensive CMR details, a bit like trying to guess the shape of a hidden object just by listening to the sound it makes. They've used a technique called contrastive learning, which is like a game of "match the pair." The computer learns to recognize that a specific ECG rhythm belongs to a specific heart shape by comparing thousands of pairs, pulling matching pairs closer together in its memory and pushing different ones apart. But until now, these computer games were a bit too broad, treating the whole heartbeat as one big, blurry blob.

Enter a new method called PTACL (Patient and Temporal Alignment Contrastive Learning), developed by researchers at the Technical University of Munich and Imperial College London. Think of the old way of learning as trying to match a whole song to a whole movie scene just because they feel similar. PTACL changes the game by matching specific beats of the song to specific frames of the movie. The researchers realized that the heart's electrical signal (ECG) and its physical movement (CMR) happen in a precise, synchronized dance. By teaching the computer to align not just the whole patient's data, but also the tiny, split-second moments within a single heartbeat, they created a much sharper picture.

The team tested this on a massive dataset from the UK Biobank, involving 27,951 people who had both an ECG and a CMR scan. They found that by adding this "fine-grained" time-alignment, the computer became much better at two things. First, it got significantly better at finding patients with similar heart conditions just by looking at their ECGs. Second, and perhaps more impressively, the computer could predict specific heart measurements—like the volume of the heart's chambers and its pumping efficiency (ejection fraction)—directly from the ECG with much higher accuracy than before.

What makes this discovery particularly clever is that it didn't require the computer to learn any new, complicated rules or add extra "brain power" (learnable parameters). Instead, it simply changed how the computer compared the data, forcing it to pay attention to the timing. The results showed that this approach consistently outperformed previous methods, suggesting that we can get a much deeper understanding of heart health from a simple, cheap ECG test if we just teach the AI to listen to the rhythm with a bit more precision. While the method currently relies on having paired data to learn from, it opens a promising door for making advanced heart diagnostics more accessible to everyone.

Technical Summary: PTACL for Joint Representations from Cardiac MRI and ECG

Problem Statement
Cardiovascular diseases remain a leading cause of mortality globally. While electrocardiograms (ECGs) are a widely used, cost-effective, and non-invasive tool for detecting electrical abnormalities (e.g., arrhythmias), they lack the capacity to directly measure critical structural and functional cardiac parameters, such as ventricular volumes, myocardial mass, and ejection fraction. Conversely, Cardiac Magnetic Resonance (CMR) imaging is the gold standard for these assessments but is expensive, time-consuming, and less accessible. Existing multimodal approaches attempt to bridge this gap using contrastive learning to align ECG and CMR embeddings globally. However, these methods often treat each modality as a holistic representation, overlooking fine-grained temporal relationships. Since CMR captures dynamic sequences across the cardiac cycle while ECG records continuous electrical activity, global alignment alone lacks the temporal precision required to effectively integrate both modalities.

Methodology: PTACL
The authors propose PTACL (Patient and Temporal Alignment Contrastive Learning), a multimodal contrastive learning framework designed to enhance ECG representations by integrating spatio-temporal information from CMR. The framework operates in three phases:

Single Modality Pre-training (SMP): Separate ECG and CMR encoders are pre-trained independently on unlabeled data using the Masked Autoencoder (MAE) framework.
- ECG: Multi-lead signals are embedded via 1D convolution into time-based tokens, with positional embeddings added before random masking.
- CMR: 2D+T short-axis sequences are processed via 3D convolution to generate spatio-temporal tokens, followed by masking and reconstruction.
Multimodal Contrastive Pre-training: The encoders are jointly trained using a combination of two contrastive losses:
- Global Contrastive Loss: Aligns patient-level representations. It pulls ECG and CMR embeddings from the same patient closer together while pushing apart embeddings from different patients, utilizing the InfoNCE loss.
- Local Contrastive Loss: Enforces fine-grained temporal alignment. This loss contrasts encoded ECG segments (derived from a single heartbeat representation) with corresponding encoded CMR frames at the same cardiac phase.
  - Mechanism: The local loss utilizes a temporal alignment matrix ( $P_{k,t}$ ) that can be either hard ( $\sigma=0$ , exact matching) or soft ( $\sigma>0$ , Gaussian weighting based on temporal distance).
  - Parameter Efficiency: Notably, this local alignment is completely parameter-free, introducing no additional learnable weights to the model.
Inference: The trained ECG encoder is evaluated on downstream tasks without requiring the CMR encoder.

Key Contributions

Novel Framework: Introduction of PTACL, which combines global patient-level alignment with local temporal-level alignment to create richer ECG representations.
Temporal Precision: Unlike prior global-only methods, PTACL explicitly models the temporal correspondence between ECG segments and CMR frames, leveraging the physiological alignment provided by ECG-gated CMR acquisition.
Parameter Efficiency: The method achieves improved performance without introducing new learnable parameters for the local alignment component.
Self-Supervised Approach: The framework is fully self-supervised, trained on paired data from 27,951 subjects in the UK Biobank.

Experimental Results
The authors evaluated PTACL on two clinically relevant tasks using UK Biobank data (27,951 training subjects, 5,991 test subjects):

Patient Retrieval: The task involved retrieving patients with similar CMR-derived phenotypes based on ECG embeddings.
- PTACL (Global + Local) consistently outperformed the baseline Global model across all tested phenotypes (e.g., LV/RV volumes, ejection fraction, mass).
- Improvements were observed in Precision@k (P@k), Mean Rank (MnR), and Median Rank (MdR). For instance, for Left Ventricular End-Diastolic Volume (EDVLV), P@1 improved from 0.492 (Global) to 0.510 (PTACL).
Phenotype Regression: The task involved predicting CMR-derived cardiac function parameters from ECG embeddings using $R^2$ $R^{2}$ scores.
- PTACL achieved superior regression performance compared to single-modality baselines (ECG-only or CMR-only) and previous multimodal approaches (e.g., ECCL, CMAE, MMCL).
- For example, in predicting Left Ventricular Ejection Fraction (LVEF), PTACL with full fine-tuning achieved an $R^2$ of 0.283, outperforming the Global baseline (0.258) and other multimodal methods.
- The results indicate that local contrastive alignment transfers more insights between modalities than global alignment alone.

Significance and Claims
The paper claims that PTACL successfully enhances non-invasive cardiac diagnostics by enriching ECG representations with diagnostic information beyond electrical activity. By capturing fine-grained temporal correspondences, the method improves the ability to assess cardiac structure and function using only ECG data. The authors highlight that this is achieved without the computational overhead of additional learnable parameters.

The study acknowledges limitations, specifically the dependence on paired ECG-CMR data, which may restrict applicability where such paired datasets are unavailable. The authors suggest future work could explore extending the framework to additional modalities (e.g., ultrasound) or developing strategies to mitigate reliance on paired multimodal datasets. The code is made available to the community to facilitate further research.

Technical Summary: PTACL for Joint Representations from Cardiac MRI and ECG

More like this