CARDIAC-FM: A Multimodal Foundation Model for Cardiovascular Risk Prediction Using ECG and Cardiac MRI

CARDIAC-FM is a multimodal foundation model trained on paired ECG and cardiac MRI data from the UK Biobank that achieves superior, generalizable cardiovascular risk prediction across diverse cohorts by learning joint representations, while remaining deployable in clinical settings using only ECG and standard risk scores.

The Gist

Imagine your heart is like a complex, high-performance engine in a car. For decades, doctors have tried to predict when this engine might break down (leading to heart failure or irregular rhythms like atrial fibrillation) by looking at the dashboard gauges (traditional risk factors like age, blood pressure, and cholesterol) or listening to the engine's hum (an ECG).

But sometimes, the dashboard looks fine, and the hum sounds okay, yet the engine is actually developing a hidden crack inside. To see that crack, you usually need to open the hood and take a high-resolution photo of the engine's insides (an MRI). The problem? MRIs are expensive, slow, and not available everywhere.

Enter CARDIAC-FM: The "Super-Listener" for Your Heart.

This new research introduces a smart computer system called CARDIAC-FM. Think of it as a brilliant mechanic who has spent years studying thousands of cars. Here's how it works, broken down into simple concepts:

1. The "Dual-Training" Gym (Multimodal Pre-training)

Most AI models are like students who only study one textbook. They might be great at reading the dashboard (ECG) but terrible at understanding the engine's internal structure (MRI).

CARDIAC-FM is different. It went to a special "dual-training gym" using data from 57,609 people in the UK Biobank.

• The Workout: The AI was shown pairs of data: an ECG (the sound) and an MRI (the picture) from the same person at the same time.
• The Lesson: It learned to connect the dots. It figured out, "Ah, when the ECG hums this specific way, it usually means the heart muscle is thickening that specific way on the MRI."
• The Result: The AI learned a "universal language" of heart health. It learned that the ECG contains hidden clues about the heart's structure that even the best human doctors might miss.

2. The "Magic Translator" (Inference without MRI)

Here is the coolest part. Once the AI finished its training, you don't actually need the MRI anymore to get a great prediction.

Imagine you have a translator who learned a language by reading books and listening to movies. Once they are fluent, they can understand the movie just by reading the script.

• How it works: CARDIAC-FM can look at a simple, cheap, 10-second ECG (which almost every doctor's office has) and say, "Based on what I learned from the MRIs, this ECG pattern suggests the heart has early signs of strain, even though we can't see the MRI right now."
• The Benefit: It effectively "imagines" the MRI details from the ECG alone, giving you a high-tech diagnosis without the high-tech cost.

3. The "Team Player" (Adding Risk Scores)

The researchers also tested if this AI works better when it teams up with traditional risk calculators (like the ones doctors use that ask about your age, weight, and smoking habits).

• The Analogy: Think of the traditional risk score as a weather forecast based on historical data (it's good, but general). Think of the ECG as a live radar showing the storm right now.
• The Outcome: When CARDIAC-FM combined the "live radar" (ECG) with the "historical forecast" (risk scores), it became incredibly accurate. They complemented each other, catching risks that either one would miss alone.

4. The "Chameleon" (Generalizing to New Groups)

Usually, if you train a robot to drive in New York City, it crashes in London because the streets are different.

• The Test: The researchers took CARDIAC-FM, which was trained on UK data, and tested it on two completely different groups of Americans (older adults in the "Cardiovascular Health Study" and a diverse mix of ethnicities in the "MESA study").
• The Result: It didn't crash! It performed just as well. This proves the AI learned the fundamental rules of heart disease, not just the specific habits of the UK population. It can adapt to new people with very little extra training.

Why Does This Matter?

• Early Warning: It can spot heart trouble earlier than current methods, potentially preventing heart attacks or strokes before they happen.
• Accessibility: Since it works with just an ECG, it can be used in rural clinics, developing countries, or at home, without needing a million-dollar MRI machine.
• Versatility: It doesn't just predict heart failure; it can predict strokes, heart attacks, and even death, making it a "Swiss Army Knife" for heart health.

In a nutshell: CARDIAC-FM is a super-smart AI that learned to "see" the inside of your heart just by listening to its electrical heartbeat. It combines the best of old-school risk checks with modern AI to give us a clearer, more accurate picture of our heart's future health, making high-level heart care available to everyone, everywhere.

Technical Summary

1. Problem Statement

Cardiovascular diseases, specifically atrial fibrillation (AF) and heart failure (HF), impose a massive global health burden. Current risk prediction models face several critical limitations:

• Limited Generalizability: Most models are trained on highly selected clinical cohorts and fail to perform well across diverse populations.
• Single-Modality Reliance: Existing AI models typically rely on a single data source (e.g., ECG or MRI), missing the synergistic information available when combining electrical and structural data.
• Task-Specific Training: Conventional deep learning approaches require training separate models for every specific outcome, which is inefficient and requires large labeled datasets for each new disease endpoint.
• Data Scarcity: In real-world settings and external cohorts, positive samples for specific rare outcomes are often limited, making standard supervised learning difficult.

2. Methodology

The authors developed CARDIAC-FM (Contrastive Alignment for Risk Detection In Cardiovascular Foundation Model), a multimodal foundation model trained on paired 12-lead ECG and Cardiac MRI data.

Data Sources

• Training: UK Biobank (57,609 paired, high-quality ECG-MRI samples).
• External Validation: Cardiovascular Health Study (CHS) and Multi-Ethnic Study of Atherosclerosis (MESA).
• Outcomes: Primary endpoints were incident AF and HF (5-year prediction). Secondary endpoints included myocardial infarction, ischemic stroke, cardioembolic stroke, cardiovascular death, and all-cause death.

Model Architecture & Training Framework

The framework operates in two stages:

1. Multimodal Pre-training (Self-Supervised):

   • Encoders:
     • ECG Encoder: Based on the ECG Foundation Model (ECG-FM), utilizing a CNN feature extractor and a BERT-Large transformer.
     • MRI Encoder: A CNN-LSTM architecture processing short-axis (SAX), four-chamber (4CH), and two-chamber (2CH) views (25 frames each).
   • Alignment: The model uses symmetric InfoNCE contrastive loss to align the latent representations of ECG and MRI from the same patient. This forces the ECG encoder to learn structural and functional features typically captured only by MRI.
   • Objective: Learn a shared latent space where ECG and MRI representations are mutually predictive.

2. Downstream Fine-Tuning (Supervised):

   • A lightweight prediction head is added to the pre-trained encoders.
   • The model is fine-tuned for specific clinical outcomes (e.g., AF, HF) using supervised learning.
   • Input Flexibility: The model supports four configurations:
     1. ECG only
     2. ECG + MRI
     3. ECG + Clinical Risk Score (e.g., CHARGE-AF, PREVENT-HF)
     4. ECG + MRI + Clinical Risk Score

Key Technical Innovations

• Cross-Modal Transfer: The ECG encoder learns to "impute" MRI-derived structural features (like Left Ventricular Mass or Ejection Fraction) through contrastive alignment, allowing the model to predict structural risks using ECG alone.
• Few-Shot Adaptation: The model is designed to be fine-tuned on external cohorts with minimal labeled data (e.g., using only 20% of the external cohort data), demonstrating robustness in data-scarce environments.
• Late Fusion: Clinical risk scores are integrated via late fusion (logistic regression) to combine traditional risk factors with deep learning representations.

3. Key Contributions

• First ECG-MRI Foundation Model: CARDIAC-FM is the first foundation model to integrate ECG and Cardiac MRI using contrastive learning for prospective cardiovascular risk prediction.
• Superior Generalizability: The model was validated across three independent cohorts (UK Biobank, CHS, MESA) with diverse demographic characteristics, demonstrating consistent performance without retraining from scratch.
• ECG-Only Inference with MRI-Level Insight: A major contribution is the ability to use the model with ECG input only while retaining the predictive power gained from MRI pre-training. This allows for broad clinical deployment where MRI is unavailable.
• Cross-Modality Translation: The model successfully infers MRI-derived structural parameters (e.g., LVEF, LVM) from ECG signals, enabling the study of cardiac structure-disease relationships in cohorts lacking imaging data (e.g., CHS).

4. Results

Performance in UK Biobank (Internal Validation)

• Atrial Fibrillation (AF): CARDIAC-FM (ECG + Risk Score) achieved an AUROC of 0.82, outperforming the clinical risk score alone (CHARGE-AF) and the unimodal ECG-FM. Adding MRI provided minimal additional gain for AF, suggesting ECG captures the electrical instability of AF effectively.
• Heart Failure (HF): CARDIAC-FM (ECG + Risk Score) achieved an AUROC of 0.81. Adding MRI improved performance further (AUROC 0.83), indicating that structural information from MRI is crucial for HF prediction.
• Feature Prediction: ECG-predicted MRI features (e.g., LVM, LVEDV) showed strong correlations (Pearson $r$ = 0.51–0.79) with actual MRI measurements.

External Validation (CHS and MESA)

• Zero-Shot Performance: When applied directly to CHS and MESA without fine-tuning, CARDIAC-FM outperformed traditional risk scores.
  • In CHS, CARDIAC-FM (ECG + Risk Score) improved AF prediction AUROC by 3 points and HF by 2 points over risk scores alone.
• Few-Shot Fine-Tuning: Fine-tuning on just 20% of external data further improved performance, with CARDIAC-FM achieving AUROCs of 0.75 (AF) and 0.82 (HF) in CHS.
• Broad Outcomes: The model consistently outperformed unimodal ECG-FM across diverse outcomes (MI, Stroke, Death) with minimal fine-tuning, confirming the learned representations capture a comprehensive view of cardiac health.

Risk Stratification

• High-risk tertiles identified by CARDIAC-FM showed significantly higher hazard ratios for incident events compared to low-risk tertiles, often outperforming traditional risk scores.
• The model showed particular strength in predicting Heart Failure with Reduced Ejection Fraction (HFrEF), where structural remodeling is prominent.

5. Significance and Clinical Implications

• Scalability: By leveraging ECG as the primary input (which is ubiquitous and low-cost) while learning from MRI during pre-training, CARDIAC-FM bridges the gap between high-fidelity imaging and routine clinical practice.
• Efficiency: The foundation model approach eliminates the need to train separate models for every new cardiovascular outcome, significantly reducing computational and data requirements.
• Equity and Access: The model's ability to function without MRI makes it suitable for population-scale screening in resource-limited settings or historical cohorts where imaging data is missing.
• Complementary Risk Dimensions: The study demonstrates that ECG signals and traditional risk factors capture complementary aspects of cardiovascular risk; combining them yields additive gains in discrimination.

In conclusion, CARDIAC-FM represents a paradigm shift in cardiovascular AI, moving from task-specific, single-modality models to a generalizable, multimodal foundation model that enhances risk prediction even when only ECG data is available.