⚡ electrical engineering

Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention

This paper introduces a novel multimodal latent diffusion model that utilizes cross-attention and variational autoencoders to jointly synthesize coherent MRI volumes and tabular clinical data within a shared latent space, demonstrating high-fidelity generation capabilities on the German National Cohort dataset and establishing a proof-of-concept for creating synthetic digital twins in healthcare.

Original authors: Daniel Mensing, Jan Kapar, Jochen G. Hirsch, Matthias Günther, Horst Hahn, Marvin N. Wright

Published 2026-05-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daniel Mensing, Jan Kapar, Jochen G. Hirsch, Matthias Günther, Horst Hahn, Marvin N. Wright

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two different types of puzzle pieces that usually live in separate boxes. One box contains 3D MRI scans (detailed, colorful pictures of the inside of a body), and the other box contains tabular data (a spreadsheet with facts like age, height, weight, and gender).

Usually, if you want to create a fake patient, you might make up the spreadsheet first and then try to guess what their MRI looks like, or vice versa. But this paper introduces a new way to do it: a single "master chef" that cooks both the picture and the facts at the same time.

Here is how the authors cooked up this new method, using simple analogies:

1. The "Translation Room" (The VAE)

First, the model needs to understand both the MRI pictures and the spreadsheet numbers. They are very different languages.

The Analogy: Imagine a translator who takes a complex novel (the MRI) and a list of bullet points (the spreadsheet) and translates both into a secret, compressed code that fits into a single, small suitcase.
What they did: They used a tool called a Variational Autoencoder (VAE). This tool squishes the big MRI images and the long list of numbers into a shared "latent space" (that secret code). Crucially, they used a technique called Cross-Attention. Think of this as the translator constantly checking the suitcase: "Oh, the code says 'tall male,' so the MRI code better show a tall male skeleton." This ensures the picture and the facts stay perfectly in sync.

2. The "Denoising Sculptor" (The Diffusion Model)

Once everything is in that shared suitcase, the real magic happens.

The Analogy: Imagine a sculptor who starts with a block of marble covered in static noise (like TV snow). Instead of carving from scratch, the sculptor slowly removes the noise, step-by-step, revealing a perfect statue.
What they did: They used a Diffusion Model. This model learns how to take that noisy, scrambled code in the suitcase and slowly "clean" it until it turns into a realistic patient profile. Because the MRI and the spreadsheet were fused together in the suitcase, the model generates them simultaneously. When the noise clears, you get a brand new MRI scan and a matching spreadsheet of facts that belong to that specific person.

3. The "Two-Headed Printer" (The Decoders)

After the sculptor reveals the clean code, it needs to be turned back into something humans can see.

The Analogy: The code goes to a printing press with two different heads. One head is a high-end 3D printer that knows how to print the MRI scan. The other head is a text printer that knows how to print the numbers and categories (like "European" or "Female").
What they did: They used separate "decoders" for each. This ensures the MRI looks anatomically correct (using 3D convolution) and the numbers make sense (using transformers), even though they came from the same source.

4. The Test Drive (The Results)

The team tested this on data from over 10,000 real people from the German National Cohort (NAKO).

The MRI Test: They looked at the fake MRIs and compared them to real ones. The fake ones looked real! They had the right body shapes and anatomy that matched the fake facts (e.g., a fake person listed as having a high BMI actually looked heavier in the MRI).
The Spreadsheet Test: They compared their "master chef" model against other famous AI models that only make spreadsheets (like CTGAN and TVAE).
- The Result: Their model did a better job than the CTGAN model at capturing how different facts relate to each other (e.g., how age relates to body fat). It was slightly less perfect than a model that only focuses on spreadsheets (TabSyn), but considering it had to do two jobs at once (MRI + Spreadsheet), it performed very competitively.

The Bottom Line

The paper claims this is the first time anyone has successfully built a single AI that generates a realistic 3D MRI scan and a matching medical spreadsheet at the exact same time, ensuring they match perfectly.

They aren't claiming this can diagnose patients yet. Instead, they are showing that it's possible to create coherent synthetic patient data. This is like building a "digital twin" of a patient from scratch, where the picture and the paperwork tell the same story, which could help researchers train AI without needing to use real, private patient data.

Technical Summary: Multimodal Synthesis of MRI and Tabular Data with Diffusion in a Joint Latent Space via Cross-Attention

Problem Statement
Medical AI applications increasingly require comprehensive patient profiles that integrate high-dimensional volumetric imaging (such as MRI) with heterogeneous tabular clinical data (demographics, anthropometrics, lab values). While generative models have advanced significantly in both domains independently, creating coherent synthetic pairs of these modalities remains challenging. Deep learning architectures like diffusion models excel at imaging but struggle with mixed-type tabular data, whereas tree-based models perform well on tabular data but lack imaging capabilities. Existing multimodal approaches often rely on sequential, unidirectional pipelines (e.g., generating tabular data first, then conditioning image generation), which suffer from limited bidirectionality, impaired inter-modal dependency modeling, and error propagation. There is a lack of end-to-end architectures capable of jointly synthesizing MRI and tabular data within a unified framework.

Methodology
The authors propose a Multimodal Latent Diffusion Model (LDM) that operates within a shared latent space to fuse volumetric MRI and tabular clinical data. The architecture consists of two primary stages:

Variational Autoencoder (VAE) for Modality Fusion:
- Encoder: The model utilizes a VAE to map high-dimensional data from the image domain ( $D_{img}$ ) and tabular domain ( $D_{tab}$ ) into a lower-dimensional joint latent manifold.
- Tabular Processing: Numerical and categorical features are tokenized and processed via separate learnable embedding schemes (similar to TabSyn), mapping heterogeneous variables into a unified vector space.
- Image Processing: 3D convolutions extract spatial features from MRI volumes.
- Fusion: A cross-attention mechanism integrates the tabular embeddings with the image features within the encoder to produce a joint latent representation.
- Decoders: Distinct decoders are employed for each modality to ensure modality-appropriate reconstruction. Image reconstruction uses 3D convolutions, while tabular reconstruction uses transformers.
- Loss Function: The VAE is trained to prioritize high-fidelity reconstruction over enforcing a strictly Gaussian latent structure. The loss includes L1, perceptual, and adversarial losses for images; Mean Squared Error (MSE) and Cross-Entropy for tabular data; and a Kullback–Leibler (KL) divergence term for regularization.
Diffusion Model for Synthesis:
- Once the VAE converges, a Denoising Diffusion Probabilistic Model (DDPM) is trained on the fused latent space rather than the raw pixel space to reduce computational complexity.
- The model employs a U-Net architecture trained with a linear beta schedule.
- Sampling is accelerated using a Denoising Diffusion Implicit Model (DDIM) trajectory, which allows for deterministic generation with fewer steps.

Experimental Setup
The framework was evaluated using data from the German National Cohort (NAKO), comprising 10,748 participants.

MRI Data: Axial T1-weighted VIBE sequences (opposed-phase contrast) from 3T scanners, resampled to $160 \times 160 \times 128$ voxels.
Tabular Data: Features included age, sex, height, weight, BMI, body fat percentage, and ethnicity.
Split: 90% training, 10% testing.

Key Results

MRI Synthesis: The model generated anatomically plausible MRI volumes consistent with the synthesized tabular attributes (e.g., body composition matching the generated BMI and sex). Quantitative evaluation using the Universal bioMedical PreTrained model (UMedPT) embeddings yielded a Fréchet Distance (FD) of 1.54. Image fidelity was assessed with $\alpha$ -precision (0.871) and $\beta$ -recall (0.183).
Tabular Synthesis: The multimodal LDM was compared against unimodal baselines: TabSyn (state-of-the-art diffusion), CTGAN, and TVAE.
- As expected, the unimodal TabSyn achieved the best scores across all metrics since it does not face the constraint of joint MRI-tabular consistency.
- However, the multimodal LDM demonstrated competitive performance against CTGAN and TVAE, outperforming CTGAN in all metrics except mean univariate fit.
- The model achieved a lower correlation distance (0.035) compared to GAN-based baselines, indicating a superior ability to capture dependencies within the data.
- The $\alpha$ -precision (0.562) substantially exceeded that of CTGAN (0.127) and TVAE (0.260), suggesting better sample fidelity.

Significance and Claims
The authors claim this work is, to their knowledge, the first to demonstrate the feasibility of jointly modeling MRI and mixed-type tabular data in a single latent diffusion framework.

Proof-of-Concept: The study establishes a proof-of-concept for generating coherent synthetic multimodal patient data, moving beyond sequential pipelines to a joint generative strategy.
Digital Twins: This approach aligns with the broader goal of developing high-fidelity digital twins in healthcare by providing a more holistic representation of patient profiles in synthetic cohorts.
Future Directions: The authors note that while results are promising, further refinement is needed to bridge the performance gap with specialized unimodal models, particularly regarding univariate distribution modeling and diversity. Future work aims to integrate unified transformer architectures to enable bidirectional workflows such as cross-modality imputation and modality-conditional generation.