💻 computer science

Metadata Supervised Imaging Representations for Modelling and Controlling Acquisition Variability

This paper proposes a metadata-supervised approach to disentangle anatomical structure from acquisition-dependent variations in biomedical imaging, enabling the creation of acquisition-aware representations and a unified harmonisation model that improves generalisation, interpretability, and clinical deployment across diverse imaging protocols and sites.

Original authors: Mehmet Yigit Avci, Pedro Borges, Virginia Fernandez, Natalia Glazman, Paul Wright, Mehmet Yigitsoy, Sebastien Ourselin, Jorge Cardoso

Published 2026-07-30

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Mehmet Yigit Avci, Pedro Borges, Virginia Fernandez, Natalia Glazman, Paul Wright, Mehmet Yigitsoy, Sebastien Ourselin, Jorge Cardoso

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize a cat. If you only show it pictures of fluffy, orange cats taken in bright sunlight, the robot might think "cat" means "orange fur" and "sunlight." When you later show it a sleek, black cat in a dark room, the robot gets confused because the lighting and color are totally different, even though it's still a cat. This is a bit like what happens in medical imaging. Doctors use powerful machines called MRI scanners to take pictures of the inside of the human body, like the brain. But just like your camera, these machines have settings. One scanner might be made by Siemens, another by GE; one might use a specific type of pulse to make water look bright, while another makes it look dark. These settings are the "acquisition" details.

The problem is that these settings change how the picture looks, even if the patient's brain is exactly the same. A healthy brain might look like a dark gray blob on one machine and a bright white one on another. For a computer trying to learn from these images, this is a nightmare. The computer might get so distracted by the "style" of the picture (the lighting, the contrast, the machine type) that it forgets to learn about the actual "structure" (the shape of the brain, the size of the ventricles). This makes it hard to build smart tools that work reliably in every hospital, because every hospital might have slightly different equipment. Scientists call this "acquisition variability," and it's a major hurdle in making medical AI truly helpful for everyone.

The Paper's Big Idea: Separating the "What" from the "How"

This paper introduces a new system called MaRaI (Multimodal Acquisition-aware Radiology AI) that tries to solve this problem by teaching computers to separate the "what" (the actual anatomy of the brain) from the "how" (the specific settings used to take the picture). Think of it like a chef who can take a recipe and separate the list of ingredients (the anatomy) from the cooking style (the contrast). Usually, these two are mixed up in the final dish, but MaRaI learns to pull them apart.

The authors suggest that instead of ignoring the machine settings or trying to force all images to look the same, we should use the machine's own "receipt" (called DICOM metadata) to teach the computer what is happening. Every MRI scan comes with a digital tag that lists exactly how the picture was made: the time the pulse was sent, the strength of the magnetic field, and the type of scanner used. MaRaI treats these tags not as boring notes, but as a secret language that describes the "style" of the image.

How It Works: The Magic Translator

The system has two main tricks up its sleeve, working together like a team of translators and artists:

The Translator (MR-CLIP): First, the system learns to match the picture of a brain with the text description of how it was taken. Imagine showing a picture of a brain to a student and asking them to write a sentence describing the scanner settings. MaRaI does the reverse: it looks at the settings and learns to predict what the image should look like, or looks at the image and guesses the settings. By doing this millions of times, it learns a "dictionary" where specific settings always correspond to specific visual styles. This helps the computer understand that a "T1-weighted" image and a "T2-weighted" image are just different ways of looking at the same brain, not two different brains.
The Artist (DIST-CLIP): Once the computer understands the difference between the brain's shape and the image's style, it can start editing. The system takes a brain scan from one hospital (the "source") and asks, "What would this brain look like if we took the picture using the settings from a different hospital (the "target")?" It freezes the brain's shape (the anatomy) and only changes the colors and textures (the contrast) to match the new settings. It's like taking a black-and-white sketch of a face and coloring it in the style of a Van Gogh painting without changing the shape of the nose or eyes.

What They Found: Smarter, Cleaner, and More Honest

The researchers tested this idea on a huge collection of brain scans from thousands of patients. Here is what they discovered:

The "Shape" is Stable: When they stripped away the style and looked only at the anatomical maps (the "shape" part), the measurements of brain tissues became much more consistent. Whether the scan was taken on a Siemens machine or a GE machine, the volume of the brain's gray matter stayed the same. This is a big deal because it means doctors can compare patients from different hospitals without worrying that the machine is tricking them.
Better Diagnosis Across Borders: They tested if this helped diagnose Alzheimer's disease. They trained a computer to spot Alzheimer's using scans from Siemens machines and then tested it on scans from GE machines. Without their new method, the computer struggled because the images looked so different. But when they used the "shape-only" maps from MaRaI, the computer got much better at spotting the disease across different machines. It suggests that by removing the "noise" of the scanner settings, the computer can focus on the real biological signs of the disease.
One Model to Rule Them All: Usually, to change an image from one style to another, you need a different tool for every pair of scanners. MaRaI is different. Because it understands the "style" as a separate code, you can tell it to change a scan to any other style just by giving it the settings (the metadata) or a reference picture. It works for many different types of brain scans (T1, T2, FLAIR) and different scanners, all with the same set of trained weights.
The "Lie Detector" for Data: One of the most playful and useful findings is that this system can act as a quality control inspector. Because the system knows exactly what a brain scan should look like for a given set of settings, it can spot when the settings are lying. If a scan says it was taken with a specific pulse time, but the image looks like it was taken with a different one, the system gets confused and flags it. In their tests, when they intentionally messed up the metadata (like changing the numbers or deleting them), the system could detect the error with very high accuracy (up to 99.7% for major errors). This means hospitals could automatically find bad data before they even start analyzing it.

Why It Matters

The authors suggest that this approach changes how we think about medical imaging. Instead of treating the differences between scanners as a annoying problem to be fixed or ignored, MaRaI treats them as a structured, understandable part of the process. By using the machine's own metadata to teach the AI, they create a system that is more robust, fair, and reliable. It suggests that we can build medical AI that works in the messy, real world of different hospitals and machines, not just in the perfect, controlled labs where it was trained. While the paper notes that there is still work to be done (like handling motion or disease artifacts separately), the core idea—that we can separate the "what" from the "how" using the machine's own notes—opens a new path for making medical imaging smarter and more trustworthy.

Technical Summary: Metadata Supervised Imaging Representations for Modelling and Controlling Acquisition Variability

Problem Statement

Biomedical imaging, particularly Magnetic Resonance Imaging (MRI), suffers from substantial acquisition variability. Identical biological structures can appear markedly different due to variations in imaging devices, acquisition protocols, sites, and reconstruction settings. This "acquisition shift" causes learned representations to entangle underlying biological information with acquisition-dependent appearance, limiting model interpretability, generalization, and clinical deployment across different scanners and institutions.

Current approaches to this problem typically treat acquisition variability as a nuisance factor to be removed via domain generalization (learning invariant features) or image harmonization (translating appearance between domains). However, these methods often ignore the fact that MRI variability is a structured, physically meaningful consequence of the imaging process. Furthermore, routine clinical imaging generates rich DICOM metadata describing these physical parameters (e.g., echo time, repetition time, scanner vendor), yet this information is often underutilized or treated as unreliable in large-scale archives. There is a need for a framework that explicitly separates anatomical structure from acquisition-dependent contrast while leveraging metadata to model, audit, and control this variability.

Methodology: The MaRaI Framework

The authors introduce MaRaI (Multimodal Acquisition-aware Radiology AI), a framework that treats DICOM metadata not as auxiliary labels but as structured supervision for learning disentangled representations. The framework consists of two primary components sharing a common representation space:

1. MR-CLIP: Metadata-Guided Contrastive Learning

MR-CLIP aligns MRI volumes with natural-language prompts constructed from DICOM acquisition tags using a supervised contrastive objective.

Metadata Processing: Numerical parameters (Echo Time [TE], Repetition Time [TR], Inversion Time [TI]) are discretized into bins, while categorical tags (scanner, sequence type) are grouped. These are formatted into structured text prompts.
Contrastive Objective: The model uses a bidirectional supervised contrastive (SupCon) loss to pull together volumes sharing the same acquisition protocol (contrast-aware label) regardless of anatomy or subject identity, while pushing apart different protocols.
Architectures: The framework supports 2D (slice-based), 2.5D (slice-aggregated), and 3D (volumetric) Vision Transformer (ViT) encoders.
Output: The resulting embeddings capture contrast as a structured variable, enabling cross-modal retrieval, few-shot sequence classification, and quality control.

2. DIST-CLIP: Anatomy-Contrast Factorization and Harmonization

DIST-CLIP utilizes the pretrained MR-CLIP embeddings to factorize each scan into an acquisition-invariant anatomical map ( $\beta$ ) and a contrast representation ( $\theta$ ).

Disentanglement: A U-Net "Anatomy Mapper" extracts $\beta$ from the source image, suppressing contrast-specific intensity patterns while preserving tissue boundaries and global morphology.
Style Fusion Decoder: A new architecture synthesizes harmonized images by fusing the anatomical map $\beta$ $β$ with a target contrast code $\theta$ $θ$ .
- Conditioning: The target contrast can be derived from a reference image ( $\theta_i$ ) or directly from acquisition metadata ( $\theta_m$ ).
- Mechanism: The decoder employs Adaptive Instance Normalization (AdaIN) and cross-attention mechanisms. An MLP maps the contrast embedding to style queries, which modulate decoder features via multi-head attention and channel-wise scaling/shift parameters. This allows for structure-preserving appearance transfer.
Training: The model is trained with a composite loss function including reconstruction ( $L_{rec}$ ), perceptual ( $L_{perc}$ ), adversarial ( $L_{adv}$ ), patch-level contrastive alignment ( $L_\beta$ ), global style consistency ( $L_{global}$ ), and directional consistency ( $L_{dir}$ ) losses.

Key Contributions

Anatomy–Contrast Disentanglement with Clinical Relevance: The authors demonstrate that anatomical maps derived from DIST-CLIP reduce acquisition-driven variability in tissue morphometry across protocols. Crucially, these representations improve cross-vendor Alzheimer's disease (AD) classification on the ADNI dataset compared to raw MRI, establishing that explicit disentanglement yields measurable clinical benefits beyond visual consistency.
Unified Harmonisation Across Contrasts and Sites: The framework introduces a single model capable of high-fidelity harmonization across all major brain MRI contrasts (e.g., T1w, T2w, FLAIR, PD) and scanners. It supports both image-guided and metadata-conditioned modes using a single set of trained weights, enabling synthesis without requiring a target reference scan.
Metadata-Driven Contrast-Aware Representation Learning: By discretizing heterogeneous DICOM parameters into structured supervision signals, the authors enable large-scale supervised contrastive learning. The resulting MR-CLIP embeddings outperform supervised baselines in retrieval and few-shot classification tasks, demonstrating that acquisition metadata can effectively structure large-scale clinical archives.

Results

The paper validates the framework through extensive experiments on clinical brain MRI data (King's College Hospital, Guy's and St Thomas'), ADNI, OASIS-3, and the ON-Harmony travelling-heads dataset.

Suppression of Acquisition Variability: Anatomical representations ( $\beta$ ) showed significantly lower inter-sequence volume coefficient of variation (CV) compared to raw MRI while maintaining similar Dice overlap for tissue segmentation. In cross-vendor AD classification (Siemens to GE), models using $\beta$ -inputs achieved higher balanced accuracy (0.65 vs. 0.60) than those using raw MRI, indicating reduced sensitivity to scanner shift.
Harmonization Performance:
- Pixel Fidelity: DIST-CLIP variants achieved superior Structural Similarity Index (SSIM) and Peak Signal-to-Noise Ratio (PSNR) compared to baselines (HACA3 and TUMSyn) in cross-contrast translation tasks (e.g., T2w $\to$ PDw).
- Anatomical Consistency: On held-out clinical data, DIST-CLIP harmonized outputs achieved significantly higher Dice scores (0.78) for brain parcellation compared to HACA3 (0.70) and TUMSyn (0.38).
- Downstream Utility: In the ON-Harmony dataset, harmonized data reduced site classification accuracy (indicating removal of scanner fingerprints) from 0.79 to 0.42 while preserving brain-age prediction performance (MAE ~8.89 years), whereas baselines either failed to remove scanner signatures or degraded biological signal.
Metadata Utility:
- Retrieval & Classification: MR-CLIP achieved high recall in cross-modal retrieval (e.g., 78.7% scan-to-metadata recall@1) and significantly outperformed end-to-end 3D ResNets in few-shot sequence classification.
- Quality Control: The framework successfully detected synthetic metadata corruption (numerical errors, wrong tags, missing fields) by measuring the cosine similarity between image and metadata embeddings. At 50% corruption rates, the system achieved near-perfect AUC (0.997) for detecting large missing tags and numerical errors.

Significance and Claims

The paper argues that acquisition variability is not merely a source of noise but a structured component of the imaging process that can be modeled, audited, and controlled. By leveraging routine DICOM metadata, MaRaI provides a foundation for:

Robust Clinical AI: Systems that are resilient to the heterogeneity of real-world hospital data.
Controlled Synthesis: The ability to generate target appearances based on protocol specifications without needing a reference scan from that specific protocol.
Data Auditing: Automated detection of metadata-image inconsistencies in large archives, closing a critical loop for data quality assurance in multi-site trials.

The authors position this work as a step toward "acquisition-aware foundation models" that explicitly account for the physics of image formation, moving beyond treating domain shift as an unavoidable limitation. They acknowledge limitations, noting that the framework currently focuses on adult brain MRI and that pathology and motion artifacts are not yet fully disentangled from acquisition factors, suggesting these as natural next steps for future tri-factor decomposition.