Merlin: A Computed Tomography Vision-Language Foundation Model and Dataset

Merlin is a novel 3D vision-language foundation model trained on a massive dataset of over 15,000 abdominal CT scans and associated clinical data that outperforms existing 2D and specialized models across diverse diagnostic, prognostic, and quality-related tasks while demonstrating strong generalization across multiple institutions.

The Gist

Imagine the human body as a vast, complex city. When a doctor needs to check the "downtown" area (the abdomen), they use a special camera called a CT scanner. This camera doesn't just take a single photo; it takes hundreds of thin slices, like a loaf of bread, to build a complete 3D model of the city's streets, buildings, and pipes.

However, there's a problem: Radiologists (the doctors who read these maps) are overwhelmed. There are too many scans, not enough doctors, and reading a single 3D "loaf of bread" can take 20 minutes of intense focus. They are tired, and sometimes, tiny clues about future diseases get missed.

Enter Merlin.

What is Merlin?

Think of Merlin not as a robot doctor, but as a super-smart, tireless apprentice who has read every single medical textbook and every single city map ever made.

In the world of AI, most models are like students who only study 2D pictures (flat photos). But a CT scan is 3D. Trying to understand a 3D city by looking at flat photos one by one is like trying to understand a skyscraper by looking at individual floor plans without seeing the whole building. It's inefficient and easy to miss the big picture.

Merlin is different. It is a 3D Vision-Language Foundation Model.

• Vision: It can look at the entire 3D loaf of bread at once, seeing how the liver, kidneys, and blood vessels connect in three dimensions.
• Language: It doesn't just see the image; it understands the story the doctor wrote about it. It reads the radiology report (the text) and links it directly to the 3D image.

How was Merlin trained? (The "School" Analogy)

Usually, to teach an AI, you need a human teacher to draw boxes around every tumor or organ and say, "This is a tumor." This is expensive and slow.

Merlin learned differently. The researchers gave Merlin a massive library of paired data:

1. The Image: The 3D CT scan.
2. The Text: The actual report written by a real radiologist describing what they saw.
3. The Records: The patient's medical history (like a list of past illnesses).

Merlin learned by matching the picture to the story. It didn't need a teacher to draw boxes; it learned by reading millions of stories and looking at the pictures they described. It's like learning a language by reading books and looking at the world, rather than memorizing a dictionary.

The "One GPU" Trick:
Usually, training a brain this big requires a supercomputer the size of a warehouse. But the researchers built Merlin so efficiently that it could be trained on a single graphics card (the kind a gamer might have). This means even small hospitals could build their own "Merlin" without needing billions of dollars.

What can Merlin do? (The "Swiss Army Knife" of Radiology)

Once trained, Merlin isn't just good at one thing. It's a multi-tool:

1. The Detective (Zero-Shot Classification): You can ask Merlin, "Is there fluid around the lungs?" even if it never saw that specific question before. It uses its general knowledge to answer instantly.
2. The Time Traveler (Future Prediction): Merlin can look at a scan of a healthy person and predict, "Based on the texture of this tissue, this person is likely to develop diabetes or heart disease in the next 5 years." It finds early warning signs humans might miss.
3. The Scribe (Report Generation): It can look at a scan and draft the initial radiology report for the doctor to review, saving them hours of typing.
4. The Architect (3D Segmentation): It can automatically color-code and outline every organ (liver, spleen, kidneys) in 3D, helping surgeons plan operations.
5. The Librarian (Search): You can show it a weird scan, and it can find other patients with similar scans and reports from its massive memory bank to help with diagnosis.

Why is this a big deal?

• It sees the whole picture: Unlike other AIs that look at slices one by one, Merlin sees the whole 3D volume, understanding how organs relate to each other.
• It's resource-friendly: It proves you don't need a supercomputer to build world-class medical AI.
• It's a generalist: Instead of building a new AI for every single disease (one for cancer, one for fractures, one for heart disease), Merlin is a "foundation model" that can be adapted to do all of them.

The Bottom Line

Merlin is like giving every radiologist a super-powered, 24/7 assistant who has read every medical book and seen every scan in the world. It doesn't replace the doctor; it lifts the heavy burden off their shoulders, letting them focus on the most critical decisions while Merlin handles the heavy lifting of scanning, measuring, and drafting reports.

The researchers have even opened the doors, releasing the code and the "textbook" (the dataset) for anyone to use, hoping to speed up the arrival of this technology to hospitals everywhere.

Technical Summary

1. Problem Statement

The medical imaging field faces a critical shortage of radiologists, exacerbated by a 6% annual increase in CT scan utilization. Specifically, abdominal CTs are complex, often containing over 300 slices per series, requiring ~20 minutes for interpretation. While AI has shown promise, current state-of-the-art approaches suffer from three main limitations:

• 2D Bias: Most Vision-Language Models (VLMs) are designed for 2D images (e.g., X-rays) and fail to capture the volumetric nature of 3D CT scans. Existing 3D adaptations often aggregate 2D slice predictions inefficiently, losing inter-slice correlation.
• Data Scarcity & Labeling Costs: Training supervised models requires expensive, manually curated labels.
• Limited Supervision: Current models often rely solely on imaging data or short reports, ignoring the rich, structured data available in Electronic Health Records (EHR) and the full context of unstructured radiology reports.

There is a lack of foundation models capable of processing full 3D CT volumes while leveraging multi-modal clinical data (images, reports, and EHR codes) without requiring additional manual annotation.

2. Methodology: The Merlin Framework

Merlin is a 3D Vision-Language Foundation Model designed specifically for abdominal CT interpretation. It is trained using a resource-efficient strategy on a single GPU.

A. Dataset Construction

The authors curated the Merlin Abdominal CT Dataset, comprising:

• Imaging: 15,331 paired abdominal CT scans (6.38 million 2D slices).
• Unstructured Text: 6.03 million tokens from radiology report "findings" sections.
• Structured Data: 1.84 million ICD diagnosis codes from EHRs.
• Preprocessing: CTs were resampled to 1.5mm in-plane and 3mm slice spacing, normalized to Hounsfield units (0-1), and cropped to 224x224x160. Radiology reports were split into anatomical sections (e.g., liver, kidneys) to facilitate granular contrastive learning.

B. Model Architecture

• Image Encoder: An Inflated 3D (I3D) ResNet-152. This architecture reuses 2D ImageNet pre-trained weights and inflates them across the third dimension, allowing the model to process the entire 3D voxel volume simultaneously rather than slice-by-slice.
• Text Encoder: Clinical Longformer, chosen for its ability to handle long context lengths (up to 4,096 tokens), accommodating full radiology reports which often exceed the token limits of standard CLIP encoders (77-256 tokens).
• Embedding Dimension: 512 dimensions for both visual and text embeddings.

C. Training Strategy

Merlin employs a multi-task learning approach without requiring manual segmentation or disease labels:

1. Contrastive Learning (Image-Report): Uses InfoNCE loss to align CT image embeddings with corresponding radiology report embeddings. The model alternates between training on full reports and anatomically split sections to improve granularity.
2. Weak Supervision (Image-EHR): Uses Binary Cross-Entropy (BCE) loss to predict the presence of ICD diagnosis codes (mapped to PheWAS phenotypes) directly from the image. This provides a coarse-grained supervisory signal linking anatomy to patient health status.
3. Optimization: Trained with AdamW, FP16 mixed precision, and gradient checkpointing on a single NVIDIA A6000 GPU (48GB VRAM) for ~160 hours.

3. Key Contributions

1. First 3D Abdominal CT VLM: Merlin is the first foundation model to process full 3D abdominal CT volumes end-to-end, leveraging both structured EHR data and unstructured radiology reports.
2. Resource Efficiency: Demonstrated that a high-performance foundation model can be trained on a single GPU, making it accessible to hospitals and institutions with limited compute resources.
3. Comprehensive Evaluation Suite: Evaluated Merlin on 752 individual tasks across 6 categories:
   • Non-Adapted (Zero-Shot): Findings classification (31 findings), Phenotype classification (692 phenotypes), and Cross-modal retrieval.
   • Adapted (Fine-tuned): 5-year chronic disease prediction, Radiology report generation, and 3D semantic segmentation (20 organs).
4. Open Science: Release of the trained models, code, and the manually reviewed, de-identified dataset (25,494 CTs) to the community.
5. Data Scaling Laws: Empirical derivation of scaling laws showing performance gains relative to dataset size, guiding future data collection efforts.

4. Key Results

Performance vs. Baselines:
Merlin consistently outperformed state-of-the-art baselines, including:

• 2D VLMs (OpenCLIP, BioMedCLIP).
• 2D-to-3D lifted models (aggregating 2D predictions).
• 3D Vision-only models (SwinUNETR, I3D ResNet with SSL).
• Recent CT embedding foundation models (MedImageInsight, Google CT Foundation).

Specific Metrics:

• Zero-Shot Findings Classification: Achieved an average F1 score of 0.741 on internal validation and 0.647 on external validation, significantly outperforming 2D baselines (which scored ~0.28).
• Phenotype Classification: Achieved a macro-average AUROC of 0.812 across 692 phenotypes.
• Cross-Modal Retrieval: Demonstrated superior recall (Recall@1 > 0.78) in retrieving correct reports from images, even with large pools (N=128).
• 5-Year Disease Prediction: Outperformed ImageNet-pretrained models by 7% (AUROC 0.757) in predicting chronic diseases (CKD, Diabetes, etc.) in healthy patients.
• 3D Segmentation: In label-scarce settings (10% training data), Merlin-initialized models outperformed nnUNet by 4.7% in average Dice score.
• Report Generation: Outperformed RadFM across all anatomical sections in BLEU, ROUGE, and RadGraph-F1 metrics.

Generalizability:
Validated on 44,098 external CTs from three external sites (including chest CTs, despite being trained only on abdominal CTs). Merlin maintained robust performance across different scanner manufacturers, slice thicknesses, and reporting styles, demonstrating strong resistance to distributional shifts.

5. Significance and Impact

• Clinical Workflow Automation: Merlin can assist radiologists by drafting preliminary reports, flagging missed findings, and retrieving similar historical cases, potentially reducing the 20-minute interpretation time per exam.
• Opportunistic Screening: By predicting future chronic diseases (e.g., cardiovascular disease, osteoporosis) from a single CT scan, Merlin enables opportunistic screening for conditions not currently being investigated.
• Democratization of AI: By proving that a 3D VLM can be trained on a single GPU, Merlin lowers the barrier to entry for healthcare systems to develop their own specialized foundation models.
• Paradigm Shift: The study validates that vision-language pretraining is superior to vision-only self-supervised learning for medical imaging, as the language modality provides a stronger, more diverse supervisory signal than geometric reconstruction tasks alone.

In conclusion, Merlin represents a significant leap forward in medical AI, bridging the gap between 2D image analysis and true 3D volumetric understanding, while effectively utilizing the full spectrum of clinical data available in modern hospitals.