🤖 AI

3D Foundation Model for Generalizable Disease Detection in Head Computed Tomography

This paper introduces FM-CT, a self-supervised 3D foundation model trained on over 360,000 unlabeled head CT scans that significantly outperforms existing models in detecting diverse diseases across both internal and external datasets, particularly when labeled data is scarce.

Original authors: Weicheng Zhu, Haoxu Huang, Huanze Tang, Rushabh Musthyala, Boyang Yu, Long Chen, Emilio Vega, Thomas O'Donnell, Seena Dehkharghani, Jennifer A. Frontera, Arjun V. Masurkar, Kara Melmed, Narges Razavia

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weicheng Zhu, Haoxu Huang, Huanze Tang, Rushabh Musthyala, Boyang Yu, Long Chen, Emilio Vega, Thomas O'Donnell, Seena Dehkharghani, Jennifer A. Frontera, Arjun V. Masurkar, Kara Melmed, Narges Razavian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to be a brilliant brain doctor. Specifically, you want it to look at 3D CT scans of human heads and instantly spot problems like bleeding, tumors, or signs of dementia.

The problem is, teaching a computer this way is usually like trying to teach a child to read by showing them one book at a time, with a teacher pointing out every single word. In the medical world, this means you need thousands of scans where a human expert has manually drawn lines around every tiny problem. This is expensive, slow, and there just aren't enough of these "perfectly labeled" books to go around.

Enter FM-HCT: The "Super-Reader" Brain Model.

The researchers in this paper created a new kind of AI called a Foundation Model. Think of this not as a specialist who only knows one disease, but as a super-learner that has read the entire library of medicine before it ever sees a specific patient.

Here is how they did it, using some simple analogies:

1. The "Self-Taught" Student (Self-Supervised Learning)

Usually, AI needs a teacher to say, "This is a tumor, this is not." But the researchers had 361,663 head CT scans with no labels at all. It was like having a library of 360,000 books with no text, just pictures.

So, they used a trick called Self-Supervised Learning.

The Analogy: Imagine giving a student a puzzle where they have to cover up 75% of the picture and guess what the missing pieces look like. Or, imagine showing them two slightly different photos of the same brain and asking, "Are these the same person?"
The Result: By playing these guessing games over and over with 360,000 scans, the AI learned the "grammar" of the brain. It learned what normal brain tissue looks like, how blood vessels twist, and what a healthy skull shape is, all without a human ever telling it what a disease looks like. It became a "super-reader" of brain anatomy.

2. The 3D vs. 2D Difference

Most older AI models looked at CT scans like a stack of paper slices (2D). They would look at one slice, make a guess, then look at the next.

The Analogy: This is like trying to understand a movie by looking at one single frame at a time. You might see a character's face, but you miss the action happening in the background.
The Innovation: FM-HCT looks at the whole 3D volume at once. It sees the brain as a solid, living object, not a stack of paper. This allows it to understand the 3D structure of a tumor or a bleed much better, just like watching the whole movie gives you the full story.

3. The "Fine-Tuning" Shortcut

Once the AI had learned the "grammar" of the brain from the 360,000 unlabeled scans, they didn't need to teach it everything from scratch again.

The Analogy: Imagine you have a student who has read every book in the library. Now, you just need to show them a few examples of "How to spot a specific type of crime."
The Result: They took this "super-reader" and gave it a tiny amount of labeled data (just a few hundred examples) for specific diseases like hemorrhages or Alzheimer's. Because the AI already understood the brain so well, it learned these specific tasks incredibly fast and accurately.

4. Why This Matters (The "Magic" Results)

The paper tested this model on different hospitals and different types of CT machines (some old, some new).

The "Generalist" Power: Even when the model went to a hospital it had never seen before, it still performed like a top expert. It didn't get confused by different machine settings or patient demographics.
The "Few-Shot" Miracle: In a world where data is scarce, this model is a superhero. The researchers showed that even if they only gave the model 8 examples of a rare disease to learn from, it could still diagnose it almost as well as if they had given it thousands of examples. It's like a detective who can solve a new case after seeing just a few clues because they already know the city so well.

The Big Picture

Currently, if you want to check for brain bleeding, you get a CT scan. If you want to check for Alzheimer's, you usually need an expensive MRI, which isn't available everywhere.

This new AI model changes the game. It suggests that we can use the cheap, fast, and widely available CT scans to detect not just bleeding, but also dementia and tumors, with high accuracy. It democratizes brain health, potentially allowing a small clinic in a resource-limited area to use a simple CT scan and this AI to catch serious diseases early, saving lives and money.

In short: They built a "brain expert" AI by letting it study thousands of unlabeled scans on its own, and now it can spot diseases with very little extra training, making advanced brain diagnostics accessible to everyone.

1. Problem Statement

Head Computed Tomography (CT) is the primary imaging modality for neurological emergencies due to its speed, accessibility, and cost-effectiveness. However, developing robust AI models for head CT disease detection faces significant hurdles:

Data Scarcity & Labeling Costs: High-quality, expert-annotated datasets are rare, especially for less common conditions (e.g., specific tumor types, edema, hydrocephalus). Most public datasets are small (e.g., RSNA ~10k, CQ500 ~1k) and focus heavily on hemorrhage detection.
Limitations of 2D Approaches: Existing models often process 3D volumes slice-by-slice using 2D CNNs. This ignores the intrinsic 3D spatial structure of the brain and struggles with conditions where slice-level labels are ambiguous or non-existent (e.g., neurodegenerative diseases).
Lack of Specialized Foundation Models: While general medical foundation models exist (e.g., for chest X-rays or abdominal CTs), there is a lack of dedicated, large-scale 3D foundation models specifically trained on head CTs that can generalize across diverse pathologies and protocols.
Label Noise: Clinical datasets often rely on Electronic Health Records (EHR) for labeling, which introduces noise compared to radiologist-reviewed datasets, hindering model training.

2. Methodology

The authors propose FM-HCT, a 3D Vision Transformer (ViT) foundation model trained via Self-Supervised Learning (SSL) on a massive, unlabeled dataset.

A. Data Curation & Preprocessing

Pre-training Dataset: 361,663 non-contrast 3D head CT scans from NYU Langone (2009–2023), covering diverse protocols (slice thickness 0.5mm–5mm) and manufacturers (Siemens, Toshiba).
Downstream Datasets: Evaluated on four datasets:
- Internal: NYU Langone (EHR-labeled, 10 tasks).
- External: NYU Long Island (EHR-labeled, 10 tasks), RSNA (Radiologist-labeled, 5 tasks), and CQ500 (Radiologist-labeled, 10 tasks).
Preprocessing:
- Conversion to NIfTI format.
- Spatial normalization to Right-Anterior-Superior (RAS) orientation.
- Resampling to isotropic resolution (1.0, 1.0, 1.0).
- Multi-windowing: Stacking three Hounsfield Unit (HU) windows (Soft tissue, Contrast/Blood, Bone) to create a 3-channel 3D volume.
- Input size: Resized to 96 × 96 × 96.

B. Model Architecture

Backbone: Customized Vision Transformer (ViT-Base).
Patch Strategy: The 96³ volume is divided into 512 patches of size 12 × 12 × 12. This smaller patch size (compared to standard 16³) captures finer-grained 3D features, balancing performance with computational cost (attention complexity scales with $O(s^6)$ where $s$ is the patch reduction factor).
Embedding: 768 dimensions, 12 layers, 12 attention heads.

C. Self-Supervised Pre-training

The model was pre-trained using two SSL frameworks, with DINO (Self-Distillation) selected as the final approach due to superior performance over MAE (Masked Autoencoders):

DINO (Discriminative): Uses a student-teacher framework. The teacher weights are an exponential moving average of the student. The model learns to match representations of different augmented views of the same scan.
- Augmentations: Global/local crops, random flips, intensity shifts, Gaussian blur.
- Training: 1000 epochs on 4x NVIDIA A100 GPUs.
MAE (Generative): Reconstructs masked patches (75% masking ratio).
- Training: 400 epochs.

D. Downstream Evaluation

Fine-tuning: Full fine-tuning of all weights for specific disease detection tasks (10 classes: Hemorrhages, Tumors, Edema, Hydrocephalus, ADRD).
Probing: Linear and Attentive probing to test representation quality without full fine-tuning.
Few-Shot Learning: Evaluated performance with limited labeled data (8 to 256 samples per class).
Retrieval: Volume-to-volume retrieval tasks to assess feature similarity for hemorrhage subtypes.

3. Key Contributions

FM-HCT Model: The first large-scale, 3D foundation model specifically designed for head CT scans, trained on 361k+ scans without manual annotations.
3D Native Architecture: Demonstrates that modeling head CTs as 3D volumes (using 12³ patches) outperforms 2D slice-based approaches and video-based adaptations, capturing essential volumetric context.
Superior Generalization: The model achieves state-of-the-art performance across diverse datasets (internal and external) and protocols, showing robustness to scanner manufacturers and slice thickness variations.
Label Efficiency: Proves that pre-trained weights enable high-performance diagnosis even with extremely limited labeled data (few-shot learning), making it viable for rare diseases where annotations are scarce.
EHR Integration: Successfully leverages noisy EHR-derived labels for large-scale pre-training and fine-tuning, validated against high-quality radiologist labels.

4. Key Results

Performance Gains:
- On internal NYU Langone data, FM-HCT (fine-tuned) achieved a macro-AUC of 0.852, a 16.07% relative improvement over models trained from scratch (0.734).
- On external datasets (NYU Long Island, RSNA), improvements were 20.86% and 12.01% respectively over scratch models.
- Outperformed existing CT foundation models (Merlin, Google CT, CT-FM) by significant margins (e.g., 13.05% higher macro-AUC than Merlin on average).
Few-Shot Learning:
- With only 256 samples per class, the model achieved performance comparable to training on the full dataset.
- For Intraventricular Hemorrhage (IVH) on RSNA, an 8-shot model achieved an AUC > 0.90, rivaling full-data training.
Retrieval Performance:
- Achieved superior volume-to-volume retrieval mAP for hemorrhage subtypes compared to Merlin and CT-FM.
Generalization:
- The model maintained high performance on external datasets (OOD) without site-specific fine-tuning, demonstrating strong transferability.
- Visual attention maps (Fig. 5) confirmed the model focuses on clinically relevant regions (e.g., ventricles for ADRD, hemorrhage locations) after fine-tuning.

5. Significance and Impact

Clinical Applicability: FM-HCT addresses the critical bottleneck of data scarcity in medical AI. By enabling accurate diagnosis with minimal labeled data, it facilitates the deployment of AI for rare conditions and in resource-limited settings where MRI is unavailable.
Standardization: Sets a new benchmark for 3D head CT analysis, moving beyond 2D slice processing to fully volumetric understanding.
Scalability: The study confirms scaling laws in medical imaging; performance improves significantly with larger pre-training datasets, suggesting that multi-institutional collaborations and federated learning could further enhance model capabilities.
Broad Utility: Beyond acute hemorrhage detection, the model shows promise for detecting chronic conditions (Alzheimer's, tumors) and could support prognosis analysis (e.g., predicting hematoma expansion).

In conclusion, FM-HCT represents a significant leap forward in medical AI, demonstrating that self-supervised learning on massive, unlabeled 3D datasets can create robust, generalizable foundation models capable of transforming head CT diagnostics.