💻 computer science

Segmentation of Gray Matters and White Matters from Brain MRI data

This paper proposes a modified MedSAM foundation model with a fine-tuned prompt encoder and an extended three-class mask decoder to achieve accurate multi-class segmentation of gray and white matter in brain MRI data, achieving a Dice score of 0.8751 on the IXI dataset.

Original authors: Chang Sun, Rui Shi, Tsukasa Koike, Tetsuro Sekine, Akio Morita, Tetsuya Sakai

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chang Sun, Rui Shi, Tsukasa Koike, Tetsuro Sekine, Akio Morita, Tetsuya Sakai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

🧠 The Big Picture: Teaching a Robot to See the Brain's Neighborhood

Imagine your brain is a bustling city. It has two main districts:

Gray Matter: The "downtown" where the busy city planners (neurons) live and work.
White Matter: The "highways" that connect these planners, allowing them to send messages quickly.

Doctors need to map these districts perfectly to understand diseases like Alzheimer's or to see how aging affects the brain. But looking at a brain scan (MRI) is like trying to read a map where the city is covered in fog and the borders between downtown and the highways are blurry.

This paper is about teaching a super-smart AI robot to cut through that fog and draw a perfect map of these two districts.

🛠️ The Problem: Old Tools vs. New Superpowers

The Old Way (Traditional Methods):
Imagine trying to separate red and blue marbles in a jar using a pair of tweezers. You have to pick them out one by one, or use a very specific sieve that only works if the marbles are a certain size.

In the past, scientists used tools like FSL FAST to do this. They were good, but they were like those tweezers: they needed a lot of manual tweaking for every new jar (every new patient scan) and often got confused by different lighting conditions.

The New Way (Foundation Models):
Enter MedSAM. Think of MedSAM as a "Universal Robot" trained on millions of different images (cats, cars, trees, and medical scans). It already knows what "edges" and "shapes" look like.

The catch? The original robot was only taught to find one thing at a time (like "find the brain" vs. "find the skull"). It didn't know how to say, "Okay, this part is the downtown, and that part is the highway."

🚀 The Solution: A Quick Software Update

The researchers in this paper didn't build a new robot from scratch. Instead, they gave the existing Universal Robot (MedSAM) a quick software update to handle three things at once:

Background (The empty space outside the brain).
Gray Matter (The downtown).
White Matter (The highways).

How they did it:

Freezing the Brain: They kept the robot's "eyes" (the image encoder) frozen. These eyes were already perfect at seeing shapes, so they didn't want to mess them up.
Training the Hands: They only trained the robot's "hands" (the decoder). They taught the hands to point to three different colors instead of just one.
The Result: The robot could now look at a brain scan and instantly paint a map: "Red for downtown, Green for highways, Black for empty space."

🥣 The Recipe: How They Prepared the Data

You can't just feed a robot a raw MRI scan; it needs to be prepped like a meal.

Skull Stripping (FSL BET): Imagine peeling an orange. The robot first removes the "peel" (the skull and scalp) so it only sees the fruit (the brain).
Slicing: The 3D brain is sliced into 2D pictures, like cutting a loaf of bread. They looked at the bread from the top (axial), the side (sagittal), and the front (coronal).
The "Pseudo-Ground Truth": Since they didn't have a human expert to draw the map for every single slice (which takes forever), they used the old "tweezer" tool (FSL FAST) to create a draft map. They taught the new robot to copy this draft map.

🏆 The Race: Which View Was Best?

They trained four different versions of the robot:

One that only looked at slices from the Top.
One that only looked at slices from the Side.
One that only looked at slices from the Front.
One that looked at all three mixed together.

The Winner:
Surprisingly, the robot that only looked at the Front view (Coronal) was the champion!

Why? Imagine looking at a loaf of bread. If you slice it from the front, you can clearly see the layers of the crust and the inside. If you slice it from the top, the layers might look squished or blurry. The "Front view" showed the clearest line between the Gray and White matter, making it easier for the robot to learn.

The Twist:
The robot that tried to learn from all views at once didn't do as well as the specialist robots. It was like trying to learn to drive a car, a boat, and a plane all at the same time; it got a little confused. Specializing in one view made it sharper.

⚠️ The Catch (Limitations)

The authors are honest about the flaws:

The "Copycat" Problem: The robot was trained on a map drawn by the old "tweezer" tool, not by a human doctor. So, the robot is really just learning to agree with the old tool, not necessarily the absolute truth.
Healthy Only: They only trained the robot on healthy brains. If a patient has a tumor or a lesion (a "pothole" in the city), the robot might get confused because it's never seen that before.
2D vs. 3D: The robot looks at one slice at a time, like looking at one page of a book. It doesn't see the whole story (the 3D volume) at once.

💡 The Takeaway

This paper proves that we don't need to build a new brain for every new medical task. We can take a powerful, pre-trained AI (MedSAM), give it a tiny update, and it becomes a master at separating brain tissues.

It's like taking a general-purpose Swiss Army Knife and adding one specific attachment to make it the perfect tool for brain surgery. While it's not perfect yet, it's a huge step toward making medical imaging faster, cheaper, and more accurate for everyone.

1. Problem Statement

Accurate segmentation of brain tissues, specifically Gray Matter (GM) and White Matter (WM), is critical for studying brain anatomy, diagnosing neurological disorders, and monitoring disease progression.

Limitations of Traditional Methods: Conventional tools like FSL FAST generate tissue probability maps but often require task-specific adjustments and struggle with diverse imaging conditions.
Limitations of Existing Deep Learning: While models like U-Net and nnU-Net are effective, they are typically task-specific, requiring training from scratch or extensive re-tuning for every new segmentation task.
The Gap: Foundation models like MedSAM (based on the Segment Anything Model) offer prompt-based flexibility but have primarily been explored for binary segmentation (foreground vs. background). Their application to multi-class brain tissue segmentation (distinguishing GM, WM, and background simultaneously) remains underexplored.

2. Methodology

The authors propose a modified MedSAM architecture tailored for multi-class brain tissue segmentation using a 2D slice-based approach.

A. Data Preprocessing Pipeline

The study utilizes the IXI Dataset (581 T1-weighted MRI scans from healthy subjects). The preprocessing involves:

Skull Stripping: Using FSL BET to remove non-brain tissues (skull, scalp).
Tissue Segmentation (Pseudo-Ground Truth): Using FSL FAST to generate probability maps for GM, WM, and Cerebrospinal Fluid (CSF).
Label Generation:
- Probability maps are thresholded (probability > 0.5) to create discrete labels.
- Voxels are assigned to three classes: Background (0), Gray Matter (1), and White Matter (2).
- CSF and other tissues are labeled as background.
Slicing: 3D volumes are sliced into 2D images along three orthogonal planes: Axial, Coronal, and Sagittal. Slices with minimal brain content are excluded.

B. Model Architecture Modification

The core contribution is a minimal architectural modification to the standard MedSAM:

Image Encoder: Frozen (pre-trained ViT-B weights) to preserve general feature representations and prevent overfitting on the relatively small MRI dataset.
Prompt Encoder: Fine-tuned.
Mask Decoder: Modified from a single-channel (binary) output to a three-channel output (one logit map per class: BG, GM, WM).
Final Output: The final segmentation is determined by taking the argmax across the channel dimension for each pixel.

C. Training Strategy

Datasets: Separate models were trained on Axial, Coronal, and Sagittal slices. A "Unified" model was also trained on a combination of all three orientations.
Hyperparameters: Trained for 10 epochs using the AdamW optimizer (learning rate $1e^{-4}$ , batch size 2).
Loss Function: Weighted Cross-Entropy Loss was used to handle class imbalance, assigning lower weights to the background class to force the model to focus on GM and WM regions.
Freezing Strategy: Only the prompt encoder and the modified mask decoder were updated; the image encoder remained frozen.

3. Key Contributions

Multi-Class Adaptation of MedSAM: Successfully extended MedSAM from binary segmentation to a 3-class segmentation task (Background, GM, WM) with minimal architectural changes (only modifying the output head).
Efficient Fine-Tuning: Demonstrated that freezing the pre-trained image encoder while fine-tuning the prompt encoder and decoder yields competitive results, significantly reducing computational costs and overfitting risks.
Orientation Analysis: Systematically evaluated how anatomical orientation (Axial vs. Coronal vs. Sagittal) affects segmentation performance in foundation models.
Pipeline Integration: Established a robust pipeline using FSL BET and FSL FAST to generate pseudo-ground truth labels for training deep learning models without manual expert annotation.

4. Results

Experiments were conducted on a test set of 1,000 randomly sampled slices. Performance was measured using Dice Similarity Coefficient (DSC) and Intersection over Union (IoU).

Best Performance: The Coronal model achieved the highest performance with a Dice score of 0.8751 and an IoU of 0.7935.
Ranking:
1. Coronal Model (Best)
2. Sagittal Model
3. Axial Model (Lowest among single-orientation models)
Unified Model: Contrary to expectations, the unified model (trained on all orientations combined) did not outperform the single-orientation models. In fact, it performed slightly worse or comparable to the best single-orientation model.
Visual Analysis:
- Coronal slices showed sharper boundaries between GM and WM.
- Axial models occasionally misclassified small GM regions as WM.
- All models showed high confidence in central brain regions but uncertainty near tissue boundaries.

5. Significance and Limitations

Significance:

The study validates that foundation models like MedSAM can be effectively adapted for complex, multi-class medical imaging tasks with minimal architectural modifications.
It highlights the potential of "prompt-based" segmentation to generalize across tasks without the heavy retraining costs associated with traditional CNNs like U-Net.
The findings suggest that anatomical orientation significantly impacts segmentation quality, with Coronal views providing the most distinct features for GM/WM separation in this context.

Limitations:

Pseudo-Ground Truth: Labels were derived from FSL FAST (an algorithm) rather than manual expert annotation. Therefore, the reported Dice scores measure agreement with FAST, not absolute anatomical truth.
Data Diversity: The dataset (IXI) consists only of healthy subjects and T1-weighted scans. It lacks pathological cases (tumors, lesions) and diverse scanner protocols.
2D Approach: The study relies on 2D slices, ignoring the 3D spatial context which might improve segmentation continuity.
Background Handling: The model was not trained on pure background slices, potentially making it vulnerable to non-brain inputs.

Future Work:
The authors suggest extending MedSAM to 3D, fine-tuning on larger and more diverse datasets (including pathological cases), and exploring different fine-tuning strategies to improve the unified model's performance.