💻 computer science

DiffuSAM: Diffusion-Based Prompt-Free SAM2 for Few-Shot and Source-Free Medical Image Segmentation

DiffuSAM is a prompt-free medical image segmentation framework that adapts SAM2 by synthesizing mask-like embeddings through a lightweight diffusion prior conditioned on spatial consistency, enabling effective few-shot and source-free domain adaptation without requiring user prompts or extensive fine-tuning.

Original authors: Tal Grossman, Noa Cahan, Lev Ayzenberg, Hayit Greenspan

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tal Grossman, Noa Cahan, Lev Ayzenberg, Hayit Greenspan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Smart but Clueless" Robot

Imagine you have a super-intelligent robot named SAM2. This robot was trained on millions of photos of cats, dogs, and cars. It is amazing at looking at a picture and saying, "That's a dog!" or "That's a car!" just by you pointing a finger at it (a "prompt").

However, when you show this robot an X-ray or an MRI scan, it gets confused. Medical images look very different from photos of pets; they are grayscale, have different textures, and show internal organs instead of fur.

The Issue: To make SAM2 work on medical scans, doctors usually have to spend a lot of time and money "re-training" it, and they still have to manually point at every single organ for the robot to understand what to cut out. This is slow and requires a human to hover over the screen constantly.

The Solution: DiffuSAM (The "Magic Translator")

The authors created a new system called DiffuSAM. Think of it as a specialized translator that sits between the raw medical image and the robot (SAM2).

Instead of asking a human to point at the liver or kidney, DiffuSAM does the heavy lifting. It looks at the medical image and imagines what the "instruction" should be, then feeds that instruction to SAM2.

Here is how it works, step-by-step:

1. The "Guessing Game" (Diffusion)

The core of DiffuSAM is a Diffusion Model.

The Analogy: Imagine a blurry, static-filled TV screen (pure noise). A skilled artist (the Diffusion Model) slowly removes the static, one layer at a time, until a clear picture emerges.
In the Paper: The system starts with random noise. It uses the medical image as a "guide" to slowly clean up that noise until it forms a perfect digital instruction card (called a "memory embedding"). This card tells SAM2 exactly where the organs are, without a human ever touching the screen.

2. The "Frozen Brain" (SAM2)

The authors didn't re-teach the whole robot (SAM2) how to see. That would be like trying to teach an adult how to read from scratch.

The Analogy: They kept SAM2's brain frozen (locked in its original state). They only trained the "translator" (the Diffusion Model) to speak SAM2's language.
The Result: The translator takes the medical image, creates the "instruction card," and hands it to the frozen SAM2. SAM2 then instantly draws the outline of the organs.

3. The "3D Puzzle" (Volumetric Consistency)

Medical scans aren't just single pictures; they are stacks of slices (like a loaf of bread). If you slice a loaf of bread, the shape of the bread in slice #5 should look very similar to slice #4 and slice #6.

The Analogy: If you are drawing a 3D object on paper, you don't want your drawing of the top slice to look like a circle, while the slice below it looks like a square.
In the Paper: DiffuSAM looks at the slices next to the one it's currently working on. It uses the "neighbor" slices to make sure the organ looks consistent as it moves through the 3D volume. This prevents the robot from getting confused and drawing a kidney that suddenly disappears or changes shape between slices.

Why is this a big deal? (The Results)

The paper tested this on two major challenges:

Few-Shot Learning (Learning with very little data):
- Scenario: Imagine you only have 3 examples of a specific organ to teach the system, but you need it to work on 27 new scans.
- Result: DiffuSAM was able to learn from those tiny examples and perform better than other methods that required massive amounts of data or manual pointing. It achieved an average accuracy (Dice score) of 87.24%.
Source-Free Domain Adaptation (The "Stranger" Test):
- Scenario: You train the system on CT scans (one type of medical image) but then ask it to work on MRI scans (a completely different type of image) without ever showing it an MRI during training.
- Result: Usually, robots fail this test. But because DiffuSAM learned the "shape" of the organs in the abstract space, it could adapt to the new MRI style without needing the original CT images anymore. It performed just as well as the best existing methods for this specific task.

Summary

DiffuSAM is a clever trick that lets a general-purpose AI (SAM2) work on medical images without needing a human to point at every organ. It uses a "noise-removing" AI to generate the instructions automatically and checks neighboring slices to ensure the 3D anatomy makes sense. It works well even when you have very little training data or when switching between different types of medical scanners.

1. Problem Statement

Medical image segmentation is critical for clinical diagnosis but faces significant hurdles with current state-of-the-art foundation models like the Segment Anything Model (SAM) and its successor SAM2.

Domain Gap: SAM and SAM2 are trained on natural images and struggle to generalize to medical data due to differences in texture, contrast, and anatomical structures.
Prompt Dependency: These models require user-provided prompts (points, boxes, or text) to function, which is impractical for automated clinical workflows.
Data Scarcity: Supervised fine-tuning requires large, expert-annotated datasets, which are expensive and time-consuming to create.
Limitations of Existing Adaptations: Previous attempts to adapt SAM/SAM2 to medical data often require computationally heavy fine-tuning or still rely on prompts. Furthermore, existing prompt-free variants do not leverage the volumetric memory capabilities of SAM2.

2. Methodology: DiffuSAM

The authors propose DiffuSAM, a framework that adapts SAM2 for prompt-free medical segmentation using a conditional latent diffusion model. The core idea is to synthesize "memory embeddings" that SAM2 can use to generate masks, effectively replacing the need for user prompts.

Core Architecture

Frozen Backbone: The SAM2 image encoder and memory encoder remain frozen during training. The model only learns a lightweight diffusion prior.
Diffusion Prior Training:
- Input: The model takes the image embedding ( $z_{img}$ ) from the frozen SAM2 image encoder and the class label ( $l$ ).
- Target: It learns to generate the corresponding memory embedding ( $z_{mem}$ ) that SAM2 would normally produce if given a prompt.
- Process: A UNet-based diffusion model is trained under a Denoising Diffusion Probabilistic Model (DDPM) schedule. It learns to denoise a noised memory embedding conditioned on the image embedding and label.
- Loss Function: The training objective combines a prior loss (reconstructing the memory embedding) and a segmentation loss (ensuring the generated embedding produces accurate masks when passed through SAM2's mask decoder).
Inference (Prompt-Free Generation):
- At inference, the diffusion model starts with pure noise and the image embedding.
- It iteratively denoises to synthesize a memory embedding ( $\hat{z}_{mem}$ ).
- This synthesized embedding is injected into SAM2's memory attention and mask decoder to produce the final segmentation mask without any user input.

Volumetric Consistency (3D Conditioning)

To address the 3D nature of medical volumes (CT/MRI), DiffuSAM enforces spatial consistency across slices:

When generating the memory embedding for a specific slice $i$ , the model conditions on the segmentation predictions of adjacent slices ($adj$).
The adjacent slice predictions are encoded via the memory encoder and fused with the current slice's data.
This ensures anatomical continuity and prevents inconsistencies between slices in the 3D volume.

Source-Free Unsupervised Domain Adaptation (SF-UDA)

The framework is designed for SF-UDA, where the model is trained on a source domain (e.g., CT) and tested on a target domain (e.g., MRI) without access to source images during target adaptation.

The method leverages the feature affinity property: SAM2 image embeddings for the same class tend to cluster closely even across different domains.
The source-trained diffusion prior is applied directly to target domain slices without further fine-tuning, generating pseudo-memory embeddings for the target domain.

3. Key Contributions

Novel Framework: Introduction of DiffuSAM, the first diffusion-based adaptation of SAM2 for prompt-free medical image segmentation.
Latent Space Operation: The model operates in SAM2's latent embedding space, allowing it to capture high-level feature distributions without modifying the heavy backbone.
Volumetric Integration: Integration with SAM2's memory attention mechanism to ensure anatomical continuity across 3D slices via cross-slice conditioning.
Efficiency: The approach requires training a lightweight UNet rather than fine-tuning the entire foundation model, resulting in low memory usage (<5GB GPU) and fast inference (only 2 diffusion steps).

4. Experimental Results

The method was evaluated on two major datasets: BTCV (CT scans) and CHAOS (MRI scans), focusing on segmenting the spleen, kidneys, and liver.

Few-Shot Setting (BTCV)

Setup: Trained on 10% of data (3 volumes), tested on 90% (27 volumes).
Performance: DiffuSAM achieved an average Dice score of 87.24%.
Comparison: It outperformed native SAM/SAM2 (which rely on prompts) and supervised UNet/UNETR models trained on the same small dataset. It was competitive with FATE-SAM (which requires ground-truth slices at test time) but achieved this without manual prompts or test-time supervision.
Ablation: Adding volumetric cross-slice conditioning improved the average Dice from 82.72% (DiffuSAM-no 3D) to 87.24%.

Source-Free UDA Setting (CT $\to$ MRI)

Setup: Trained on BTCV (CT), tested on CHAOS (MRI) without access to source images.
Performance: DiffuSAM achieved an average Dice of 85.2%.
Comparison: It significantly outperformed other source-free baselines (DPL: 56.0%) and performed on par with DFG (85.2%), a strong feature-affinity-based method.
Ablation: Volumetric conditioning improved performance from 80.4% to 85.2%.

5. Significance and Conclusion

Clinical Applicability: By eliminating the need for user prompts, DiffuSAM bridges the gap between foundation models and automated clinical workflows.
Data Efficiency: The method demonstrates that high-quality segmentation can be achieved with minimal training data (few-shot) and without access to source data during target adaptation (SF-UDA).
Computational Efficiency: Unlike full fine-tuning of large models, DiffuSAM trains a lightweight diffusion prior, making it accessible for resource-constrained environments.
Future Work: The authors acknowledge that while MedSAM (trained end-to-end with prompts) performs better, DiffuSAM offers a unique advantage by avoiding prompts and full fine-tuning. Future work will explore integrating additional SF-UDA techniques.

In summary, DiffuSAM successfully leverages the generative power of diffusion models to "hallucinate" the necessary memory embeddings for SAM2, enabling robust, prompt-free, and domain-adaptive medical image segmentation.