💻 computer science

Automatic Segmentation of 3D CT scans with SAM2 using a zero-shot approach

This paper demonstrates the feasibility of fully zero-shot 3D medical image segmentation by adapting the Segment Anything Model 2 (SAM2) to volumetric CT data through inference-only modifications that treat slices as ordered sequences, achieving coherent results on the TotalSegmentator dataset without any fine-tuning.

Original authors: Miquel Lopez Escoriza, Pau Amargant Alvarez

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Miquel Lopez Escoriza, Pau Amargant Alvarez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, 3D block of Jell-O that represents a human body, but instead of being see-through, it's made of thousands of thin, stacked slices of bread (like a loaf of bread). Your goal is to find and outline all the bones inside this loaf without cutting it apart.

This is the challenge of 3D medical segmentation. Traditionally, to teach a computer to do this, you'd need to hire a team of doctors to manually draw outlines on every single slice of every single patient's scan. This is slow, expensive, and creates a "bottleneck" where progress stops because we don't have enough labeled data.

Enter SAM2 (Segment Anything Model 2). Think of SAM2 as a super-smart, general-purpose "highlighter pen" that was trained on millions of photos of cats, cars, and trees. It's amazing at finding objects in 2D pictures, but it's never seen a 3D medical scan before.

The Problem: The "Video" vs. "Loaf" Confusion

The researchers wanted to use SAM2 to highlight bones in these 3D CT scans without retraining it (a "zero-shot" approach). This means they wanted to use the model exactly as it was, just by changing how they asked it to work.

Here's the catch: SAM2 was designed for videos.

In a video: Frame 1 is a car, Frame 2 is the car slightly moved, Frame 3 is the car a bit further. The car changes smoothly over time.
In a CT scan: Slice 1 is a bone, Slice 2 is the same bone but slightly thicker, Slice 3 is the bone getting thinner. The bone doesn't move over time; it changes over space.

If you just feed the CT scan to SAM2 like a video, the model gets confused. It might look at a slice 50 steps away and say, "Hey, that looks like the bone I saw earlier!" and accidentally highlight the wrong thing, because in a video, things usually look similar to their past selves. But in a CT scan, a bone 50 slices away might look completely different or be a totally different bone.

The Solution: Teaching the Highlighter to "Think in 3D"

The authors didn't change the model's brain (the weights). Instead, they changed the instructions (the inference pipeline) to trick the model into understanding the 3D structure. They treated the CT scan like a video but added some clever "rules of engagement."

Here are the three main tricks they used, explained with analogies:

1. The "Memory Bank" Filter (Structured Prompt Selection)

Imagine you are trying to find a specific friend in a crowded room by asking people who saw them recently.

The Old Way: You ask everyone in the room, including people who saw your friend 10 minutes ago. They might say, "Oh, I saw someone who looked like him!" but it's actually a stranger who just walked in.
The New Way: You only ask the people who saw your friend right now or very recently.
The researchers found that if they told SAM2 to only remember the "prompts" (the initial outlines) that were close to the current slice, the model stopped getting confused by distant, look-alike bones.

2. The "Smart Slicing" Trick (Intelligent Slicing)

SAM2 has a "memory" of the last few frames it saw.

The Old Way: It remembers the last 6 slices equally. But in a CT scan, the middle slices are often just boring, repetitive copies of the ones before them.
The New Way: The researchers realized that the very first and very last slices of a sequence hold the most important "context" (like the start and end of a sentence). They told the model to pay extra attention to the "bookends" of the sequence and ignore the boring middle parts. This made the model faster and more accurate.

3. The "Three-Angle" Strategy (Multi-Axis Propagation)

Imagine trying to describe a statue to someone who can only see it from the front. You might miss the details on the side.

The Old Way: The model only looked at the CT scan from one direction (top-to-bottom).
The New Way: The researchers made the model look at the scan from three different angles (top-to-bottom, front-to-back, and side-to-side). Then, they took the results from all three views and mashed them together (like averaging three different opinions) to get the final, most accurate outline.

The Results: A Magic Trick with Real Impact

The team tested this on 2,500 CT scans containing 104 different bone structures. They didn't train the model once; they just tweaked the instructions.

The Outcome: Even though the model was "frozen" (it hadn't learned anything new), it became incredibly good at finding bones.
The "Vertebra" Win: The spine (vertebrae) is notoriously hard to segment because the bones look very similar to each other. The new method improved accuracy on the spine by 12.5% compared to the basic method.
The Efficiency: By being smarter about what it remembers, the model also ran faster.

Why This Matters

This paper is like showing that you don't need to build a new car engine to drive better; sometimes, you just need to learn a better driving technique.

By proving that a general-purpose AI can do complex 3D medical work without needing expensive, time-consuming retraining, this opens the door for:

Faster Diagnosis: Doctors can get 3D outlines of bones almost instantly.
Lower Costs: Hospitals don't need massive supercomputers to train new models for every new disease.
Better Tools: It shows that "foundation models" (the big, general AIs) are powerful enough to handle specialized medical tasks if we just speak their language correctly.

In short, the researchers taught a 2D video expert how to read a 3D book, and it turned out to be a very successful story.

1. Problem Statement

The paper addresses the challenge of applying Foundation Models for image segmentation to 3D medical imaging without the need for expensive, task-specific fine-tuning.

The Gap: While models like SAM (Segment Anything) and SAM2 show strong generalization in natural images and videos, they lack inherent volumetric awareness. SAM2 treats data as a temporal video sequence, assuming smooth object evolution over time. However, 3D CT volumes are spatially sampled; consecutive slices represent spatial proximity, not temporal evolution, often leading to abrupt anatomical changes, partial visibility, or topological variations that confuse standard video-based memory mechanisms.
The Constraint: Existing medical adaptations (e.g., Medical SAM 2) rely on supervised fine-tuning, which requires large, annotated datasets and significant computational resources. This work investigates a strictly zero-shot setting: Can SAM2 segment 3D CT volumes effectively using frozen weights solely through inference-time architectural and procedural modifications?

2. Methodology

The authors propose a pipeline that reinterprets a 3D CT volume as a "pseudo-video" by mapping the depth ( $z$ ) axis to the temporal dimension. The core innovation lies in modifying the inference pipeline rather than the model weights.

A. Data and Preprocessing

Dataset: TotalSegmentator dataset (104 anatomical structures). The study focuses specifically on bone structures due to their complex geometry and similarity to surrounding tissues.
Preprocessing: Intensity windowing tailored to bone tissue is applied, followed by normalization to three-channel inputs. This step alone improved accuracy by ~30% compared to raw data.
Prompting: To ensure reproducibility, ground-truth masks are used to simulate user prompts (mask prompts) rather than points or boxes.

B. Inference-Time Modifications

The authors introduce three key strategies to adapt SAM2's video memory mechanism to 3D spatial data:

Structured Prompt Selection (SPS):
- Problem: Including all prompted frames in the memory bank with equal weight causes attention to be dominated by visually similar but anatomically distinct structures far away in the volume.
- Solution: A temporal distance threshold is applied. Only prompted slices within a specific spatial proximity (threshold $\approx 0.3$ ) are retained in the memory bank. This reduces noise from distant, irrelevant slices.
Intelligent Slicing (IS) for Memory Bank:
- Problem: The default SAM2 memory bank stores a fixed number of recent frames (e.g., 6). Analysis showed that intermediate temporal embeddings are highly redundant.
- Solution: Instead of storing a continuous window, the method retains only the two most recent frames but assigns them the first and last learned temporal embeddings. This preserves the "start" and "end" context of the sequence while minimizing redundancy, improving runtime and coherence.
Multi-Axis Propagation and Fusion:
- Problem: Processing along a single axis (e.g., Axial) introduces directional bias and may miss structures better defined in other planes.
- Solution: Segmentation is performed independently along the Axial, Coronal, and Sagittal axes. The resulting logit volumes are reoriented to a common frame and fused using an average sigmoid:
  $P_{fused} = \sigma\left(\frac{L_A + L_C + L_S}{3}\right)$
- Finding: This strategy is most effective when the prompt budget is increased (9 prompts total: 3 per axis) rather than concentrating prompts on a single axis.

3. Key Contributions

Zero-Shot Feasibility: Demonstrates that SAM2 can produce coherent 3D segmentations on medical data without any weight updates, provided the inference pipeline is carefully engineered.
Inference-Only Adaptation: Proposes a novel set of "inference-alone" modifications (SPS, IS, Multi-Axis) that bridge the gap between video-based temporal assumptions and 3D spatial realities.
Systematic Ablation: Conducts a rigorous study on 500 CT scans to isolate the impact of memory size, prompt selection, and propagation strategies, identifying that spatial proximity matters more than temporal recency in this context.
Efficiency: Shows that reducing memory bank size (via Intelligent Slicing) can actually improve performance and reduce inference time by eliminating redundant embeddings.

4. Results

The evaluation was conducted on a final subset of 2,500 CT volumes.

Performance Metrics:
- Baseline (Standard SAM2 inference): Dice Score $\approx 0.804$ .
- Optimized Zero-Shot (IS + SPS + Multi-Axis): Dice Score $\approx 0.841$ (an absolute improvement of ~4%).
- Hausdorff Distance: Improved from $6.24$ (Baseline) to $4.79$ (Optimized), indicating better boundary accuracy.
Anatomical Specifics:
- The most significant gains were observed in Vertebrae (+12.5% Dice), a structure prone to abrupt spatial changes. The proposed method successfully maintained tracking across slices where the baseline failed (often segmenting the wrong vertebra due to confusion with distant, visually similar slices).
- Prompt Budget: Distributing prompts across three axes (9 total) outperformed concentrating them on one axis, provided the total prompt count was sufficient.
Runtime: The "Intelligent Slicing" strategy reduced tracking runtime compared to the standard 6-frame memory window.

5. Significance and Conclusion

Clinical Relevance: This work proves that foundation models can be adapted for 3D medical tasks without the "annotation bottleneck" of fine-tuning. It offers a viable path for rapid deployment in clinical settings where annotated data is scarce.
Theoretical Insight: The study highlights that volumetric awareness can emerge from inference configuration alone. It challenges the assumption that large memory banks are always better, showing that for 3D data, selectivity (focusing on recent and spatially relevant frames) is superior to volume (storing many frames).
Limitations: While effective, the approach is still constrained by the model's original 2D/video design. Structures with extreme topological variations still pose challenges, and the method currently relies on simulated prompts (ground truth) rather than true interactive user input, though the authors argue the pipeline is designed for such interaction.

In summary, the paper establishes a new paradigm for 3D medical segmentation: frozen foundation models + smart inference strategies = competitive zero-shot performance.