🤖 AI

RAP: Retrieve, Adapt, and Prompt-Fit for Training-Free Few-Shot Medical Image Segmentation

The paper proposes RAP, a training-free framework for few-shot medical image segmentation that leverages morphological retrieval, boundary-aware adaptation, and strategic prompt generation to enhance the Segment Anything Model 2 (SAM2) without fine-tuning, achieving state-of-the-art performance across multiple benchmarks.

Original authors: Zhihao Mao, Bangpu Chen

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhihao Mao, Bangpu Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a doctor trying to draw a perfect outline around a tumor or a heart chamber on a patient's MRI scan. Usually, to teach a computer to do this, you need thousands of labeled examples (like showing it 1,000 pictures of hearts with the chambers already drawn). But in the real world, doctors are busy, and getting those thousands of labels is expensive and slow.

Few-shot segmentation is the idea of teaching the computer with just one example (or a few). The paper you shared, RAP, is a new, "training-free" method that does this incredibly well without needing to retrain the AI from scratch.

Here is how RAP works, explained through simple analogies:

The Problem: The "Look-Alike" Trap

Existing methods try to find a similar picture in a database and copy the drawing from it. But medical images are tricky. One scanner might make an organ look bright white, while another makes it dark gray. If the computer just copies the "look," the drawing might end up in the wrong place or the wrong shape. It's like trying to copy a map from a photo taken in the rain onto a photo taken in the sun—the shapes are there, but the details are muddy.

The Solution: RAP (Retrieve, Adapt, Prompt-Fit)

RAP solves this by acting like a super-smart, adaptable assistant who doesn't just copy-paste, but actually understands the shape and adjusts it. It uses three steps:

1. Retrieve: "Finding the Best Reference"

Imagine you have a giant library of medical scans. You need to find the one picture that looks most like the patient you are currently treating.

How it works: Instead of just looking at the colors (which change between scanners), RAP uses a "super-eyes" AI (called DINOv3) that understands the meaning of the image. It ignores the lighting differences and finds the patient whose heart or kidney has the most similar "soul" or structure.
The Analogy: It's like asking a librarian, "I need a book about a specific type of tree," and the librarian finds the book based on the type of tree, not the color of the cover.

2. Adapt: "The Stretchy Rubber Sheet"

Now that RAP has the best reference picture, it needs to fit that drawing onto your new patient's scan. But patients aren't identical; one heart might be slightly rotated or bigger.

How it works: RAP takes the outline from the reference and stretches, rotates, and shrinks it to fit the new patient. It uses a special technique called Oriented Chamfer Matching.
The Analogy: Imagine the reference outline is a rubber sheet. You place it over the new patient's scan. RAP pulls and twists the rubber sheet until the edges snap perfectly onto the boundaries of the organ in the new scan. It also uses a "semantic gate" (a smart filter) to make sure it only stretches the rubber sheet over the right area, ignoring background noise.

3. Prompt-Fit: "The GPS for the Final Touch"

Now RAP has a rough, stretched-out outline (a "pre-mask"). It's close, but maybe not perfect. It needs to tell a powerful AI (called SAM2, which is like a master artist) exactly where to draw.

How it works: Instead of just giving the artist a vague shape, RAP places specific "dots" on the image.
- Positive dots: Placed inside the organ using a Voronoi pattern (think of a honeycomb or a cracked eggshell) to ensure the dots cover the whole area evenly.
- Negative dots: Placed just outside the boundary to say, "Stop drawing here."
The Analogy: Imagine you are guiding a blindfolded artist. Instead of saying "draw a heart," you put a sticker on the center of the heart and a red line around the edge. The artist (SAM2) sees these stickers and instantly knows exactly how to fill in the shape perfectly.

Why is this a Big Deal?

No Training Required: Most AI models need to be "taught" for weeks on specific data. RAP is ready to use immediately. You just give it a database of a few examples, and it works.
Robustness: Because it focuses on the shape and structure rather than the colors or brightness, it works even if the patient's scan was taken on a different machine or in a different hospital.
State-of-the-Art: In tests on liver, kidney, and heart scans, RAP beat all other current methods, even those that required heavy training.

Summary

RAP is like a master tailor who doesn't need to learn a new trade for every customer.

Retrieves the best pattern from a library.
Adapts (stretches and fits) that pattern to the specific customer's body.
Prompts a sewing machine (SAM2) with precise pins to stitch the final, perfect outfit.

It proves that by combining smart retrieval with geometric fitting, we can get amazing medical AI results without the heavy cost of training new models every time.

1. Problem Statement

Few-Shot Medical Image Segmentation (FSMIS) aims to segment anatomical structures using only one or a few labeled support images. While deep learning has advanced this field, existing methods face three critical limitations in medical contexts:

Training Dependency: Most methods require task-specific training on base classes, limiting generalization to unseen domains or modalities.
Visual Sensitivity: Performance heavily relies on visual similarity between support and query images, which is problematic due to high variability in medical scans (different scanners, protocols, patients).
Underutilization of Priors: Existing approaches often fail to leverage the strong anatomical priors inherent in medical images (e.g., consistent organ shapes, spatial layouts, and boundary geometries).

The authors propose RAP, a training-free framework that avoids gradient updates and fine-tuning, instead relying on pre-trained foundation models and geometric adaptation.

2. Methodology: The RAP Framework

RAP operates in three distinct stages: Retrieve, Adapt, and Prompt-Fit. It integrates DINOv3 (for semantic features) and SAM2 (Segment Anything Model 2) for final segmentation.

Stage 1: Retrieve

Goal: Select the most anatomically compatible support sample from a database to reduce brittleness caused by a single support choice.
Mechanism:
- Extracts global semantic descriptors from images using DINOv3.
- Computes cosine similarity between the query image and the support database.
- Strategy: Selects the rank-2 match (second-best) rather than the top-1 to avoid potential self-matching artifacts when the query and support originate from the same patient volume.

Stage 2: Adapt

This stage aligns the support mask shape to the query image, handling domain shifts and geometric variations. It consists of three sub-components:

Frequency-Domain Style Adaptation:
- Uses Discrete Wavelet Transform (DWT) to decompose images into low-frequency (LL) and high-frequency (LH, HL, HH) subbands.
- Replaces the support image's low-frequency (global appearance) subband with the query's, while preserving the support's high-frequency edge details. This aligns brightness/contrast while keeping structural edges intact.
DINO-Based Semantic Gating:
- Clusters the support mask regions using K-Means on DINO features.
- Computes semantic similarity maps between support prototypes and the query.
- Generates a gating mask ( $G$ ) to identify candidate regions where the target structure is likely to appear, narrowing the search space for shape matching.
Oriented Chamfer Matching (OCM):
- Extracts boundary points and normal directions from the support mask.
- Partitions query edge pixels into angular bins based on gradient directions.
- Searches for the optimal transformation (translation, scale, rotation) that minimizes the oriented chamfer distance within the gating mask.
- Produces a preliminary mask ( $M_{pre}$ ) by applying the optimal transformation to the support mask and intersecting it with the gating mask.

Stage 3: Prompt-Fit

This stage converts the preliminary mask into optimized prompts for SAM2 without fine-tuning.

Voronoi-Based Positive Points:
- Applies Farthest Point Sampling (FPS) on the preliminary mask to generate seed points.
- Creates a Voronoi partition of the mask region and selects the centroid of each cell as a positive prompt point. This ensures comprehensive geometric coverage of the target.
Sector-Based Negative Points:
- Partitions the boundary exterior into angular sectors.
- Samples negative points from the exterior region based on low DINO semantic similarity to the foreground, ensuring they are placed near the boundary but outside the object.
SAM2 Inference:
- Feeds the positive points, negative points, and a bounding box (derived from $M_{pre}$ ) into SAM2 to generate the final segmentation mask.

3. Key Contributions

RAP Framework: A novel, training-free approach for FSMIS that synergistically combines vision foundation models (DINOv3, SAM2) with geometric shape priors.
Oriented Chamfer Matching with Semantic Gating: A robust shape adaptation algorithm that aligns support shapes to queries across domains by considering both semantic similarity and gradient direction, effectively handling appearance variations.
Voronoi-Based Prompt Generation: A strategy that generates geometrically aware positive and negative points, significantly improving SAM2's ability to segment complex medical structures without fine-tuning.
State-of-the-Art Performance: Demonstrates that explicit structural fitting combined with retrieval-augmented prompting offers a simple, effective route to robust medical segmentation.

4. Experimental Results

The method was evaluated on three benchmarks: Abd-MRI, Abd-CT, and Card-MRI (covering liver, kidneys, spleen, and cardiac structures).

Performance: RAP consistently outperformed both training-based baselines (e.g., PANet, SSL-ALPNet, FAMNet) and the previous training-free state-of-the-art (MAUP).
- Abd-MRI: Achieved 69.16% mean Dice (vs. 67.09% for MAUP).
- Abd-CT: Achieved 69.56% mean Dice (vs. 67.46% for MAUP).
- Card-MRI: Achieved 75.27% mean Dice, with significant gains on thin structures like the Left Ventricle Myocardium (+1.27%) and Right Ventricle (+3.58%).
Ablation Studies: Confirmed that each component contributes to performance:
- Oriented Chamfer Matching added +1.88% Dice.
- Semantic Gating added +0.85%.
- Voronoi-based Prompts added +0.85%.
Qualitative Analysis: RAP produced more accurate boundaries, particularly for thin structures (myocardium), avoiding missing regions or spurious inclusions common in other methods.

5. Significance and Impact

Training-Free Deployment: RAP eliminates the need for task-specific training or fine-tuning, allowing immediate deployment to new medical domains or modalities with only a small database of annotated samples.
Robustness: By leveraging anatomical priors and geometric alignment rather than just pixel-level similarity, RAP is highly robust to variations in scanners, protocols, and patient populations.
Efficiency: The approach bridges the gap between powerful foundation models and the specific constraints of medical imaging (shape consistency, boundary precision) without the computational cost of retraining.
Future Direction: The authors plan to extend RAP to 3D volumetric segmentation by leveraging inter-slice consistency.

In conclusion, RAP demonstrates that combining retrieval-augmented prompting with explicit structural fitting is a highly effective strategy for solving the few-shot medical image segmentation problem without the overhead of traditional training.