💻 computer science

Learning from Noisy Prompts: Saliency-Guided Prompt Distillation for Robust Segmentation with SAM

The paper introduces SPD, a Saliency-Guided Prompt Distillation framework that enhances the Segment Anything Model (SAM) for medical imaging by using anatomical saliency priors and slice consistency to transform noisy, imprecise clinical prompts into robust, expert-like guidance.

Original authors: Jingxuan Kang, Ziqi Zhang, Shaoming Zheng, Shuang Li, Uday Bharat Patel, Alexander Harry Fitzhugh, Phillip Lung, Yusuf Kiberu, Nikesh Jathanna, Shahnaz Jamil-Copley, Bernhard Kainz, Chen Qin

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jingxuan Kang, Ziqi Zhang, Shaoming Zheng, Shuang Li, Uday Bharat Patel, Alexander Harry Fitzhugh, Phillip Lung, Yusuf Kiberu, Nikesh Jathanna, Shahnaz Jamil-Copley, Bernhard Kainz, Chen Qin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a highly skilled, world-class artist (this is SAM, the "Segment Anything Model") how to color in a specific, complex shape in a coloring book.

The problem? The instructions you are giving the artist are terrible. Instead of saying, "Color exactly inside this tiny, wiggly line," you are just pointing your finger vaguely at the page and saying, "Color around here somewhere." Sometimes your finger is even pointing at the wrong shape entirely!

In the medical world, this happens all the time. Doctors are busy, so instead of drawing perfect outlines around an organ, they might just draw a single line through the middle of it. This "noisy prompt" confuses the AI, causing it to color outside the lines or miss the organ entirely.

This paper introduces a clever new system called SPD (Saliency-Guided Prompt Distillation) to fix this. Here is how it works, using three simple steps:

1. The "Intuition" Phase (The Saliency Head)

Before the artist starts coloring, we give them a quick "cheat sheet." We train a small, lightweight part of the AI to look at the image and say, "Hey, I’ve seen thousands of these before; I have a gut feeling that the organ is probably in this general area."

This creates a Saliency Map—a glowing heatmap that acts like a "searchlight," highlighting where the target is most likely to be. It’s like giving the artist a faint pencil sketch before they touch the markers to the paper.

2. The "Fact-Checker" Phase (Contextual Prompt Distillation)

Now, we look at those messy instructions (the noisy prompts) again. Instead of blindly following them, the AI uses its "searchlight" from Step 1 to act as a fact-checker.

If a doctor’s point is pointing at a bone, but the searchlight says, "No, the organ is over there," the AI says, "I'm ignoring that bad instruction."

But it goes even further! It looks at the "pages" (slices) before and after the current one. It thinks: "On the previous page, the organ was right here, and on the next page, it's there. So, even though the current instruction is a bit messy, I can use the information from the neighbors to figure out exactly where to color." It’s like a detective using clues from different crime scenes to solve a single mystery.

3. The "Smooth Transition" Phase (Pairwise Slice Consistency)

If you were flipping through a flip-book animation, you wouldn't want the character to suddenly teleport from one side of the screen to the other between frames. You want smooth movement.

The AI uses a rule called Pairwise Slice Consistency to make sure the segmentation is smooth. It ensures that the shape it colors on Page 10 looks like a natural progression from Page 9 and a logical lead-in to Page 11. This prevents the AI from making "glitchy" or jumpy mistakes.

The Result

When the researchers tested this on real medical scans (like MRIs of the intestines or CT scans of the kidneys), the results were impressive.

While older AI models got "distracted" by the messy instructions and failed, SPD stayed focused. It was able to take "bad" instructions and turn them into "perfect" segmentations. It’s like taking a blurry, shouted instruction and turning it into a clear, whispered command.

In short: SPD teaches AI to stop blindly following messy human instructions and instead use "common sense" and "context" to see the truth.

Technical Summary: Saliency-Guided Prompt Distillation (SPD)

1. Problem Statement

The deployment of foundation models like the Segment Anything Model (SAM) in clinical medical imaging is hindered by a "prompt reliability bottleneck." While SAM excels at zero-shot segmentation, its performance is highly sensitive to the quality of input prompts (points or bounding boxes).

In real-world clinical workflows, available prompts are often noisy, coarse, or ambiguous. For example, radiologists may provide centerline annotations for general anatomical tracking that do not align perfectly with a specific target organ (e.g., the terminal ileum). These noisy prompts can mislead SAM, causing it to over-segment adjacent tissues or fail to capture the complete target structure. Existing medical SAM adaptations typically assume the availability of precise, task-specific prompts, creating a gap between research benchmarks and clinical reality.

2. Methodology

The authors propose Saliency-Guided Prompt Distillation (SPD), a two-stage framework designed to emulate the reasoning process of a radiologist—who uses anatomical knowledge and contextual information from adjacent slices to make decisions.

Stage I: Anatomical Prior Learning

Objective: To learn data-driven anatomical priors that act as a reliable spatial reference.
Mechanism: A lightweight saliency head is trained using a combination of Dice and Focal losses, supervised by ground-truth masks. This head operates on features from a LoRA-adapted SAM encoder.
Output: The stage produces saliency maps ( $S_t$ ) that highlight high-confidence anatomical regions, providing a "map" of where the target structure is likely to be located.

Stage II: Prompt-Guided Segmentation
This stage uses the saliency maps to refine the noisy input prompts through Contextual Prompt Distillation (CPD) and ensures volumetric coherence via Pairwise Slice Consistency (PSC).

Contextual Prompt Distillation (CPD):
1. Local Prompt Validation: Noisy prompts from the current slice are filtered; only those falling within high-saliency regions of the current saliency map are retained.
2. Context-Aware Prompt Enrichment: The model looks at neighboring slices ( $t-i, t+i$ ). It collects prompts from neighbors that are salient in their own context and then cross-validates them against the current slice's saliency map.
3. Consensus Set Formulation: The validated local prompts and enriched contextual prompts are merged into a "consensus prompt set" ( $P^*_t$ ), providing a robust and dense guidance signal for the SAM mask decoder.
Pairwise Slice Consistency (PSC):
- To prevent anatomical discontinuities in 3D volumes, a PSC loss is applied. Instead of enforcing global smoothness (which can propagate errors), it enforces local coherence by minimizing the dissimilarity between the predicted probability maps of adjacent slice pairs.

3. Key Contributions

Novel Challenge Identification: The paper is among the first to specifically address the problem of noisy prompts (rather than just noisy labels) in the context of adapting SAM to medical imaging.
Biologically-Inspired Framework: SPD mimics clinical reasoning by using anatomical priors (saliency) and inter-slice context (distillation) to correct imperfect human annotations.
Robustness via Distillation: The framework effectively converts unreliable, generic clinical cues into high-quality, task-specific consensus prompts.

4. Experimental Results

The framework was evaluated on four challenging datasets (TI, Scar, FUMPE, and KiTS) across MRI and CT modalities.

Superior Performance: SPD consistently outperformed state-of-the-art (SOTA) methods, including conventional supervised models (nnUNet, TransUNet) and SAM-based adaptations (MedSAM, MSA, SAM-Tuning).
Metric Gains: On the Terminal Ileum (TI) dataset, SPD achieved a significant 11.08% increase in Dice Similarity Coefficient (DSC) and a 6.28 reduction in Hausdorff Distance (HD95) compared to the best existing approach.
3D Capability: SPD outperformed recent 3D-aware models like SAM2 and MedSAM2 in volumetric segmentation tasks.
Ablation Insights: Experiments confirmed that the CPD module (contextual enrichment) and the PSC module (slice consistency) are both critical for achieving high boundary precision and anatomical stability.

5. Significance

This work provides a principled path toward the practical deployment of foundation models in clinical settings. By demonstrating that models can "learn from noise" to extract reliable guidance, the authors show that high-performance medical segmentation is possible even when expert annotations are imperfect or non-specific. This reduces the burden on clinicians and increases the utility of powerful models like SAM in real-world, resource-constrained medical environments.