💻 computer science

Seeing Through the Tool: A Controlled Benchmark for Occlusion Robustness in Foundation Segmentation Models

This paper introduces OccSAM-Bench, a controlled benchmark evaluating SAM-family foundation models under synthesized surgical occlusion, which reveals distinct architectural behaviors between "occluder-aware" models that prioritize visible tissue and "occluder-agnostic" models that infer hidden anatomy, thereby guiding model selection based on specific clinical intent.

Original authors: Nhan Ho, Luu Le, Thanh-Huy Nguyen, Thien Nguyen, Xiaofeng Liu, Ulas Bagci

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nhan Ho, Luu Le, Thanh-Huy Nguyen, Thien Nguyen, Xiaofeng Liu, Ulas Bagci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a lost key in a messy room. Now, imagine a giant, shiny vacuum cleaner is hovering right over the spot where the key might be, blocking your view.

This is exactly the problem doctors face during endoscopic surgeries (looking inside the body with a camera). They need to find polyps (small growths) on the colon wall, but often, surgical tools (like the "vacuum cleaner") or overlapping tissues block the view.

For a long time, computer programs designed to find these polyps were tested only on "perfect" photos where nothing was blocking the view. The authors of this paper realized this was a dangerous oversight. They asked: "What happens when the computer tries to find the key while the vacuum cleaner is in the way?"

Here is a simple breakdown of their findings using everyday analogies:

1. The New Test: "OccSAM-Bench"

The researchers built a new testing ground called OccSAM-Bench. Instead of just showing the computer perfect pictures, they artificially placed digital "tools" and "cutouts" over the images to simulate a blocked view.

They didn't just ask, "Did you find the key?" They asked three specific questions:

The Visible Part: Did you correctly identify the part of the key you can actually see?
The Hidden Part: Did you guess where the rest of the key is under the vacuum?
The Whole Picture: Did you draw a box around the whole key, including the part under the vacuum?

2. The Two Types of "Detectives"

When they tested seven different AI models (famous ones like SAM, MedSAM, etc.), they discovered the models fell into two distinct personality types:

Type A: The "Cautious Detective" (Occluder-Aware)

Models: SAM, SAM 2, SAM 3, MedSAM3.
Behavior: These models are like a cautious detective who says, "I can only draw the outline of the key where I can clearly see it. I will not guess what's under the vacuum cleaner because I don't want to make a mistake."
Result: They are very accurate about the visible parts. They refuse to draw over the tool. This is safer for surgery because it prevents the robot from accidentally cutting healthy tissue thinking it's a polyp.

Type B: The "Bold Guesser" (Occluder-Agnostic)

Models: MedSAM, MedSAM2.
Behavior: These models are like a bold detective who says, "I see a key shape here, so I'm going to draw the whole key, even the part under the vacuum! I'm going to 'complete' the picture in my head."
Result: They often guess the hidden part correctly, which is cool. However, they sometimes get too bold and draw over the tool itself, thinking the tool is part of the polyp. In a real surgery, this is dangerous because the robot might try to cut the tool instead of the polyp!

The "Confused" Detective

Model: SAM-Med2D.
Behavior: This model didn't fit either category. It got confused by the tools and performed poorly at everything. It couldn't decide whether to be cautious or bold, so it just failed.

3. The Trap of "Full Mask" Scores

The paper highlights a major trap in how we usually test AI.

The Old Way: We used to give the AI a score based on how well it drew the entire key (including the hidden part).
The Problem: A "Bold Guesser" could get a high score just by accidentally drawing over the tool and hitting the hidden key by luck. A "Cautious Detective" would get a low score because it refused to draw over the tool, even though it was being more careful and accurate.
The Lesson: The old scoring system was like rewarding a driver for crashing into a wall because they happened to hit a target behind it. It's misleading and potentially dangerous.

4. The Big Takeaway

The most important message is that there is no single "best" AI model. It depends on what the doctor needs:

If you need safety: You want the Cautious Detective. You want the AI to say, "I see this part clearly, but I won't guess the rest," so the surgeon doesn't accidentally cut the wrong thing.
If you need to estimate hidden anatomy: You might want the Bold Guesser (specifically MedSAM2, which was the only one that managed to be bold without being too messy). This helps the doctor understand the full shape of a polyp even if it's partially hidden.

Summary

This paper is a wake-up call. It tells us that AI models trained on perfect, clean images often fail when the real world gets messy. We need to stop testing them only on "clean" data and start testing them on "messy" data with tools blocking the view.

Choosing the right AI for surgery isn't just about who gets the highest score on a test; it's about choosing the right personality for the job: do you want a cautious guardian or a bold guesser? The answer depends entirely on the specific medical task at hand.

1. Problem Statement

While foundation models like the Segment Anything Model (SAM) and its medical adaptations (MedSAM, etc.) have shown remarkable zero-shot generalization in medical image segmentation, they are typically evaluated on clean, curated datasets. A critical, underexplored challenge in clinical endoscopy is surgical occlusion, where target anatomical structures (e.g., polyps) are partially hidden by surgical instruments or overlapping tissues.

Current evaluation paradigms rely heavily on full-mask (amodal) overlap metrics (e.g., standard Dice Similarity Coefficient). The authors argue this is fundamentally flawed in surgical contexts:

False Positives: A model that "hallucinates" tissue over a surgical instrument may achieve a high full-mask score by coincidentally overlapping the hidden ground truth.
False Negatives: A model that correctly rejects the instrument and segments only the visible tissue is penalized because it fails to cover the occluded region.
Lack of Scrutiny: No existing benchmark systematically quantifies how foundation models handle surgical-tool occlusion, creating a vulnerability for downstream diagnostic or navigational errors in real-world workflows.

2. Methodology: OccSAM-Bench

The authors introduce OccSAM-Bench, a controlled benchmark designed to evaluate SAM-family models under synthesized surgical occlusion.

A. Controlled Occlusion Generation

The framework simulates two types of occlusion across three calibrated severity levels (Low: 0–20%, Medium: 20–40%, High: 40–60% of target area occluded):

Surgical Tool Paste: Real binary masks of surgical instruments (from the Kvasir-Instrument dataset) are overlaid onto polyp images. Tools are randomly scaled (0.8x–1.0x) and rotated (±45°) to mimic dynamic trajectories. This isolates the effect of visual confusion (foreign objects).
Cutout Occlusion: Rectangular regions of the target are removed without adding foreign visual content. This isolates the effect of missing data.

Note: Synthetic occlusion is used because true in-vivo occlusion lacks known ground truth for the hidden regions, making rigorous mathematical evaluation impossible.

B. Three-Region Evaluation Protocol

To address the flaws of standard metrics, the authors propose decomposing performance into three distinct evaluation targets:

Visible-Only Mask ( $M_{vis}$ ): Measures the model's ability to segment the actually visible tissue while rejecting the occluder. High scores here indicate safety and precision.
Invisible Mask ( $M_{inv}$ ): Measures predictions in the hidden region behind the occluder. High scores suggest "amodal completion" (inferring hidden structure) but must be interpreted cautiously, as they can result from over-prediction.
Full Mask ( $M_{full}$ ): The standard complete target. The authors argue this is often misleading in occlusion scenarios.

Key Insight: The protocol penalizes models that extend predictions into the tool region (false positives in $M_{vis}$ ) while rewarding those that accurately delineate visible boundaries.

C. Experimental Setup

Datasets: Three public polyp datasets (CVC-300, CVC-ColonDB, ETIS-LaribPolypDB).
Models: Seven state-of-the-art SAM-family models evaluated in a zero-shot setting (no fine-tuning on target data):
- General-purpose: SAM, SAM 2, SAM 3.
- Medical-adapted: MedSAM, SAM-Med2D, MedSAM2, MedSAM3.
Prompts: Evaluated using both bounding box prompts and single interior point prompts.

3. Key Contributions

Controlled Occlusion Framework: A principled pipeline for generating synthetic surgical occlusions with mathematically verifiable bounds, separating visual confusion from data loss.
Three-Region Evaluation Protocol: A novel metric system that exposes behaviors obscured by standard full-mask evaluation, specifically identifying "safe" segmentation vs. "hallucinatory" completion.
Comprehensive Zero-Shot Benchmark: The first systematic evaluation of seven foundation models under surgical occlusion, revealing distinct behavioral archetypes.

4. Key Results & Findings

A. Two Distinct Behavioral Archetypes

The analysis reveals that models fall into two categories based on how they handle occlusion:

Occluder-Aware Models (SAM, SAM 2, SAM 3, MedSAM3):
- Behavior: They treat the surgical instrument as background and prioritize accurate delineation of visible tissue.
- Performance: They achieve the highest Visible-Only DSC scores. They effectively reject the instrument, avoiding dangerous false positives.
- Implication: Ideal for clinical tasks requiring conservative, safe boundaries (e.g., robotic navigation).
Occluder-Agnostic Models (MedSAM, MedSAM2):
- Behavior: They confidently predict into occluded regions, exhibiting amodal completion tendencies.
- Performance: They achieve high Invisible DSC scores. However, this often comes at the cost of lower Visible-Only DSC, as they tend to "leak" predictions into the tool area.
- Exception (MedSAM2): Uniquely combines strong Invisible scores with competitive Visible DSC, likely due to its video-based fine-tuning strategy, making it a potential "boundary case" for tasks needing both visibility and hidden structure estimation.
Underperforming Models:
- SAM-Med2D: Aligns with neither archetype and underperforms across all conditions.
- MedSAM (Point Prompts): Collapses to near-zero performance with single-point prompts, indicating a critical dependency on bounding box inputs for its specific training protocol.

B. Impact of Severity and Occlusion Type

Severity: Performance degrades as occlusion severity increases for all models.
Occlusion Type: At high severity, Surgical Tool occlusion causes significantly larger degradation than Cutout occlusion. This suggests that the presence of foreign visual content (the tool) is more disruptive to model reasoning than simple missing data.

C. Prompt Sensitivity

Box Prompts: Consistently outperform point prompts across all models and datasets.
MedSAM Failure: MedSAM fails catastrophically with point prompts (DSC $\approx$ 0.02), confirming it is not robust to out-of-distribution prompt types.

5. Significance and Conclusion

The paper fundamentally challenges the reliance on standard full-mask metrics in medical segmentation. It demonstrates that:

High full-mask scores can be dangerous: They may reward models that hallucinate tissue over instruments.
Model selection is a clinical design choice:
- For safe navigation (avoiding false positives on tools), Occluder-Aware models (e.g., SAM 3) are preferred.
- For anatomical estimation (inferring hidden structures), Occluder-Agnostic models (e.g., MedSAM2) may be useful, provided their boundary consistency is verified.
Training Paradigm Flaw: Medical fine-tuning on clean datasets appears to introduce a vulnerability to occlusion, causing models to either overfit to pristine boundaries or hallucinate through occlusions.

Future Work: The authors acknowledge limitations regarding the 2D synthetic nature of the occlusions and call for future research involving diverse modalities, physically realistic rendering (lighting, deformation), and temporal tracking in video sequences.