Toward Mask Annotation-Free Surgical Instrument Segmentation from Endoscopic Images Using Text-Prompted Segment Anything Model 3 (SAM3)
This paper proposes a two-stage, annotation-free framework that combines the zero-shot capabilities of the Segment Anything Model 3 (SAM3) with a fine-tuned vision-language model to achieve instance-level surgical instrument segmentation, demonstrating significant improvements over direct text-prompted SAM3 application despite falling short of fully supervised methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot to perform surgery. To do this safely, the robot needs to know exactly where the surgical tools are in the video feed, distinguishing a pair of scissors from a piece of tissue or a shadow. This is called "surgical instrument segmentation." For a long time, the only way to teach robots this skill was to hire human experts to draw pixel-by-pixel outlines around every tool in thousands of hours of video. This is like trying to teach a child to recognize a dog by showing them a million photos where someone has painstakingly colored in the fur of every single dog. It works, but it is slow, expensive, and hard to scale.
Recently, a new kind of AI called the "Segment Anything Model" (or SAM) arrived on the scene. Think of SAM as a super-powered highlighter that can instantly color in any object in a picture if you just point at it or give it a simple text command like "tool." It was trained on millions of everyday photos, so it's great at finding things in the real world. However, when researchers tried to use this highlighter in the weird, shiny, and confusing world of inside-the-body surgery videos, it got confused. The robot didn't understand that "grasper" or "scissors" meant the same thing in a surgery video as it did in a photo of a kitchen. The gap between the robot's training and the surgical reality was too wide, and simply shouting the tool's name didn't work.
This paper by Nakul Poudel and colleagues tackles that exact problem. They wanted to see if they could make the robot find and identify surgical tools without needing any human-drawn outlines (masks) or manual pointing. They didn't just try to force the robot to understand specific tool names; instead, they built a two-step "detective team." First, they used the highlighter (a new version called SAM3) with a very simple, generic command: "tool." This successfully found the shape of the tools, even if it didn't know their names yet. Then, they passed these shapes to a second AI, a "brain" called Qwen, which was taught to look at the highlighted shapes and guess what kind of tool it was.
The results were a mix of "not quite perfect yet" and "very promising." The team found that while their new two-step method didn't beat the best robots that were trained with thousands of human-drawn outlines, it was a massive improvement over just trying to use the highlighter directly with tool names. In fact, the direct method often failed completely, missing tools entirely or getting the names wrong. The new method successfully identified the tools in many cases, proving that you can get a robot to "see" surgical tools without the heavy burden of manual drawing. However, the authors are careful to note that the system still makes mistakes, especially when tools are hidden or when the highlighter draws messy shapes. They suggest that while this isn't a magic solution that solves everything today, it opens a door to a future where surgical robots can learn much faster and cheaper, without needing armies of humans to draw every single picture.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.