🤖 AI

Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion

This paper introduces SurgRef, a novel motion-guided framework paired with the Ref-IMotion dataset, which achieves state-of-the-art referring segmentation of surgical instruments by leveraging dynamic motion cues rather than static visual features to overcome challenges like occlusion and ambiguous terminology.

Original authors: Meng Wei, Kun Yuan, Shi Li, Yue Zhou, Long Bai, Nassir Navab, Hongliang Ren, Hong Joo Lee, Tom Vercauteren, Nicolas Padoy

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Meng Wei, Kun Yuan, Shi Li, Yue Zhou, Long Bai, Nassir Navab, Hongliang Ren, Hong Joo Lee, Tom Vercauteren, Nicolas Padoy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in a crowded, chaotic kitchen where five chefs are cooking at once. They are all wearing identical white uniforms, and they are moving so fast that it's hard to tell who is doing what. Suddenly, a customer shouts, "Stop the chef who is currently flipping the pancake!"

If you only looked at the chefs' faces (their static appearance), you might get confused because they all look the same. But if you watch how they move, it becomes easy. You see one chef flipping, another chopping, and a third stirring. The motion tells you exactly who the customer is talking about.

This is exactly the problem the researchers in this paper are solving, but instead of a kitchen, they are looking at surgical videos.

The Problem: "Which Tool?"

In surgery, especially minimally invasive surgery (where cameras are inside the body), the view is often dark, blurry, or covered in blood. There are many metal tools that look almost identical.

Old AI: If you asked an old AI, "Show me the scissors," it would look for a tool that looks like scissors. If the scissors were hidden behind an organ or if there were two pairs of scissors, the AI would get lost.
The Limitation: Existing AI relies on "what things look like" (static cues). But in surgery, "what things look like" is often confusing.

The Solution: "Where It Moves, It Matters"

The authors, led by Meng Wei and colleagues, created a new system called SurgRef. Instead of asking the AI to recognize the shape of a tool, they taught it to recognize the dance of the tool.

Think of it like this:

Old Way: "Find the red car." (Hard if there are two red cars).
New Way (SurgRef): "Find the car that is turning left and speeding up." (Easy, even if there are ten red cars).

SurgRef understands natural language instructions that describe motion, such as:

"The tool that enters from the top right and pulls the tissue down."
"The instrument that is currently cutting while moving up and down."

By focusing on movement, the AI can find the right tool even if it's partially hidden, if the lighting is bad, or if the surgeon uses a tool the AI has never seen before.

The New "Textbook": Ref-IMotion

To teach this new skill, the researchers couldn't just use old data. They had to build a new "textbook" called Ref-IMotion.

Imagine they took thousands of hours of surgery videos and hired experts to write down exactly what the tools were doing, moment by moment.

They didn't just say "This is a grasper."
They wrote: "The grasper enters from the left, grabs the gallbladder, and pulls it gently to the right."

They created 718 of these motion descriptions paired with precise video frames. This dataset is the first of its kind to focus heavily on how tools move rather than just what they are called.

How the AI Works (The "Key-Frame" Trick)

Surgical videos are long and full of boring moments where nothing happens (like a chef standing still waiting for an ingredient).

The Problem: If the AI tries to analyze every single second of a 30-minute video, it gets tired and slow.
The Fix: SurgRef has a smart "Key-Frame Selection" module. It's like a film editor who watches the whole movie and only keeps the most important 8 seconds where the action happens.
- If the instruction is "the tool cutting," the AI ignores the boring parts and focuses only on the frames where the cutting motion is happening.
- This makes the AI faster and much more accurate.

Why This Matters

This isn't just a cool trick; it's a safety upgrade for the future of surgery.

Intelligent Assistants: Imagine a surgeon speaking to a robot: "Highlight the tool that is holding the artery." The robot instantly knows which one to highlight, even if the view is messy.
Training: Medical students could ask, "Show me how the expert moves the needle here," and the AI would replay exactly that motion.
Robotic Surgery: Robots could follow verbal commands like "Cut where the grasper is holding," making surgery more automated and precise.

The Bottom Line

The researchers proved that motion is the universal language of surgery. By teaching AI to listen to descriptions of movement rather than just names, they created a system that is smarter, faster, and better at understanding the chaotic, high-stakes world of the operating room.

In short: Don't just ask the AI what the tool is; ask it what the tool is doing. That's how you find the needle in the haystack.

1. Problem Statement

The paper addresses the challenge of Referring Surgical Instrument Segmentation, a task where a model must localize and segment specific surgical instruments in a video based on natural language descriptions. While existing methods utilize static visual cues (appearance, spatial position) and predefined instrument names, they struggle in complex surgical environments due to:

Visual Ambiguity: Occlusions, similar-looking tools, and challenging imaging conditions (bleeding, smoke, poor lighting).
Linguistic Variability: Lack of standardized terminology across institutions; the same tool may be described differently.
Static Limitations: Current models rely heavily on static appearance or explicit names (e.g., "Prograsp forceps"), failing to generalize to unseen procedures or handle dynamic scenarios where tools are indistinguishable by appearance alone.

The authors argue that instrument motion (trajectories, interactions, retraction patterns) is a more consistent and discriminative signal for localization in surgical scenes than static appearance.

2. Methodology: SurgRef

The authors propose SurgRef, a motion-guided referring video segmentation framework designed to ground free-form language expressions in surgical video by explicitly modeling instrument motion.

Architecture Components

Backbone: Utilizes a Swin Transformer for extracting multi-scale spatial features from video frames.
Text Encoder: Employs a frozen RoBERTa-base model to convert natural language expressions into dense semantic embeddings.
Language-Guided Transformer Decoder:
- Unlike standard object queries (e.g., in Mask2Former), SurgRef initializes queries by injecting the language embedding ( $F_{text}$ ) into the query vectors.
- It uses a Classification Head to score query relevance and a Mask Embedding Head to generate binary segmentation masks via dot-product with pixel features.
Key-Frame Selection Module (KFS):
- To address temporal redundancy in long surgical videos, this module adaptively selects a subset of frames ( $T'$ ) most relevant to the referring expression.
- It computes a relevance score for each frame based on language-guided object queries. Frames with high alignment scores (e.g., showing active motion) are retained, while redundant frames are discarded.
Inter-Frame Attention: After key-frame selection, a self-attention mechanism aggregates temporal motion information across the selected frames to capture object dynamics before final mask generation.

Training Strategy

The model is trained with a composite loss function combining:

Frame-level loss: Classification (cross-entropy) and mask prediction (binary cross-entropy + Dice).
Video-level loss: Temporal similarity loss to enforce consistency of query embeddings across adjacent frames.

3. Key Contributions

A. The Ref-IMotion Dataset

The authors constructed Ref-IMotion, the first large-scale, multi-institutional dataset for motion-centric referring segmentation in surgery.

Sources: Aggregates data from Cholec80 (laparoscopic cholecystectomy), EndoVis (robot-assisted porcine surgery), and GraSP (robotic prostatectomy).
Scale: 319 video clips, 21,350 annotated frames, and 718 referring expressions.
Annotation Innovation:
- Motion-Centric Expressions: Unlike previous datasets with static labels, Ref-IMotion includes descriptions of trajectories and interactions (e.g., "the tool entering from the top and pulling the gallbladder to the left").
- Dense Spatiotemporal Masks: Every expression is paired with frame-level masks and temporal intervals.
- Variety: Includes 358 motion-based expressions, covering diverse surgical modalities (laparoscopic and robotic).

B. The SurgRef Framework

A novel architecture that integrates motion-guided language grounding with video segmentation.
Introduces a Key-Frame Selection mechanism that improves temporal efficiency and focuses computation on semantically salient moments.
Capable of handling both static descriptions (appearance/spatial) and dynamic descriptions (motion/interaction).

4. Experimental Results

Performance Metrics

Evaluated using J (region similarity), F (contour accuracy), J&F (mean), Dice, and IoU.

State-of-the-Art (SOTA) Performance: SurgRef outperforms existing surgical (VIS-Net) and general-purpose (VISA, MPG-SAM 2) models across all datasets.
- Example: On EndoVis-IM17, SurgRef + KFS achieved 89.91 J and 89.42 J&F, surpassing MPG-SAM 2 (88.20 J).
- Example: On EndoVis-IM18, it achieved 84.48 J&F, beating the next best (83.03).

Generalization (Zero-Shot)

The model trained on GraSP-IM (robotic prostatectomy) was tested zero-shot on CholecSeg8k-IM (laparoscopic cholecystectomy).
Despite significant differences in anatomy, tools, and imaging modalities, SurgRef maintained strong performance (70.62 J&F), demonstrating robustness to domain shifts where static-appearance models typically fail.

Ablation Studies

Motion Supervision: Models trained with motion-based expressions significantly outperformed those trained without, even when tested on static (appearance/spatial) queries. This confirms that learning motion dynamics improves general feature representation.
Key-Frame Selection: The proposed adaptive selection strategy achieved optimal performance with $T'=8$ frames. Uniform sampling required more frames ( $T'=32$ ) to reach similar performance, proving the efficiency of the motion-guided selection.
Expression Styles: The model showed strong generalization across different expression types (appearance, spatial, motion), particularly when trained with motion cues.

5. Significance and Impact

Paradigm Shift: Moves surgical video analysis from static appearance-based recognition to dynamic, motion-based understanding. This is crucial for handling occlusions and tool ambiguity.
Clinical Application: Enables intuitive, language-driven interaction in the operating room. Surgeons can query systems using natural language (e.g., "highlight the tool retracting the gallbladder") for:
- Augmented Reality (AR) Training: Querying instruments or actions during simulation.
- Intraoperative Assistance: Real-time highlighting of targets and answering verbal queries.
- Robotic Control: Enabling fine-grained semantic control for autonomous surgical robots (e.g., "cut where the grasper is holding").
Benchmarking: Sets a new standard for evaluating language-driven segmentation in surgery through the Ref-IMotion dataset, addressing the lack of motion-centric benchmarks in the field.

In conclusion, SurgRef demonstrates that leveraging instrument motion as a primary signal, combined with a curated motion-centric dataset, significantly enhances the robustness, generalization, and precision of language-driven surgical video segmentation.