Give me scissors: Collision-Free Dual-Arm Surgical Assistive Robot for Instrument Delivery

This paper presents a collision-free dual-arm surgical assistive robot that leverages a vision-language model for zero-shot instruction interpretation and a real-time quadratic programming framework to ensure safe, reactive obstacle avoidance while autonomously delivering instruments with an 83.33% success rate.

The Gist

Imagine a high-stakes game of "catch" happening inside a sterile operating room. Usually, the surgeon is the quarterback, and the scrub nurse is the receiver, constantly running back and forth to hand over scalpels, scissors, and forceps. This is exhausting work for the nurse, and if they get tired or distracted, the whole team's efficiency drops.

This paper introduces a robotic assistant designed to be the ultimate "robotic nurse." Its job is simple: listen to the surgeon, grab the right tool, and hand it over without ever bumping into anything.

Here is how this robot works, broken down into simple concepts:

1. The "Brain": The Robot That Understands Context

Most robots are like toddlers who need you to say, "Pick up the red block, then move two inches left." If you say "Give me the scissors," a normal robot might freeze because it doesn't know what a scissor looks like or where it is.

This robot uses a Vision-Language Model (VLM). Think of this as giving the robot a brain that has read every medical textbook and watched every surgery video.

• The Magic: When the surgeon says, "Give me the scissors," the robot doesn't need a pre-programmed map. It looks at the table, sees the scissors, understands the command, and figures out how to grab them on its own. It's like having a smart assistant who can read your mind and the room simultaneously.

2. The "Eyes": Seeing the Invisible

The operating room is a chaotic place. The surgeon's hands are moving, other tools are on the table, and the robot itself has two long arms that can get tangled.

• The Problem: If the robot moves too fast, it might crash into the surgeon or twist its own arms into a knot (self-collision).
• The Solution: The robot uses a special "distance radar." Instead of just seeing objects, it constantly calculates the exact distance between its metal arms and everything else in the room. It's like driving a car that has sensors constantly whispering, "You are 10 centimeters from that tree... 5 centimeters... 2 centimeters... STOP!"

3. The "Reflexes": The Safety Net (The QP Framework)

This is the most technical part, but think of it as the robot's super-fast reflexes.

• Imagine the robot is trying to walk a tightrope while juggling. It has a goal (hand over the tool), but it also has rules (don't hit the surgeon, don't hit yourself).
• The researchers built a mathematical "safety filter" called a Quadratic Programming (QP) framework.
• The Analogy: Think of the robot's movement plan as a river flowing toward a destination. The obstacles (surgeon, table, other arm) are rocks in the river. The QP framework is the water itself; it naturally flows around the rocks without stopping the river. If a rock suddenly appears (the surgeon moves their hand), the water instantly changes shape to go around it, then flows back to the original path once the rock is gone. This happens in milliseconds, ensuring the robot never jerks or crashes.

4. The Results: Did It Work?

The team tested this robot in a real-world scenario (and simulations) with real surgical tools.

• The Score: It successfully handed over the correct tools 83% of the time.
• The Safety Record: In all the tests, zero collisions occurred. The robot never hit the surgeon, the table, or its own arms.
• The Smoothness: Unlike older robots that might jerk or stop abruptly when avoiding an obstacle, this one moved smoothly, like a dancer dodging a partner.

Why This Matters

Previously, robotic nurses needed a strict "train track" to move on. If the surgeon moved slightly off-track, the robot would get confused or unsafe.

• The Old Way: Like a train on fixed rails.
• This New Robot: Like a self-driving car. It can handle unexpected changes, understand natural language commands, and navigate a messy, moving environment safely.

In a nutshell: This paper presents a robot that acts like a highly skilled, tireless, and incredibly cautious surgical assistant. It listens to the doctor, finds the tools, and hands them over with the grace of a dancer and the safety of a bubble wrap bubble, all without needing a pre-written script.

Technical Summary

1. Problem Statement

In surgical settings, scrub nurses perform repetitive, high-frequency tasks of delivering instruments to surgeons. This leads to physical fatigue, decreased focus, and potential errors, especially in understaffed units. While robotic scrub nurses have been proposed, existing solutions suffer from two major limitations:

• Lack of Generalizability: They rely on predefined paths and instrument categories, making them unable to adapt to dynamic or unstructured operating room environments.
• Safety Risks: They often lack real-time collision avoidance capabilities, posing significant safety hazards in dynamic environments where surgeons and other staff are moving.

The goal is to develop a robotic system that can autonomously understand natural language instructions, identify surgical instruments without prior training (zero-shot), and perform safe, collision-free dual-arm movements in real-time.

2. Methodology

The proposed system integrates a Vision-Language Model (VLM) for high-level task planning with a unified Quadratic Programming (QP) framework for low-level reactive motion control.

A. High-Level Task Planning (VLM)

• Input Processing: The system accepts multi-modal inputs: surgeon voice commands (text), RGB images, and depth data.
• Semantic Understanding: A VLM (specifically GPT-4o) interprets the surgeon's instructions to generate high-level task sub-goals.
• Visual Prompting: To bridge the gap between semantic understanding and geometric execution, the system uses:
  • DINOv2 for pixel-level feature extraction.
  • SAM (Segment Anything Model) to generate object masks.
  • Keypoint Generation: 3D keypoints are calculated for instruments and the surgeon's hand. These are projected as visual markers ($I_p$) to guide the VLM.
• Output: The VLM generates a sequence of sub-goals ($G_{task}$) defining grasp points, release points, and intermediate waypoints, which are converted into desired Cartesian positions via Inverse Kinematics (IK).

B. Real-Time Obstacle Perception

• Environment Modeling: The system uses point clouds from RGB-D cameras.
• Robot Modeling: The robot's complex geometry is approximated by a capsule model to reduce computational cost.
• Distance Prediction: Instead of calculating distances to every point in the point cloud (computationally expensive), a neural network ($\hat{d}_{env}$) predicts the minimum distance between the robot links and environmental obstacles.
• Self-Perception: A similar network ($\hat{d}_{self}$) predicts the minimum distance between the two robotic arms to prevent self-collision.
• Filtering: Points belonging to the robot itself are filtered out using image segmentation masks mapped to depth data to ensure only external obstacles are considered.

C. Reactive Collision Avoidance (Unified QP Framework)

The core safety mechanism is a unified Quadratic Programming (QP) solver that acts as a real-time safety filter. It optimizes joint-space increments ($\Delta q$) to satisfy motion objectives while adhering to constraints.

Objective Function:
$$\min \frac{1}{2}\alpha \| \Delta q - J^\dagger v_{de} \|^2 + \frac{1}{2}\beta \| \Delta q + q_c - q_{de} \|^2 + \Delta q^T Q \Delta q$$

• Term A: Minimizes deviation from the desired Cartesian velocity.
• Term B: Minimizes the gap between current and desired joint configurations.
• Term C: Minimizes joint variation for smoothness.

Constraints:

1. Obstacle Avoidance: $\ln(\frac{\hat{d}_{env}}{\lambda}) + \Delta q \cdot \nabla_q \hat{d}_{env} \geq 0$. This uses a logarithmic barrier function based on the predicted distance to force the robot away from obstacles when the distance falls below a safety threshold ($\lambda$).
2. Self-Collision Avoidance: Similar constraint using $\hat{d}_{self}$ and threshold $\mu$.
3. Joint Limits: Constraints on joint positions and velocities.

3. Key Contributions

1. Zero-Shot Dual-Arm System: A novel robotic scrub nurse that uses VLMs to autonomously plan grasping and delivery paths based on natural language instructions without fine-tuning or predefined paths.
2. Unified Safety Framework: A real-time QP framework that simultaneously handles reactive obstacle avoidance (environment) and self-collision avoidance (dual-arm interaction) using minimum distance perception, eliminating the need for visual markers.
3. Neural Distance Prediction: The integration of a neural network to approximate minimum distances, enabling fast computation suitable for high-frequency control loops (40 Hz).

4. Experimental Results

Simulation Experiments

• Setup: Compared against three State-of-the-Art (SOTA) methods: DawnIK, CollisionIK, and CBF-QP.
• Performance:
  • Success Rate: The proposed method achieved 100% success in avoiding both obstacles and self-collisions, whereas DawnIK failed obstacle avoidance.
  • Efficiency: It had the shortest optimization time (0.022s vs. 0.034s–0.038s for others), demonstrating superior real-time capability.
  • Smoothness: It recorded the lowest maximum acceleration (17.77 $m/s^2$ vs. ~32–39 $m/s^2$ for others), indicating smoother motion trajectories.

Real-World Experiments

• Collision Avoidance: In 10 trials where a human approached the robot, the system successfully avoided collisions and self-collisions without visual markers, outperforming CBF-QP in optimization speed and motion smoothness.
• Instrument Delivery:
  • Task: The robot delivered 4 types of instruments (scalpel, tweezer, scissors, hemostat) in 30 trials (6 combinations $\times$ 5 repetitions).
  • Success Rate: The overall success rate was 83.33%.
  • Safety: Zero collisions occurred in any trial.
  • Failure Analysis: Failures were primarily due to the difficulty of grasping thin, smooth instruments on flat surfaces or VLM misidentification of similar-looking tools (scissors vs. hemostat).

5. Significance and Future Work

• Significance: This work represents a significant step toward autonomous surgical assistance by moving away from rigid, predefined programming to flexible, instruction-driven autonomy. The ability to operate safely in unstructured, dynamic environments without markers makes it highly applicable to real operating rooms.
• Limitations: Current grasping strategies struggle with thin, smooth objects on flat surfaces. The system's performance is also dependent on the accuracy of the VLM's object recognition and keypoint generation.
• Future Directions: The authors plan to use the VLM as a monitor for closed-loop task correction and develop specialized grasping strategies for difficult-to-handle instruments.

Project Resources: The source code and project page are available at https://give-me-scissors.github.io/.