Embodied AI in the Operating Room: A Voice-Interactive Robotic Scrub Technician for Surgical Instrument Handoffs

Technical Summary: Embodied AI in the Operating Room

Problem Statement
Current surgical robotics largely rely on direct human teleoperation, limiting their ability to perceive, reason, and interact autonomously within dynamic operating room (OR) environments. While the scrub technician role is critical for maintaining operative flow through coordinated instrument exchange, it requires complex visual recognition, contextual understanding, and real-time interaction that existing automated systems struggle to replicate. Previous robotic scrub systems have primarily focused on one-way instrument delivery using rule-based vision or fixed layouts, lacking the ability to perform bidirectional exchanges (handoff and take-back) or recover from interference without full task reinitialization.

Methodology
The authors developed a voice-interactive robotic scrub technician platform designed to execute surgical instrument handoffs and take-backs using a Vision-Language-Action (VLA) model.

• System Architecture: The system utilizes a bimanual robotic platform integrated with real-time multicamera observations. The core learning architecture is based on the pretrained π0.5 model (developed by Physical Intelligence), which was fine-tuned end-to-end on a custom dataset.
• Data Collection: The model was trained via imitation learning using 350 human-teleoperated demonstrations. These episodes captured synchronized video observations, surgeon voice commands, and robot joint actions for instrument handoff and take-back tasks.
• Speech Interface: To enable natural interaction, the system integrates Whisper large-v2 for automatic speech recognition (ASR) and Cartesia API for text-to-speech (TTS) synthesis. This allows surgeons to issue verbal commands (e.g., "hand me the forceps") and receive audible acknowledgments from the robot.
• Workflow: The autonomous pipeline consists of four stages:
  1. Command Interpretation: Transcribing speech to text.
  2. Instrument Localization: Identifying the requested instrument on a variable tray configuration.
  3. Robotic Grasping: Acquiring a secure grasp.
  4. Handoff/Take-back: Transporting the instrument to the surgeon's hand or returning it to the tray.
• Evaluation Protocol: The system was tested in a simulated OR environment across 100 autonomous trials involving five tasks: needle-driver handoff, forceps handoff, needle-and-suture handoff, needle-driver take-back, and forceps take-back. Each task included 10 trials with "seen" tray configurations (present in training) and 10 with "held-out" configurations (unseen during training).

Key Results

• Overall Performance: The system achieved an 81% end-to-end success rate (81/100 trials) with a mean completion time of 7.1 ± 1.4 seconds for successful trials.
• Task-Specific Success:
  • Take-back tasks: Highly reliable, with 95% success for both needle-driver and forceps take-backs.
  • Handoff tasks: Forceps handoff succeeded in 85% of trials; needle-driver handoff in 80%; needle-and-suture handoff in 50%.
• Generalization: Performance dropped when tested on held-out tray configurations, decreasing from 92% success on seen layouts to 70% on held-out layouts. The decline was most pronounced in handoff tasks (dropping from 86.7% to 56.7%), while take-back performance remained relatively stable (100% to 90%).
• Stage-wise Analysis: Speech command interpretation was successful in 99% of trials. Once a secure grasp was achieved, downstream manipulation (transport, orientation, release) succeeded in 81 of 82 trials, indicating that the primary failure point is visual localization and grasp acquisition rather than trajectory execution.
• Failure Modes: The predominant failure mode was missed grasp (57.9% of failures), followed by policy timeouts (21.1%) and incorrect instrument selection (10.5%). No failures involved unstable grasps, poor final orientation, or safety stops requiring emergency termination.

Key Contributions

1. Unified Bidirectional Policy: Unlike prior systems focused solely on delivery, this work demonstrates a single learned policy capable of both delivering instruments and retrieving them (take-back), reflecting the full loop of a scrub technician's duties.
2. Voice-Interactive Embodied AI: The integration of ASR and TTS creates a multimodal communication loop where the robot acknowledges commands verbally, making its intent observable to the surgical team.
3. End-to-End Fine-Tuning: The study utilizes the π0.5 foundation model fine-tuned directly on teleoperated data, avoiding the need for separate, manually annotated detection or segmentation modules required by previous approaches (e.g., RoboNurse-VLA).
4. Recovery Capabilities: The system demonstrated the ability to recover from interference and execution failures without complete task reinitialization, adapting its actions based on updated visual observations.

Significance and Claims
The authors position this work as a proof-of-concept demonstrating that embodied AI can perform voice-conditioned, bidirectional surgical instrument exchange. The study claims that the primary limitation of current systems lies in grasp acquisition (particularly for small, complex targets like needles with sutures) and generalization to unseen spatial configurations, rather than in the downstream execution of trajectories once a grasp is secured.

The paper asserts that this platform represents an early step toward co-adaptive human-machine partnerships in the OR. It emphasizes that the technology is intended to be assistive rather than substitutive, aiming to reduce workload and improve coordination for repetitive exchanges while human personnel retain responsibility for sterility, clinical judgment, and managing unexpected events. The authors conclude that while the system shows meaningful generalization, clinical translation requires expanded training data, realistic workflow evaluation, and rigorous safety validation before integration into complete back-table management.