💻 computer science

Imitation Learning for Robot Assistance in Open Surgery: A Multi-Policy Evaluation on Suture Following

This study presents the first multi-policy evaluation of general-purpose imitation learning for surgeon-robot collaborative assistance in open surgery, demonstrating that the $\pi_0$ policy achieves superior robustness and a 92% stitch completion rate while identifying depth perception and end-effector design as critical priorities for clinical translation.

Original authors: Xucheng Wang, Zhizhou Yang, Xiaoman Zhang, Sung Eun Kim, Romain Hardy, Pranav Rajpurkar

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xucheng Wang, Zhizhou Yang, Xiaoman Zhang, Sung Eun Kim, Romain Hardy, Pranav Rajpurkar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where robots aren't just clumsy metal boxes that knock things over, but nimble helpers that can learn by watching us. This is the realm of Imitation Learning, a branch of artificial intelligence where a computer doesn't need to be programmed with every single rule for how to move. Instead, it acts like a super-observant apprentice: you show it what to do a few times, and it tries to copy your moves. Think of it like teaching a dog to fetch; you don't write a manual on how to calculate the dog's jump trajectory, you just throw the ball, and the dog learns the pattern. In the high-stakes world of surgery, this idea is a game-changer. Surgeons are incredibly skilled, but they often need a second pair of hands to hold things steady, pull threads tight, or organize tools while they focus on the delicate work of cutting and stitching. The big question researchers have been asking is: Can a robot learn to be that helpful assistant, not just in a sterile, perfect video game world, but in the messy, real-life chaos of an operating room?

This paper dives right into that question, but with a twist: instead of trying to build a robot that performs the surgery itself (which is incredibly dangerous and complex), the authors asked if a robot could learn the job of the assistant. Specifically, they focused on a task called "suture following." Picture a surgeon stitching up a wound; as they pull the thread tight, the trailing end of the thread dangles around. A human assistant has to constantly grab that loose thread, pull it taut, and hold it out of the way so the surgeon can make the next stitch without the thread getting tangled or loose. It's a repetitive, fiddly job that requires precision but isn't the "main event." The researchers wanted to see if general-purpose AI robots, trained on a small set of demonstrations, could learn to do this specific, helpful dance.

The team set up a "training camp" using an open-source robot arm and a fake piece of skin (a silicone phantom) with a thread running through it. They had a human operator control a "leader" robot arm to demonstrate the grabbing and pulling motion 160 times, creating a dataset of over 32,000 video frames. Then, they pitted four different AI "student" brains against each other to see which one learned the best. These students ranged from a standard transformer model to a massive, pre-trained "vision-language-action" model (a type of AI that has already seen millions of images and actions before this specific training).

The results were a mix of "not bad" and "very promising." Under perfect, controlled conditions, the robots managed to grab the thread successfully between 50% and 75% of the time. The star of the show was the pre-trained model, which succeeded 75% of the time. However, the researchers found a specific weak spot: depth perception. While the robots were good at seeing the thread left or right (lateral movement), they often struggled to judge exactly how far to reach forward. It's like a person who can see a ball clearly but keeps misjudging whether to step forward or backward to catch it. This was the main reason for failure across all models.

The study also tested how well these robots handled "real world" messiness. When the researchers changed the background or removed the perfect blue surgical drape to show a cluttered table, the robots trained from scratch (the ones that hadn't seen the world before) completely fell apart, failing almost every time. But the pre-trained model? It kept working, dropping its success rate only slightly. This suggests that having a "brain" that already understands the world helps a robot adapt when the environment changes.

Finally, the team took their best robot into a live trial with a real surgeon. In this test, the robot didn't have to be perfect; the surgeon could make tiny adjustments if the robot was slightly off. In this more forgiving, real-world scenario, the robot successfully helped complete 23 out of 25 stitches (a 92% success rate). The authors conclude that while the technology isn't ready to replace a human assistant tomorrow, it suggests that a robot can learn to be a helpful partner in open surgery. The biggest hurdles remaining are teaching the robot to judge depth better and designing a gripper that can hold a slippery, thin thread without dropping it. But the door is open: a robot assistant that learns by watching is no longer just a sci-fi dream; it's a feasible reality waiting for a few more refinements.

Technical Summary: Imitation Learning for Robot Assistance in Open Surgery

Problem Statement

Open surgery remains the primary treatment for the majority of surgical conditions globally, yet it relies heavily on a collaborative team structure where a surgical assistant performs repetitive, critical maneuvers such as maintaining suture tension and organizing thread during suturing. Despite the global shortage of trained surgical personnel and the limited availability of skilled assistants, existing robotic surgical systems have predominantly focused on enhancing or automating the primary surgeon's tasks within minimally invasive surgery (MIS) paradigms (e.g., da Vinci). The intra-procedural assistant role in open surgery remains largely unstudied.

This paper addresses the gap in robotic assistance for open surgery by defining suture following as a target task. In this scenario, a robot must act as a surgical assistant: visually locating a trailing suture thread, grasping it, and pulling it taut to maintain appropriate tension while the surgeon drives the needle and places stitches. This task presents specific challenges: the suture thread is thin and low-contrast, requiring sub-centimeter precision and reliable depth reasoning in 3D space, all while operating in a dynamic environment where the thread's position varies.

Methodology

Hardware and Data Collection

The study utilizes a low-cost, open-source robotic platform based on the SO-101 follower arm (part of the LeRobot ecosystem), equipped with a 1-DoF parallel-jaw gripper and driven by six servomotors. Visual input is provided by two commodity USB webcams: a wrist-mounted "on-arm" camera and a fixed side-view camera.

Dataset: The authors collected 160 teleoperated demonstrations (32,374 frames) using a two-person team. One operator teleoperated a "leader" arm to mirror motions, while a second person simulated the surgeon's role (driving the needle and presenting the thread).
Environment: Demonstrations were collected on a silicone tissue phantom with a standard 3/0 silk suture. Training data was gathered in a "clean" background (blue surgical drape), while evaluation included a "randomized" background condition (exposed lab clutter) to test robustness.

Imitation Learning Policies

The study benchmarks four architecturally diverse state-of-the-art policies:

ACT (Action Chunking with Transformers): A conditional VAE with a ResNet-18 backbone, trained from scratch.
Diffusion Policy: A visuomotor policy modeling trajectories via conditional denoising diffusion with a CNN backbone, trained from scratch.
SmolVLA: A lightweight open-source Vision-Language-Action (VLA) model (~450M parameters), fine-tuned with a frozen vision-language backbone.
$\pi_0$ : A large-scale VLA flow-matching model (~3.5B parameters) pretrained on diverse multi-robot data. It was fine-tuned with a frozen PaliGemma vision-language backbone, updating only the action expert head.

Experimental Design

The evaluation was conducted along three clinically motivated axes:

Dataset Size: Training on 20%, 40%, 60%, 80%, and 100% of the 160 episodes to assess data efficiency.
Camera Viewpoint: Comparing dual-camera (full), on-arm only, and side-camera only configurations.
Environmental Robustness: Testing baseline models in randomized background conditions to simulate visual distribution shifts.

A total of 28 models were trained and evaluated across 32 configurations. Success was defined as the robot autonomously locating, grasping, and tensioning the suture thread without human intervention.

Key Results

Baseline Performance

Under ideal conditions (dual camera, 100% data, clean background), the policies achieved task success rates between 50% and 75%:

$\pi_0$ : 75% success (highest).
Diffusion Policy & SmolVLA: 65% success.
ACT: 50% success.

Failure Modes: Across all architectures, depth error (undershooting or overshooting the thread along the approach axis) was the dominant failure mode (20–35% of episodes). Lateral errors were less frequent, and "task breakdowns" (incoherent behavior) were rare in the baseline regime. This indicates that while lateral localization is largely solved, depth estimation for thin, low-contrast objects remains a bottleneck.

Impact of Variables

Dataset Size: $\pi_0$ demonstrated superior data efficiency, reaching 65% success with only 64 episodes (40% of data) and maintaining performance with minimal gains from additional data. In contrast, ACT and Diffusion Policy required significantly more data to reach competitive rates.
Camera Viewpoint: The dual-camera setup was essential. Removing the side camera caused depth errors to spike (40–65%), while removing the on-arm camera increased lateral errors (30–45%).
Environmental Robustness: Background randomization caused severe degradation for policies trained from scratch (ACT dropped to 10%, Diffusion Policy to 10% with 90% task breakdowns). However, the VLA-based policies ( $\pi_0$ and SmolVLA) retained substantial performance ( $\pi_0$ at 50%, SmolVLA at 40%), failing primarily due to depth errors rather than incoherent behavior. This suggests pretrained vision-language backbones provide robustness to visual distribution shifts.

Live In-Loop Trial

In a live trial where $\pi_0$ assisted a surgeon performing running sutures on a phantom, the system achieved a 92% stitch completion rate (23 of 25 stitches). The trial revealed that the strict "millimeter-precise" success criteria of the controlled benchmark were overly conservative for clinical collaboration; the surgeon could naturally compensate for sub-centimeter offsets, making the system clinically usable even when not perfect by lab standards. Failures occurred only when errors were large enough to break the surgical workflow.

Efficiency

$\pi_0$ demonstrated the most suitable execution profile for surgical collaboration, averaging 6.6 seconds per episode with smooth, decisive trajectories. Diffusion Policy was the slowest (27.8s) with high variance and frequent micro-corrections, which is less compatible with the rhythm of live surgery.

Key Contributions and Significance

Feasibility of Collaborative Open Surgery Assistance: This study provides the first empirical evidence that general-purpose imitation learning, when combined with accessible open-source hardware, can support a robot in the surgical assistant role during open surgery. It establishes collaborative open-surgical assistance as a viable application domain for machine learning in healthcare.
Value of Pretrained Vision-Language Backbones: The evaluation demonstrates that policies leveraging pretrained VLA backbones ( $\pi_0$ , SmolVLA) are significantly more data-efficient and robust to visual distribution shifts (e.g., background changes) compared to policies trained from scratch. This suggests that foundation models are particularly well-suited for clinical environments where visual conditions inevitably differ from training data.
Identification of Technical Bottlenecks: The work identifies depth perception for thin, low-contrast objects and end-effector design (specifically for deformable threads) as the primary barriers to clinical translation. The dominance of depth errors across all architectures highlights the need for improved stereo or depth-sensing capabilities and grippers with larger capture envelopes.
Benchmarking and Open Science: The authors release the first systematic benchmark for surgical assistance in open surgery, including the full dataset (160 episodes), training code, evaluation protocols, and model weights, facilitating future research in this under-explored area.

The paper concludes that while current systems are not yet ready for full autonomy in complex biological tissues, the demonstrated ability to perform repetitive assistive tasks with high success rates in a live, collaborative setting marks a significant step toward addressing the global shortage of surgical assistants and augmenting surgical teams.