🤖 AI

Supervised Mixture-of-Experts for Surgical Grasping and Retraction

This paper introduces a supervised Mixture-of-Experts architecture that enables lightweight imitation learning policies to achieve robust, data-efficient surgical grasping and retraction on deformable tissue using only stereo endoscopic images and fewer than 150 demonstrations, significantly outperforming standard models in both in-distribution and challenging out-of-distribution scenarios while demonstrating successful zero-shot transfer to ex vivo and preliminary in vivo porcine surgeries.

Original authors: Lorenzo Mazza, Ariel Rodriguez, Rayan Younis, Martin Lelis, Ortrun Hellig, Chenpan Li, Sebastian Bodenstedt, Martin Wagner, Stefanie Speidel

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Mazza, Ariel Rodriguez, Rayan Younis, Martin Lelis, Ortrun Hellig, Chenpan Li, Sebastian Bodenstedt, Martin Wagner, Stefanie Speidel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes dance between a human surgeon and a robotic assistant inside a patient's body. The surgeon is the lead dancer, pointing out exactly where to step, while the robot is the partner who must instantly understand the cue, grab a slippery piece of tissue (like a noodle), and hold it steady without dropping it or hurting anything.

This paper presents a new way to teach the robot how to dance this specific dance, even when the robot has never seen that exact move before and has very little practice time.

Here is the breakdown of their approach using simple analogies:

The Problem: The "One-Size-Fits-All" Robot Failed

Usually, when we try to teach robots complex tasks, we use massive "generalist" models. Think of these like a super-smart student who has read every book in the library but has never actually held a scalpel.

The Issue: When you ask this super-smart student to perform a delicate surgery, they get confused. They might try to grab the tissue before they are ready, or they might pull too hard.
The Reality: In this study, these "generalist" models failed completely. They couldn't learn the task even when given standard training data. They were too big, too slow, and too confused by the specific, messy reality of surgery.

The Solution: The "Specialized Team" (Mixture-of-Experts)

Instead of one giant brain trying to do everything, the authors built a Supervised Mixture-of-Experts (MoE) system.

Imagine a small, agile team of specialists rather than one generalist:

The Team: Instead of one robot brain, you have a team of five tiny, specialized experts.
- Expert 1: Only knows how to "Idle" (wait).
- Expert 2: Only knows how to "Approach and Grab."
- Expert 3: Only knows how to "Hold still."
- Expert 4: Only knows how to "Pull back gently."
- Expert 5: Only knows how to "Maintain tension" (keep holding without moving).
The Manager (The Gating Network): There is a smart manager who watches the surgery. When the surgeon points to a spot, the manager instantly wakes up the "Grabber" expert and tells the others to take a nap. When the tissue is grabbed, the manager switches to the "Holder" expert.
The Training: Crucially, the researchers didn't just let the manager guess who to pick. They gave the manager a cheat sheet (labeled data) that said, "Right now, we are in the 'Grabbing' phase." This forced the team to learn their specific jobs perfectly.

The Results: Fast, Light, and Accurate

The team tested this "Specialized Team" against the "Generalist" models and a standard robot policy.

Speed: The specialized team is incredibly fast (27 times per second), which is necessary for real-time surgery. The generalist models were so slow (3 to 10 times per second) that they would be useless in a real operation.
Success Rate:
- The Generalist models: 0% success. They couldn't finish the task.
- The Standard Robot: 50% success. It worked half the time but often dropped the tissue.
- The Specialized Team (MoE): 85% success. It grabbed and held the tissue reliably.
Data Efficiency: The team learned this complex dance with fewer than 150 demonstrations. That's like learning a new dance routine after watching it performed only a few times, whereas other methods usually need thousands of tries.

The "Magic" Tricks: Generalization

The paper claims two major "magic tricks" where the robot succeeded without extra training:

The "New Angle" Trick: The robot was trained looking at the scene from one specific angle. However, when they tested it from completely different angles (zoomed in, zoomed out, tilted), it still worked. It learned the concept of the task, not just the specific picture it saw during training.
The "Zero-Shot" Trick (Ex Vivo): The robot was trained entirely on a fake plastic model (a phantom). When they swapped the plastic for real pig tissue (which is slippery, soft, and moves differently), the robot still succeeded 80% of the time without any new training. It transferred its skills from the fake world to the real world instantly.

The Bottom Line

The authors show that for delicate, multi-step surgical tasks, you don't need a giant, slow, general-purpose AI. Instead, you need a lightweight, specialized team that knows exactly which part of the job it is doing at any given moment.

By using this "Specialized Team" approach, they taught a robot to grab and hold tissue using only a camera view, very little data, and no extra sensors, proving it can work even when the camera angle changes or when moving from a plastic model to real tissue. They have also shown early, successful tests of this system working inside a live pig during surgery, suggesting it is ready for the next steps toward real human use.

Technical Summary: Supervised Mixture-of-Experts for Surgical Grasping and Retraction

Problem Statement
Imitation learning (IL) has advanced robotic manipulation, yet its application to Minimally-Invasive Surgery (MIS) faces critical barriers: data scarcity, constrained workspaces, and the necessity for extreme safety and predictability. While Vision-Language-Action (VLA) foundation models have shown promise in general robotics, they rely on massive datasets and high computational resources, making them ill-suited for surgical domains where demonstration data is limited (often <150 episodes) and inference latency must be low. Furthermore, existing surgical learning approaches often require multi-camera setups (e.g., wrist-mounted cameras) or extensive demonstration counts (e.g., 16,000+ episodes) to achieve reliable performance, which are infeasible in standard laparoscopic settings where only a single endoscopic view is available.

Methodology
The authors propose a Supervised Mixture-of-Experts (MoE) architecture designed to augment lightweight action transformer policies, specifically the Action Chunking Transformer (ACT). The core innovation lies in leveraging the explicit, phase-structured nature of surgical tasks to stabilize training and improve data efficiency.

Architecture: The system extends the standard ACT framework by inserting a Phase-Aware MoE block. This block consists of $H$ $H$ parallel experts (one for each surgical phase: Idle, Approach & Grasp, Hold, Retract, Maintain Tension) and a gating network.
- Experts: Separate "Action Experts" and "Gripper Experts" are trained to model the specific action distributions for each phase, conditioned on latent variables and visual states.
- Gating Network: Unlike unsupervised MoE approaches that often suffer from mode collapse or unstable convergence, this method employs supervised gating. A phase classifier is explicitly trained using ground-truth surgical phase labels to route the current state to the appropriate expert.
Observation Space: The policy is strictly vision-only, utilizing stereo endoscopic image pairs. Proprioceptive data is excluded to ensure the policy is independent of hardware calibration and kinematic noise.
Training Objective: The model optimizes a variational lower bound (ELBO) that includes:
1. Action reconstruction loss ( $L_1$ ).
2. Phase cross-entropy loss (CE) to supervise the gating network.
3. Gripper binary cross-entropy loss (BCE).
4. KL divergence regularization.
Experimental Setup: Experiments were conducted using an OpenHELP phantom with two UR5e robotic arms (one holding a stereo endoscope, the other a grasper). The dataset comprises 120 episodes with fixed viewpoints and 50 episodes with randomized camera angles.

Key Contributions

Novel Architecture: Introduction of a supervised MoE extension for action transformers that enables learning complex, long-horizon surgical tasks from fewer than 150 demonstrations using only endoscopic visual feedback, without multi-view or depth sensors.
Human-Robot Collaboration Task: Definition of a novel collaborative task where a human surgeon indicates a target bowel segment via a laparoscopic instrument, and the robot autonomously executes grasping, retraction, and tension maintenance.
Benchmarking VLAs: Empirical validation that generalist VLA models (SmolVLA, $\pi_0.5$ ) fail to acquire high-precision surgical policies in data-scarce regimes, whereas specialized, lightweight architectures succeed.
Generalization Prerequisites: Demonstration of two critical capabilities for in vivo translation:
- Viewpoint Invariance: Training with randomized camera angles allows the policy to generalize to unseen viewpoints without explicit 3D representations.
- Zero-Shot Transfer: The policy transfers to ex vivo porcine tissue with an 80% success rate without additional training, despite being trained solely on phantom data.

Results

In-Distribution Performance: On the fixed-viewpoint phantom dataset, the proposed ACT + MoE achieved an 85% end-to-end success rate (17/20), a significant improvement over the standard ACT baseline (50%, 10/20) and a complete failure (0%) for VLA baselines. The MoE model also demonstrated superior dexterity, reducing tissue slippage during critical phase transitions.
Out-of-Distribution (OOD) Performance: The MoE policy maintained robustness under reduced illumination, partial occlusions, and novel grasp locations, achieving a 65% end-to-end success rate (13/20) compared to 30% (6/20) for the standard ACT.
Zero-Shot and Viewpoint Generalization:
- Ex Vivo: The policy achieved an 80% success rate (12/15) on ex vivo porcine bowel.
- Random Viewpoints: When trained on a combined dataset including randomized camera angles, the policy achieved an 82% success rate (18/22) on unseen camera viewpoints.
Efficiency: The ACT + MoE policy operates at 27 Hz, comparable to the baseline ACT, whereas VLA models operated at significantly lower frequencies (10 Hz and 3.3 Hz), rendering them unsuitable for real-time surgical control.

Significance and Claims
The paper claims that supervised MoE architectures provide a data-efficient pathway for learning multi-step dexterous manipulation in visually constrained environments. By explicitly modeling the phase structure of surgical tasks, the approach overcomes the instability of unsupervised MoE training and the data hunger of foundation models. The authors assert that this method addresses the specific constraints of MIS—limited data, single-view inputs, and strict safety requirements—offering a viable route toward autonomous surgical assistance. The successful zero-shot transfer to ex vivo tissue and preliminary in vivo roll-outs (shown qualitatively) suggest the system is robust to the geometric and visual domain shifts inherent in dynamic clinical environments. The work concludes that combining imitation learning with supervised MoE allows for the automation of multi-step MIS tasks using only endoscopic vision, moving closer to the goal of in vivo deployment.