💻 computer science

Surgical WAM: A World-Action Model for Data-Efficient Surgical Robot Learning

The paper introduces Surgical WAM, a generative model that leverages abundant action-free endoscopic video to learn visual dynamics priors, significantly improving data efficiency and closed-loop control performance in surgical robot tasks when fine-tuned on limited action-labeled demonstrations.

Original authors: Wenrui Bao, Tianyun Jiang, Zhiben Chen, Ser-Nam Lim, Peter D. Peng, Yuzhang Shang

Published 2026-08-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenrui Bao, Tianyun Jiang, Zhiben Chen, Ser-Nam Lim, Peter D. Peng, Yuzhang Shang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to perform delicate surgery, like threading a needle or tying a knot inside a human body. This isn't just about moving a mechanical arm; it's about understanding how soft, squishy tissue reacts when touched, how tools slide against each other, and how to coordinate two hands working in perfect sync. In the world of robotics, this is the "Holy Grail" of automation: making machines that can do complex, life-saving tasks with the steady precision of a master surgeon.

To teach a robot these skills, scientists usually need "demonstrations." Think of these as training videos where a human expert performs the task while the robot's computer records every tiny movement of its arms and hands. However, collecting these specific, synchronized videos is incredibly hard, expensive, and slow. It requires special surgical robots, expert surgeons, and sterile operating rooms. As a result, robots often have very few examples to learn from, making them clumsy or unsafe. On the other hand, there is a mountain of "action-free" video available—thousands of hours of surgical footage recorded from cameras inside the body, but without the data on exactly how the robot's arms moved. The big question scientists have been asking is: Can we teach a robot to understand the visual world of surgery using all that cheap video, and then use that knowledge to learn the actual movements with very few examples?

This paper introduces a clever new method called Surgical WAM (World-Action Model) to answer that question. The researchers treat the robot's learning process like a two-step cooking recipe. First, they feed the robot a massive "soup" of action-free surgical videos. During this stage, the robot doesn't learn how to move its arms; instead, it learns to predict what the next frame of the video will look like. It learns the "physics" of the scene: how a needle bends, how tissue stretches, and how light reflects off wet surfaces. It's like a student watching hours of cooking shows to understand how ingredients behave, without ever touching a stove.

Once the robot has built this strong mental model of the surgical world, the second step begins. The researchers give it a small, fixed amount of "action-labeled" data—the expensive, synchronized videos of actual robot movements. Because the robot already understands the visual dynamics from the first step, it learns the actual movements much faster and more accurately. The model acts as a "closed-loop" controller, meaning it constantly predicts the future, checks if its prediction matches reality, and adjusts its plan in real-time, just like a human surgeon who watches their hand and corrects their grip instantly.

The results from their experiments in a simulated surgical environment are promising. When they tested the robot on four different tasks, the version that used the video pre-training succeeded 77.8% of the time, compared to only 63.5% for the version that skipped the video training. The improvement was even more dramatic on the hardest tasks, like transferring a peg between two points, where the success rate jumped by 20 percentage points. The paper suggests that this method allows robots to learn effectively even when the "expensive" movement data is scarce. However, it is important to note that these results were achieved in a simulation, not on real human patients or physical robots in an operating room yet. The authors also tested their method on real-world video recordings from a public dataset (JIGSAWS) and found the same trend held true, suggesting the robot learned genuine surgical dynamics rather than just memorizing a video game.

The paper explicitly argues against the idea that robots need millions of expensive, synchronized movement recordings to learn. Instead, it suggests that the bottleneck isn't the lack of movement data, but the lack of a way to use the abundant video data we already have. By separating the learning of "what the world looks like" from "how to move," the researchers show that we can get much better results with the same amount of expensive data. They also found that this approach is most helpful for tasks that involve a lot of physical contact or require two hands working together, which are the exact kinds of tasks that are usually the hardest for robots to master.

In short, this research proposes that if we want robots to become expert surgeons, we shouldn't just force them to watch a few expensive training tapes. Instead, we should let them binge-watch thousands of hours of surgery videos to understand the world, and then give them a small, focused lesson on how to move. This approach, the authors suggest, could be the key to scaling up surgical robot learning without needing to hire thousands of extra surgeons to record data.

Technical Summary: Surgical WAM

Problem Statement

Learning reliable surgical manipulation policies is currently bottlenecked by the scarcity of action-labeled demonstrations. Collecting teleoperated surgical robot trajectories (e.g., dVRK) with synchronized kinematic labels requires specialized hardware, expert operators, and costly setup, resulting in datasets with only hundreds to a few thousand demonstrations. In contrast, endoscopic video is abundant and inexpensive to collect but lacks action labels. While existing surgical world models utilize video for simulation or policy evaluation, they rarely translate learned dynamics into closed-loop control. The central question addressed is: Under a fixed budget of action-labeled demonstrations, does action-free video pretraining improve closed-loop surgical manipulation?

Methodology

The authors propose Surgical WAM (World-Action Model), a unified generative model based on the Cosmos Policy architecture. The model jointly predicts future endoscopic observations and executable surgical robot action chunks within a single latent sequence.

Architecture

Model Class: A World-Action Model (WAM) that learns the joint distribution $p_\theta(o_{t+1:t+K}, a_{t:t+H_c} | o_{\le t}, s_{\le t}, c)$ , where $o$ represents visual observations, $s$ represents proprioceptive states, and $c$ is a task embedding.
Backbone: The model utilizes a diffusion-transformer where visual tokens, robot-state tokens, and action tokens share a unified sequence processed by a single network. This allows weights learned from action-free video to directly transfer to action prediction.

Two-Stage Training Recipe

The training procedure decouples visual dynamics learning from action supervision to leverage data asymmetry:

Stage 1: Action-Free Video Pretraining: The world-model component is trained on large-scale, unlabeled surgical video corpora. The objective is to denoise future video latents ( $L_{vid}$ ), while action and state slots are masked. This stage learns general spatiotemporal dynamics and visual representations without requiring kinematic labels.
Stage 2: Joint Fine-Tuning: The pretrained model is fine-tuned on a fixed budget of action-labeled surgical demonstrations ( $D_{va}$ ). The objective ( $L_{joint}$ ) jointly denoises both the future video and the action chunks. This grounds the learned visual dynamics in executable commands.

Closed-Loop Execution

At deployment, Surgical WAM operates as a receding-horizon controller:

The model receives the current endoscopic image and robot proprioception.
It jointly samples a future video sequence and an action chunk (length $H_c$ ).
The robot executes a short prefix of the predicted actions ( $H_e$ ).
The system re-observes the environment and replans, allowing for continuous correction of predictions and actions.

Key Contributions

First Surgical WAM: The paper introduces the first world-action model applied to surgical robot learning, unifying future video prediction and action generation in a single architecture to drive closed-loop execution.
Validation of Video Pretraining: The authors demonstrate that action-free video pretraining significantly improves closed-loop surgical manipulation under a fixed action-label budget. The pretrained model achieves higher success rates with fewer fine-tuning steps compared to training from scratch on action labels alone.
Characterization of Benefits: Ablation studies reveal that the benefits of video pretraining are most pronounced in contact-rich and bimanual tasks. The pretrained initialization converges faster (peaking with half the optimization steps) and remains robust across different execution horizons.

Experimental Results

The method was evaluated on the SurRoL benchmark, a suite of four simulated dVRK manipulation tasks: Needle Pick, Peg Transfer, Needle Regrasp, and BiPeg Transfer.

Performance Gains: On the full task suite, video pretraining improved the average closed-loop success rate from 63.5% (without pretraining) to 77.8%.
Task-Specific Improvements: The most significant gain was observed on the Peg Transfer task, with an absolute increase of 20 percentage points (from 66% to 86%).
Comparison to Baselines: Surgical WAM with pretraining outperformed generalist Vision-Language-Action (VLA) models (e.g., $\pi_0.5$ , ALOHA, Diffusion Policy), which struggled to transfer to the surgical domain, achieving only ~22% average success.
Efficiency: The pretrained model reached its peak performance (86%) in 80k fine-tuning steps, whereas the non-pretrained model required 160k steps to reach a lower peak (79%).
Real-World Validation: Experiments on real dVRK video from the JIGSAWS dataset confirmed that the performance gains are not artifacts of simulation but persist under real-world visual complexity.

Significance and Claims

The paper positions data-efficient, action-free video pretraining as a practical path toward scaling surgical robot learning. By shifting the bottleneck from collecting expensive teleoperated demonstrations to curating the abundant surgical video already produced at scale, the approach enables the learning of robust manipulation policies with limited action supervision. The authors claim that this method effectively transfers visual dynamics priors to control, particularly for tasks requiring precise contact handling and bimanual coordination, without increasing the cost of action labeling.