Learning Surgical Robotic Manipulation with 3D Spatial Priors

1. Problem Statement

Autonomous surgical robotics requires millimeter-level precision to manipulate delicate tissues and instruments. A critical bottleneck is achieving 3D spatial awareness using only the standard stereo endoscopic cameras, which are the primary visual sensors in clinical settings.

Existing approaches face two major limitations:

• Multi-stage Pipelines: Methods that explicitly reconstruct 3D scenes before planning suffer from error accumulation and cannot be optimized end-to-end.
• Hardware Constraints: Recent end-to-end methods that add wrist-mounted cameras to supplement endoscopic views are impractical for clinical use. The trocars (ports) impose strict spatial constraints that prevent additional cameras from passing through, and they can interfere with robot arm motion.

Furthermore, there is a severe scarcity of 3D-annotated data in surgical domains, making it difficult to train robust geometric models directly on real surgical scenes.

2. Methodology: Spatial Surgical Transformer (SST)

The authors propose SST, an end-to-end visuomotor policy that learns 3D spatial priors directly from stereo endoscopic images without requiring additional sensors or explicit 3D reconstruction steps during inference. The framework consists of three core components:

A. Surgical3D Dataset

To address the lack of 3D data, the authors constructed Surgical3D, a large-scale synthetic dataset containing 30,000 photorealistic stereo endoscopic image pairs with precise 3D ground truth (depth maps and point clouds).

• Assets: It combines open-source human organ models with 10 realistic 3D meshes captured from real organs using an iPad scanner.
• Domain Adaptation: To bridge the sim-to-real gap, the authors generated pseudo-labels for unlabeled real surgical data using a model finetuned on synthetic data, creating a hybrid training set that improves robustness.

B. Geometry Transformer (Spatial Prior Extractor)

The policy leverages a Geometry Transformer (based on MASt3R) to extract robust 3D latent embeddings from stereo images.

• Finetuning: The model is first finetuned on the Surgical3D dataset with a 3D reconstruction objective (regressing dense point maps).
• Architecture: It uses a Vision Transformer (ViT) structure where shared encoders process stereo views, and cross-attention mechanisms aggregate spatial relationships.
• Output: Instead of outputting explicit 3D point clouds (which can be noisy), it produces compact 3D latent embeddings that capture geometric cues.

C. Multi-Level Spatial Feature Connector (MSFC) & Policy Decoder

• MSFC: A lightweight module that aggregates latent embeddings from multiple layers of the Geometry Transformer. Lower layers capture fine-grained local details (crucial for grasping), while higher layers capture global context (crucial for motion planning). These are concatenated and aligned with the robot's action space via an MLP.
• Endoscope-Centric Action Space: Since surgical robots (like the da Vinci or Torin) lack precise forward kinematics for their patient-side manipulators (PSMs), the policy predicts relative actions (translation and rotation differences) in the endoscope coordinate frame rather than absolute joint states.
• Training: The system uses an imitation learning framework (Action Chunk Transformer) to predict a sequence of future actions, minimizing Mean Squared Error (MSE) against expert demonstrations.

3. Key Contributions

1. Surgical3D Dataset: The creation of the first large-scale synthetic dataset with 30k stereo image pairs and precise 3D geometric ground truth, addressing the critical data scarcity in surgical robotics.
2. Spatial Surgical Transformer (SST): An end-to-end visuomotor policy that integrates learned 3D spatial priors via a finetuned Geometry Transformer and a Multi-Level Spatial Feature Connector, eliminating the need for wrist cameras or explicit 3D reconstruction pipelines.
3. Endoscopic-Centric Control: A novel action space design that operates entirely within the endoscope frame, accommodating the kinematic uncertainties of surgical robots.
4. Real-Robot Validation: Successful deployment and evaluation on a real surgical robot (Torin) across diverse tasks, demonstrating state-of-the-art performance and strong spatial generalization.

4. Experimental Results

The authors evaluated SST on a real surgical robot across three tasks: Peg Pickup, Knot Tying, and Ex-vivo Gallbladder Dissection.

• Performance: SST achieved State-of-the-Art (SOTA) performance.
  • Peg Pickup: 100% success rate in both training and generalization (Test2) regions.
  • Knot Tying: 70% success rate on the full task (Grasp + Loop), outperforming baselines like SRT (which uses wrist cameras) in the "Loop" subtask despite SST having no wrist cameras.
  • Gallbladder Dissection: 60% success rate on complex real-organ dissection, a task where baseline methods (ACT, Diffusion Policy) failed completely (0% success).
• Spatial Generalization: SST demonstrated superior ability to adapt to unseen 3D geometries (e.g., irregular silicone liver models) and varying depths, whereas baseline methods often failed when objects were placed outside the training distribution.
• Ablation Studies:
  • Finetuning: Removing the finetuning of the Geometry Transformer on Surgical3D caused performance to collapse, proving the necessity of domain-specific 3D priors.
  • Connector Design: The proposed MSFC (Multi-Level) significantly outperformed single-layer (Last-Layer) or separate-layer connectors, confirming the value of fusing fine-grained and global features.
  • Efficiency: MASt3R (used as the backbone) offers a favorable latency (56ms) compared to heavier models like VGGT (140ms), making it suitable for real-time control.

5. Significance

This work represents a significant step toward the clinical deployment of autonomous surgical robots. By proving that high-precision manipulation can be achieved using only standard endoscopic cameras through learned 3D spatial priors, the authors remove the hardware barriers (wrist cameras) that have hindered previous approaches. The framework demonstrates that integrating geometric understanding directly into visuomotor policies is critical for handling the complex, unstructured, and delicate nature of surgical environments. The release of the Surgical3D dataset and code further accelerates research in this field.