💻 computer science

Towards Safer Robot-Assisted Surgery: A Markerless Augmented Reality Framework

This paper proposes a markerless augmented reality framework that integrates advanced stereo reconstruction and segmentation to detect minimum distances between surgical instruments and blood vessels, thereby enhancing safety during robot-assisted surgery by preventing collisions without adding extra load to the surgeon.

Original authors: Ziyang Chen, Laura Cruciani, Ke Fan, Matteo Fontana, Elena Lievore, Ottavio De Cobelli, Gennaro Musi, Giancarlo Ferrigno, Elena De Momi

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyang Chen, Laura Cruciani, Ke Fan, Matteo Fontana, Elena Lievore, Ottavio De Cobelli, Gennaro Musi, Giancarlo Ferrigno, Elena De Momi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes video game where you control a tiny, super-precise robot arm to perform delicate surgery. The goal is to remove a specific target without accidentally hitting a fragile, invisible wire running right next to it. In the real world, this is exactly what surgeons face during robot-assisted operations. They have powerful tools, but they often lack a clear view of what lies just beneath the surface of the tissue they are working on. To fix this, scientists are exploring "Augmented Reality" (AR). Think of AR like the "heads-up display" in a fighter jet or a video game: it takes a 3D map of the patient's body (created from scans taken before the surgery) and projects it directly onto the surgeon's screen, overlaying the real world with helpful digital information. The big challenge, however, is making sure that digital map lines up perfectly with the squishy, moving organs inside the patient without needing to stick physical markers or stickers on the body. If the map is even slightly off, it could be dangerous. This is the puzzle a team of researchers set out to solve: how to create a "ghostly" guide that sticks perfectly to the real anatomy, warns the surgeon if they are getting too close to a blood vessel, and does it all without slowing the robot down.

The researchers behind this study, working with the da Vinci Research Kit (a robotic platform used for testing surgical ideas), proposed a "markerless" framework. This means their system doesn't need any physical tags or extra cameras to know where the blood vessels are. Instead, it acts like a super-smart pair of eyes that watches the live video feed from the robot's camera. The system uses two powerful AI "brains" working together. The first brain is a "stereo reconstruction" network that looks at the 3D video feed and figures out the depth of everything it sees, essentially turning the flat video into a 3D point cloud. The second brain is a "segmentation" network that acts like a digital highlighter, instantly identifying and outlining the blood vessels in that 3D space.

Once the system knows exactly where the blood vessels are in 3D space, it performs a magic trick called "registration." It takes the pre-operative 3D model (the map made from the patient's scans) and snaps it perfectly onto the live 3D view of the blood vessels. Now, the surgeon sees the real blood vessels with a digital overlay of the pre-operative model, giving them a complete picture. But the system doesn't just show a picture; it acts as a safety guard. It constantly measures the distance between the robot's surgical tools and the blood vessels. If the tool gets too close, the system warns the surgeon. In their experiment, they simulated a lymph node removal surgery (a common procedure where lymph nodes are removed near blood vessels) using a dry lab setup with 3D-printed organs. They asked ten human volunteers to perform the task twice: once with the standard view and once with their new AR safety system.

The results were promising. The study found that when the volunteers used the AR system, they kept a safer distance from the blood vessels. Specifically, the minimum distance between the tool and the vessel increased from an average of 0.0472 cm (in the standard setup) to 0.1864 cm (with AR). More importantly, the number of "collisions" or near-misses where the tool got dangerously close (within 0.5 cm) dropped significantly, from an average of 23.8 times down to 13.7 times. The researchers also checked if the system made the surgery harder or slower. They found that the total time it took to finish the task and the total distance the robot arms moved were almost the same in both groups, meaning the AR system didn't add any extra burden or slow the surgeons down. The volunteers also rated the AR system as more user-friendly and easier to use in a survey.

The paper explicitly rules out the idea that manual alignment or pre-installed markers are necessary for this kind of safety system, showing that a fully automated, vision-based approach works just as well, if not better. However, the authors are careful to note that these results come from a "dry lab" simulation using 3D-printed models, not from live human surgery. They also point out a limitation: their system currently models the surgical tools as simple cylinders, which might not be perfectly accurate for every type of complex surgical instrument. While the system showed it could enhance safety in this simulated environment, the authors suggest that future work will need to test this in more complex, real-world clinical settings and perhaps combine the visual warnings with physical "force feedback" (where the robot pushes back if you get too close). For now, this research suggests that a smart, markerless AR system could be a valuable tool to help surgeons avoid accidental bleeding, making robot-assisted surgery safer without adding extra work.

Technical Summary: Towards Safer Robot-Assisted Surgery: A Markerless Augmented Reality Framework

Problem Statement
Robot-assisted surgery (RAS) offers improved patient outcomes but faces significant safety challenges, particularly intra-operative bleeding caused by the accidental collision of surgical instruments with delicate blood vessels. While Augmented Reality (AR) has the potential to enhance surgeon performance by overlaying pre-operative models (derived from CT/MRI) onto intra-operative views, current implementations suffer from critical limitations. Existing methods often rely on manual alignment, pre-installed markers, or external hardware (e.g., ultrasound, projectors), which hinder real-time application and generalization. Furthermore, accurately localizing soft tissues in 3D space to register pre-operative models without external aids remains a complex challenge.

Methodology
The authors propose a markerless augmented reality framework integrated into the da Vinci Research Kit (dVRK). The system operates without external markers or devices, relying solely on stereo endoscopic video and advanced neural networks. The methodology consists of four primary stages:

System Calibration: The framework utilizes the dVRK kinematics and performs three calibration steps:
- Camera Calibration: Using Zhang's method to determine intrinsic and extrinsic parameters for image rectification.
- Hand-Hand Calibration: Establishing the rigid transformation between the left and right end effectors using Horn's method.
- Hand-Eye Calibration: Determining the transformation between the left end effector and the left camera using RANSAC.
Intra-operative Reconstruction and Segmentation:
- Stereo Reconstruction: The stereo image pair is rectified and resized (1920×1080 to 640×360). A stereo reconstruction network estimates a disparity map, which is converted to a 3D point cloud using the camera baseline and focal length.
- Segmentation: A segmentation network processes the left rectified image to generate a binary mask distinguishing the blood vessel from the background.
- 3D Generation: The disparity pixels corresponding to the segmented blood vessel are reprojected into 3D space to reconstruct the intra-operative blood vessel point cloud.
- Model Selection: The authors evaluated 14 stereo reconstruction networks and 8 segmentation networks. HSM [14] was selected for depth estimation due to its balance of accuracy and inference speed, while UNet [23] was chosen for segmentation based on its reliable performance in the specific phantom environment.
Registration:
- A pre-operative 3D model (created in Blender for this study) is registered to the reconstructed intra-operative point cloud.
- The process employs a global registration using RANSAC for initial alignment, followed by local registration using the Iterative Closest Point (ICP) algorithm for fine-tuning.
Distance Detection and Visualization:
- The system calculates the minimum distance between the surgical instruments (modeled as cylinders based on kinematics) and the reconstructed blood vessel using a fast k-nearest-neighbor search.
- AR Visualization: The pre-operative model is overlaid onto the endoscopic images.
- Safety Feedback: Two gauges display the minimum distance for left and right instruments. The color of the overlaid model changes dynamically based on the smaller distance (e.g., turning red for high-risk proximity) to alert the surgeon.

Key Contributions

Markerless AR Framework: A novel system that visualizes pre-operative models and detects minimum distances between instruments and blood vessels without relying on manual alignment, pre-installed markers, or external hardware.
Comprehensive Network Evaluation: A systematic comparison of state-of-the-art stereo reconstruction and segmentation networks to identify the optimal combination for accurate and fast 3D blood vessel recovery in medical scenes.
Human Subject Usability Study: A validation study involving ten human subjects performing a simulated robot-assisted lymphadenectomy on the dVRK platform, comparing standard endoscopic views against the proposed AR-assisted views.

Results

Framework Performance: The complete pipeline processes frames at approximately 6.91 FPS (0.1448s per frame), providing smooth visual feedback. Calibration errors were minimal (e.g., registration error $E_{regis} \approx 0.43$ cm).
Usability Study Findings:
- Safety Improvement: With AR assistance, the minimum distance ( $D_{min}$ ) between instruments and vessels increased significantly from 0.0472 cm to 0.1864 cm ( $p=0.0098$ ).
- Collision Reduction: The number of collisions ( $N_c$ , defined as distance < 0.5 cm for >1s) decreased from 23.8 to 13.7 ( $p=0.0077$ ).
- No Extra Load: There were no statistically significant differences in the overall movement path ($Sp$) or execution time ( $T_{exe}$ ), indicating the framework does not introduce an extra physical or cognitive burden.
- User Satisfaction: The System Usability Scale (SUS) score improved from 66 (Control) to 73 (Experiment), with a significant difference ( $p=0.0104$ ).

Significance and Claims
The paper claims that the proposed framework successfully enhances surgical safety by providing real-time visual feedback on instrument-to-vessel proximity, effectively reducing collision risks without compromising operational efficiency. The authors emphasize that the system's independence from specific devices or external hardware makes it highly generalizable to other robotic platforms and surgical tasks.

The study acknowledges limitations, including the simplified cylindrical modeling of diverse surgical instruments and the use of a phantom (dry lab) environment rather than complex, dynamic in vivo patient tissues. However, the authors posit that the framework offers a flexible foundation for integrating AR into medical robotics to prevent intra-operative bleeding, a high-risk complication. Future work is suggested to explore virtual fixtures (force feedback) to complement the current visual feedback mechanisms.

Technical Summary: Towards Safer Robot-Assisted Surgery: A Markerless Augmented Reality Framework

More like this