SurfSurg6D: Geometry Consistent Dense Correspondence for Textureless Surgical Instrument Pose Estimation

Technical Summary: SurfSurg6D: Geometry Consistent Dense Correspondence for Textureless Surgical Instrument Pose Estimation

Problem Statement
Surgical instrument pose estimation is critical for autonomous robotic surgery, skill assessment, and workflow standardization. However, the task faces significant challenges in Minimally Invasive Surgery (RMIS) environments: instruments are often textureless and reflective, frequently occluded by tissue, and lack reliable depth sensors or kinematic ground truth due to cable-driven mechanisms. Existing general object pose estimation methods struggle in this domain due to data scarcity, the "domain gap" between synthetic and real images, and the inability of template-based or regression-based approaches to handle the lack of visual features and frequent occlusions.

Methodology
The authors propose a two-pronged approach: the creation of a large-scale synthetic dataset and a novel dense correspondence framework.

1. SynSurg6D Dataset Generation:
   To address data shortage, the authors constructed SynSurg6D, a synthetic dataset containing over 60,000 rendered images of six common surgical instruments (e.g., Needle Driver, Scissors, Forceps). The generation pipeline involves:

   • Scene Reconstruction: Using EndoSurf to reconstruct 3D mesh models of endoscopic environments from the EndoNeRF dataset to serve as realistic backgrounds.
   • Physics-Constrained Sampling: Instruments are positioned using a Remote-Center-of-Motion (RCM) constraint, mimicking clinical trocar insertion. The pipeline samples insertion depth, axial roll, and wrist bending angles within kinematically plausible ranges.
   • Validation: Collision detection (mesh penetration) and visibility constraints (rejecting samples where the instrument is too occluded or outside the field of view) ensure physical plausibility.
   • Rendering: Randomized lighting and background variations are applied to narrow the sim-to-real gap. The dataset provides RGB, depth, segmentation, normal maps, and 6-DoF pose annotations.

2. SurfSurg6D Framework:
   Building upon the dense correspondence paradigm (similar to Surfemb), the framework estimates the 6-DoF pose by learning 2D-3D correspondences between image pixels and instrument surface points. It consists of an Instrument Latent Field (encoding 3D surface points) and an Image Recognition Network (predicting per-pixel embeddings). The authors introduce two key mechanisms to improve performance on textureless objects:

   • Distance-Aware Hard Negative Mining: In contrastive learning, the authors observe that standard negative pairs are often trivial. They introduce a distance-based penalization strategy that up-weights "hard" negative pairs—those where non-matching surface points are spatially close to the query point. This forces the model to distinguish between locally similar but geometrically distinct fragments, reducing confusion on textureless surfaces.
   • Geometric-Embedding Consistency Loss: To prevent the embedding manifold from becoming fragmented (where nearby 3D points map to distant latent vectors), a consistency loss is added. This loss enforces that the distance between embeddings correlates with the geometric distance between surface points, ensuring a smooth, continuous embedding field. This improves robustness against slight prediction errors during the RANSAC-PnP pose solving stage.

Key Contributions

• SynSurg6D: A large-scale synthetic dataset with diverse modalities and accurate 6-DoF annotations, designed to diversify pose distributions and alleviate data scarcity in surgical robotics.
• SurfSurg6D: A dense correspondence framework that integrates hard negative mining and embedding consistency regularization. These mechanisms specifically target the challenges of textureless surfaces and self-occlusion.
• Comprehensive Evaluation: Extensive experiments demonstrating that the proposed method outperforms state-of-the-art (SOTA) regression, template-based, and correspondence-based methods.

Experimental Results
The method was evaluated on three benchmarks: SurgRIPE, EndoVis2018, and SurgPose.

• SurgRIPE (6-DoF Pose): SurfSurg6D achieved the best performance among all methods. In the "syn+real" training setting, it reached an ADD-10 accuracy of 47.46% and an average accuracy (0-5mm) of 60.12%, surpassing the baseline Surfemb (45.01% and 58.97%, respectively). The results indicate that the generated SynSurg6D dataset consistently improves the performance of existing methods, not just the proposed framework.
• EndoVis2018 & SurgPose (2D Projection): The method demonstrated strong generalization to real clinical scenes, achieving a Dice score of 0.7952 on EndoVis2018 and a mean Average Precision (mAP) of 0.6897 on SurgPose, outperforming other SOTA approaches.
• Ablation Studies: Results confirmed that hard negative mining primarily improves translation accuracy by reducing erroneous correspondences near edges, while consistency regularization improves rotational accuracy by ensuring geometric smoothness in the embedding space.
• Foundation Models: Zero-shot inference using foundation models (MegaPose, FoundPose) yielded poor results, highlighting the difficulty of applying general models to textureless, reflective surgical instruments without domain-specific data.

Significance and Claims
The paper claims that the scarcity of high-quality annotated data is the primary bottleneck for surgical instrument pose estimation. By introducing SynSurg6D, the authors provide a resource that diversifies pose distributions and enhances the robustness of data-driven approaches. The SurfSurg6D framework offers a robust solution for precise, RGB-only pose estimation in challenging surgical environments. The authors emphasize that their approach effectively handles textureless objects and significant occlusions, achieving competitive inference speeds (1.25 FPS) suitable for practical RMIS applications, provided that bounding box detection is handled separately. The work suggests that future improvements could involve tracking articulated configurations and integrating temporal tracking modules to further increase effective inference speed for video streams.