PinPoint: Monocular Needle Pose Estimation for Robotic Suturing via Stein Variational Newton and Geometric Residuals

1. Problem Statement

The paper addresses the challenge of real-time 3D needle pose estimation for autonomous robotic suturing in monocular endoscopic settings (e.g., transendoscopic and intraluminal procedures).

• The Core Challenge: Unlike stereoscopic systems, monocular vision lacks depth cues. Curved needles possess rotational symmetry, leading to geometric ambiguity where multiple 3D configurations produce identical 2D image projections.
• The Failure of Existing Methods:
  • Deterministic Estimators: Fail to represent the inherent multimodal nature of the problem, often collapsing to a single, potentially incorrect hypothesis.
  • Particle Filters (PF): While probabilistic, they suffer from "weight collapse" under tight geometric constraints and high-dimensional state spaces, losing hypothesis diversity and failing to maintain multimodal distributions.
  • Learning-Based Methods: Often lack integration with robot kinematics and do not explicitly model the multimodal posterior structure.

2. Methodology: The PinPoint Framework

PinPoint is a probabilistic variational inference framework that estimates the needle pose as a distribution over $SE(3)$ rather than a single point estimate. It combines image observations with robot-grasp constraints using Stein Variational Newton (SVN).

A. Likelihood Formulation (Geometric Residuals)

The system minimizes a Negative Log-Likelihood (NLL) objective composed of three analytical terms with closed-form Jacobians:

1. Sparse Keypoint Likelihood ($L_{sparse}$): Uses 2D detections of the needle tip and tail to constrain image-plane position and orientation via reprojection error.
2. Dense Backbone Likelihood ($L_{dense}$): Uses a segmentation mask of the needle backbone. It models the needle as a 3D circular arc and minimizes the Sampson distance between the projected conic and the segmented pixels, providing global shape constraints even under partial occlusion.
3. Robot Kinematics Likelihood ($L_{grasp}$): Leverages the C-Lock grasper (a rigid needle holder for Concentric Tube Robots).
   • Position: Fixes the needle at a known point relative to the robot, anchoring the depth (z-axis) which is otherwise unobservable.
   • Orientation: Enforces a perpendicularity constraint between the gripper axis and the needle tangent, reducing rotational ambiguity.

B. Inference Engine: Stein Variational Newton (SVN)

Instead of stochastic sampling (like Particle Filters), PinPoint uses SVN to deterministically transport a set of particles toward high-probability regions while preserving diversity.

• Particle Transport: Particles are updated in the tangent space of $SE(3)$ using a velocity field derived from the gradient of the log-posterior.
• Gauss-Newton Preconditioning: A key innovation. The update step is preconditioned by the Gauss-Newton Hessian approximation ($H = J^T J$). This adapts the step size to the local curvature of the problem, allowing efficient convergence even in ill-conditioned (ambiguous) scenarios.
• Kernel-Based Repulsion: A kernel function prevents particles from collapsing into a single mode, ensuring the representation of multimodal distributions (e.g., maintaining two distinct hypotheses for the needle's orientation).
• Whitened Kernel: Distances are computed in a whitened tangent space to handle the scale disparity between translational (mm) and rotational (radians) degrees of freedom.

3. Key Contributions

1. Multimodal Inference for Monocular Vision: PinPoint is the first framework to explicitly treat monocular needle pose estimation as a multimodal inference problem, preserving alternative hypotheses rather than forcing a premature single estimate.
2. Hybrid Geometric-Robot Constraints: It uniquely fuses monocular image geometry with rigid robot-grasp constraints (C-Lock) to ground the 3D pose, effectively solving the depth ambiguity without stereo vision.
3. Second-Order Variational Inference: The application of Stein Variational Newton with Gauss-Newton preconditioning allows for rapid, deterministic convergence and better uncertainty calibration compared to first-order methods (SVGD) or stochastic filters (PF).
4. Robustness to Occlusion: The framework maintains stable tracking during partial and heavy occlusions (e.g., when the needle is embedded in tissue) by relying on the robot kinematic constraints when visual data is lost.

4. Experimental Results

The system was evaluated on a Virtuoso Endoscopy System using a 4-0 suture needle, comparing PinPoint against a Particle Filter (PF) baseline and ablation variants.

• Accuracy Improvements:

  • Translational Error: Reduced by 80% compared to PF (down to 1.00 mm).
  • Rotational Error: Reduced by 78% compared to PF (down to 13.80°).
  • Performance remained stable across slow and normal-speed motions, whereas PF errors degraded significantly with speed.

• Multimodal Performance:

  • On sequences with induced out-of-plane rotations (high ambiguity), PinPoint maintained a bimodal posterior 84% of the time, compared to only 28.9% for the PF baseline.
  • In 100% of bimodal cases, the ground truth fell within the dominant mode of PinPoint's distribution, whereas PF only achieved this 60% of the time.

• Uncertainty Calibration:

  • PinPoint demonstrated significantly better-calibrated uncertainty (lower NLL-S and Interval Scores), accurately reflecting the true error distribution.
  • The system correctly identified high uncertainty in the depth (z-axis) and out-of-plane rotation directions, while maintaining high precision in well-constrained axes.

• Ablation Studies:

  • Removing image data caused the largest error increase, confirming the necessity of visual geometry.
  • Removing robot constraints caused a significant rise in depth and rotational errors, proving their role in anchoring the solution.
  • Replacing SVN with first-order SVGD (removing Hessian preconditioning) increased errors by ~75%, highlighting the importance of second-order curvature information.

• Ex Vivo Suturing:

  • During tissue piercing (heavy occlusion), PinPoint maintained an average translational error of 1.34 mm and rotational error of 19.18°, recovering quickly once the needle re-emerged.

5. Significance

This work represents a significant step forward in autonomous robotic surgery, specifically for minimally invasive procedures where stereo vision is unavailable.

• Safety and Reliability: By explicitly modeling uncertainty and maintaining multiple hypotheses, the system avoids the "overconfidence" failures common in deterministic estimators, which is critical for preventing tissue damage during autonomous actions.
• Active Sensing Potential: The second-order Hessian structure allows the system to identify directions of high uncertainty (eigenvectors), enabling future active sensing strategies where the robot can reposition the needle to reduce ambiguity before making critical decisions.
• Clinical Applicability: The ability to track needles through occlusions and in monocular settings makes this approach viable for a broader range of clinical procedures, including those using concentric tube robots in narrow anatomical pathways.

In summary, PinPoint successfully bridges the gap between geometric vision and robotic constraints, using advanced variational inference to solve a notoriously ill-posed problem with high accuracy and robust uncertainty quantification.