Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

1. Problem Statement

Endoluminal surgery (performed through natural orifices like the mouth, anus, or urethra) offers minimally invasive treatment for early-stage gastrointestinal and urinary tract cancers. However, current practices face significant challenges:

• Instrument Limitations: Traditional endoscopes lack dexterity, making precise tissue resection and retraction difficult.
• Steep Learning Curve: Success heavily relies on surgeon experience due to the difficulty of maneuvering long, flexible instruments in confined spaces.
• Perception Gaps: Continuum robots (highly flexible instruments) lack rigid joints, making kinematic modeling difficult. Existing state estimation often relies on expensive, intrusive sensors (e.g., FBG, EM) or external imaging (CT/MRI), which are impractical for real-time intraoperative use.
• Data Scarcity: There is a lack of realistic training data for flexible robots in endoluminal environments, and existing simulators focus on rigid laparoscopic tools.

The core problem is the need for a real-time, monocular vision-based perception framework that can simultaneously segment flexible instruments, estimate their 3D states, and reconstruct the 3D geometry of the surgical scene (including tissue depth) to enable precise robotic control without external sensors.

2. Methodology

The authors propose an integrated perception framework consisting of three learning-based modules, supported by a physically realistic simulator for data generation.

A. Physically-Realistic Simulator

To overcome the lack of labeled real-world data, the authors developed a simulator using Unity:

• Robot Modeling: Uses a piece-wise constant curvature model to simulate flexible instruments divided into three segments (Shoulder, Proximal, Distal) with four state parameters ($\alpha, \beta, \gamma, \delta$).
• Scene Generation: Utilizes the Colonoscopy 3D Video Dataset (C3VD) and Colon10K to extract high-fidelity 3D organ geometries and textures via Poisson surface reconstruction.
• Data Output: Generates synthetic endoscopic images with ground-truth depth maps, surface normals, and robot state labels.

B. 2D Annotation-Efficient Segmentation

• Approach: Utilizes a Knowledge Distillation strategy. A powerful pre-trained foundation model (Segment Anything Model - SAM) acts as the "teacher," while a lightweight U-Net acts as the "student."
• Process: SAM segments instruments using simple bounding box prompts (automatically updated frame-by-frame). The lightweight U-Net learns from SAM's masks using Negative Log Likelihood (NLL) loss.
• Augmentation: Uses semantic-guided data augmentation (pasting instrument masks onto diverse backgrounds from the EAD dataset) to improve robustness.

C. 3D Robot State Estimation

• Goal: Estimate the 3D shape of the continuum robot from 2D segmentation masks.
• Probabilistic Modeling: Instead of direct regression (which struggles with rotation spaces), the model predicts a Matrix Fisher Distribution over the rotation space ($SO(3)$).
• Output: Provides not only the state parameters (bending angles, yaw, roll) but also an uncertainty metric (via Singular Value Decomposition of the distribution parameters), indicating the confidence of the prediction.

D. 3D Monocular Depth Estimation

• Challenge: Standard depth models fail in dynamic scenes with flexible instruments and lack ground truth in clinical settings.
• Illumination Modeling: The authors model the endoscopic light as a point source, deriving a Per-Pixel Rendering (PPR) field that correlates light attenuation, direction, and surface normals with image intensity.
• Training Strategy (Synthetic-to-Real):
  1. Teacher Network: Trained on synthetic data with supervised depth loss and PPR loss.
  2. Student Network: Trained on both synthetic data and unlabeled real clinical data.
  3. Self-Supervision: On real data, the student uses the Teacher's output as pseudo-ground truth and minimizes a self-supervised PPR loss (correlating predicted PPR with actual image intensity).
• Architecture: Combines a foundation depth model (Depth Anything) with a custom improvement module (PRMod) that fuses image features and PPR features via cross-attention.

3. Key Contributions

1. Integrated Perception Framework: The first work to comprehensively address 2D segmentation, 3D robot state estimation, and 3D depth estimation for continuum robots using solely monocular vision.
2. Probabilistic State Estimation: Introduction of a Matrix Fisher distribution approach to estimate robot states with associated uncertainty, addressing the non-linear nature of rotation spaces.
3. Physically-Realistic Simulator: A novel simulation platform that accurately models flexible continuum instruments and generates high-fidelity synthetic endoluminal data, bridging the synthetic-to-real gap.
4. Illumination-Aware Depth Estimation: A novel depth estimation method that incorporates explicit illumination modeling (PPR) and a teacher-student transfer learning strategy to achieve high accuracy on real clinical data without ground-truth depth.

4. Experimental Results

The framework was evaluated on ex-vivo porcine GI tracts and synthetic datasets.

• 2D Segmentation:
  • Achieved 94.93% Dice score and 90.38% IoU on real data, significantly outperforming baselines trained only on synthetic data (Syn) or with simple copy-paste augmentation.
  • Inference speed: 52.1 fps (lightweight model) vs. 1.4 fps for the foundation model.
• 3D Robot State Estimation:
  • Achieved a median angular error of ~2.08° for bending angles and ~5.51° for roll/yaw, outperforming keypoint-based, skeleton-based, and standard regression methods.
  • Successfully correlated uncertainty with error rates (high uncertainty = high error).
• 3D Depth Estimation:
  • Achieved an RMSE of 4.020 mm on synthetic test data.
  • The student network (trained with self-supervision) significantly outperformed the teacher (synthetic-only) on real clinical data, demonstrating effective domain adaptation.
• System-Level Evaluation (Real-World Control):
  • Remote-Control Calibration: Reduced manipulation time for trajectory-following tasks by >71% for professionals and ~72% for inexperienced users by correcting controller-instrument mapping errors.
  • 3D Shape Monitoring: Provided real-time visualization of instrument shapes and uncertainty, allowing surgeons to detect when instruments were partially occluded.
  • Instrument-Tissue Distance: Successfully inferred 3D deviations between the instrument tip and tissue, aiding in safe palpation and contact detection.

5. Significance

This work represents a significant step forward in robotic endoluminal surgery by:

• Enhancing Safety and Precision: Providing surgeons with real-time 3D awareness of flexible instruments and tissue, reducing the risk of collisions and unintended tissue damage.
• Lowering the Learning Curve: The automated calibration and state monitoring significantly reduce the time required for surgeons (especially novices) to control complex flexible robots.
• Enabling Autonomous Capabilities: The robust perception of instrument-tissue interactions and 3D geometry is a prerequisite for future autonomous or semi-autonomous surgical tasks in confined spaces.
• Bridging the Data Gap: The proposed simulator and training strategy offer a scalable solution for developing perception algorithms in medical robotics where real-world labeled data is scarce and expensive to obtain.

The paper concludes that while in-vivo validation is a future step, the proposed framework significantly improves the robustness and controllability of continuum robotic systems, paving the way for more precise and reliable minimally invasive surgeries.