Monocular Endoscopic Tissue 3D Reconstruction with Multi-Level Geometry Regularization

This paper proposes a novel 3D Gaussian Splatting framework for monocular endoscopic tissue reconstruction that integrates surface-aware mesh constraints and multi-level geometric regularization to achieve both real-time rendering and high-quality, physically plausible deformable surface reconstruction.

The Gist

Imagine you are a surgeon performing a delicate operation inside a patient's body. You are looking through a tiny camera (an endoscope) that gives you a 2D view of squishy, moving organs. To perform robot-assisted surgery safely, the robot needs a perfect, real-time 3D map of what it's seeing. But here's the problem: human tissue is soft, it stretches, it folds, and it gets covered by surgical tools. Creating a 3D movie of this from a single camera angle is like trying to guess the shape of a squishy balloon just by looking at its shadow.

This paper introduces a new "magic trick" to solve that problem. The authors, Yangsen Chen and Hao Wang, have built a system that creates a smooth, real-time 3D model of moving tissue using a technology called 3D Gaussian Splatting.

Here is how they did it, explained with everyday analogies:

1. The Problem: The "Floating Cloud" Issue

Previous methods tried to build these 3D models using "NeRFs" (Neural Radiance Fields). Think of NeRFs like a very slow, high-quality painter. They can make a beautiful picture, but it takes hours to paint one frame, and the robot can't wait that long.

Other newer methods use "3D Gaussian Splatting." Imagine this as a cloud of millions of tiny, fuzzy balloons (Gaussians) that float in space to form the shape of the tissue. It's incredibly fast (like a video game), but because the balloons are fuzzy, they often don't stick together perfectly. The result looks like a wobbly, glitchy cloud rather than a smooth skin. Sometimes, parts of the tissue look like they are floating in mid-air (artifacts) or the surface looks bumpy and unrealistic.

2. The Solution: "The Mesh Skeleton"

The authors realized that to make the fuzzy balloons look like smooth skin, they needed a skeleton to hold them in place.

• Step 1: Build a Wireframe (The Mesh): First, they take the very first frame of the video and build a rigid 3D wireframe mesh (like a digital chicken wire) of the tissue surface.
• Step 2: The "Velcro" Trick: They then attach their fuzzy balloons (Gaussians) directly to this wireframe. Imagine gluing the balloons onto the chicken wire. Now, the balloons can't float away or drift into weird shapes; they are forced to follow the smooth surface of the wireframe. This solves the "bumpy surface" problem.

3. The Challenge: The "Moving Jello" Problem

Once the tissue starts moving (because the surgeon is pulling or cutting it), the wireframe needs to move too. But tissue isn't a rigid rock; it's like Jello. It bends in some places and stays stiff in others.

If you just let the balloons move freely, the 3D model might twist into impossible shapes. The authors added two "rules of physics" to keep the movement realistic:

• Rule A: The "Local Stiffness" (Semi-Rigidity): They identified key spots on the tissue (like where blood vessels cross) and told the balloons: "Hey, the area right around these spots shouldn't stretch too much. Stay stiff like a rock." This is like putting a little splint on a specific part of a bending finger.
• Rule B: The "Global Flow" (Non-Rigidity): For the rest of the tissue, they told the balloons: "You can bend and stretch, but you must move smoothly with your neighbors. Don't let one balloon fly off in a different direction than the one next to it." This ensures the whole piece of tissue moves like a single, cohesive unit of Jello, not a bag of marbles.

4. Handling the "Blind Spots"

During surgery, tools often block the camera's view, creating black holes in the video. The authors added a "video inpainting" feature. Think of this as a smart AI artist that looks at the parts of the tissue hidden by the tool and guesses what the tissue looks like underneath based on how it was moving a second ago. It fills in the missing pieces so the 3D model doesn't have holes.

The Result: Fast, Smooth, and Real

The result is a system that is:

• Fast: It renders the 3D scene in real-time (over 100 times faster than the old "painter" methods).
• Smooth: The tissue looks like real skin, not a glitchy cloud.
• Accurate: It handles the stretching and bending of organs without breaking the 3D model.

In summary: The authors took a fast but messy 3D technology (the fuzzy balloons) and gave it a rigid skeleton (the mesh) and a set of smart movement rules (local stiffness and global flow). This allows a surgical robot to see a perfect, real-time 3D map of squishy, moving organs, making surgery safer and more precise.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of 3D reconstruction of deformable endoscopic tissues for robot-assisted surgery. Existing methods face significant limitations:

• NeRF-based methods: While capable of high-quality reconstruction, they suffer from slow training times and rendering speeds that are not real-time, hindering clinical application.
• 3D Gaussian Splatting (3DGS) methods: While fast, standard 3DGS approaches often fail to produce consistent, smooth surface reconstructions for soft tissues, leading to artifacts and "floaters" (floating 3D noise).
• Environmental Challenges: Endoscopic videos suffer from limited viewpoints (monocular or stereo constraints), significant occlusions by surgical instruments, and the highly non-rigid, topologically changing nature of soft tissue.

2. Methodology

The authors propose a novel framework based on 3D Gaussian Splatting enhanced with Multi-Level Geometry Regularization. The pipeline consists of three main stages:

A. Preparatory Procedures

• Sparse Key Point Matching: Uses Scale-Invariant Feature Transform (SIFT) to identify and track sparse feature points (e.g., vascular intersections) across frames. These tracks guide the deformation learning process.
• Video Inpainting: To handle occlusions caused by surgical tools, the authors employ a Transformer-based video inpainting model (fine-tuned on the StereoMIS dataset). This generates a coherent video sequence by filling in masked regions while preserving spatio-temporal consistency.

B. Surface-Aware Reconstruction (Initial Frame)

To ensure the initial 3D structure is geometrically accurate and smooth:

1. Mesh Generation: A static mesh is reconstructed for the first frame using NeuS2 (a Neural Implicit Surface method) supervised by RGB, depth, and Eikonal losses.
2. Mesh-Restricted Gaussian Splatting: 3D Gaussian kernels are initialized at the centroids of the mesh triangles.
   • Scale Regularization ($L_{scale}$): Constrains the size of Gaussian kernels relative to the inscribed circle of the binding triangle to prevent them from becoming too large and covering multiple triangles.
   • Shift Regularization ($L_{shift}$): Penalizes large displacements of Gaussians away from their binding triangles to maintain local continuity.
   • This ensures the Gaussians adhere strictly to the underlying surface geometry.

C. Semi-Rigidity Deformation (Dynamic Frames)

To model realistic tissue deformation while preventing artifacts, the method introduces Multi-Level Geometry Regularization:

1. Local Rigidity Restriction (ARAP Loss): Inspired by "As-Rigid-As-Possible" (ARAP) mesh deformation, this loss guides the motion of Gaussians in local neighborhoods (defined by sparse key points) to remain rigid. This preserves fine details like vascular structures.
2. Global Non-Rigidity Restriction:
   • Rotation Consistency ($L_{rot}$): Encourages neighboring Gaussians to share similar rotation patterns over short time horizons.
   • Isometry Loss ($L_{iso}$): Enforces that the distance between pairs of Gaussians remains constant over longer time horizons, preventing the scene from drifting apart or collapsing.

3. Key Contributions

1. Surface-Aware Endoscopic Reconstruction: A novel integration of RGB, depth, and optical flow data with a mesh-constrained 3DGS approach to achieve consistent and smooth geometry, eliminating the "fuzzy" surfaces common in standard 3DGS.
2. Semi-Rigidity Deformation Guidance: A dual-constraint system (Local Rigidity + Global Non-Rigidity) that models physically plausible deformations, effectively avoiding 3D floaters and ensuring structural coherence during tissue movement.
3. Real-Time Performance with High Fidelity: The proposed method achieves state-of-the-art reconstruction quality in both texture and geometry while enabling real-time rendering (>100 FPS) and significantly reducing training time compared to NeRF-based methods.
4. Occlusion Handling: The first work to integrate a video inpainting module specifically tailored for the spatial-temporal masking issues in endoscopic 3D reconstruction.

4. Experimental Results

The method was evaluated on two datasets: EndoNeRF (in-vivo prostatectomy) and SCARED (porcine cadaver abdominal anatomy).

• Quantitative Performance:
  • On the EndoNeRF dataset, the proposed method achieved a PSNR of 38.05 (Cutting) and 38.27 (Pulling), outperforming NeRF-based methods (EndoNeRF, EndoSurf) and other Gaussian-based methods (EndoGS, EndoGaussian).
  • It achieved an SSIM of 0.965 and a low LPIPS of 0.047, indicating superior structural similarity and perceptual quality.
  • On the SCARED dataset, it achieved a PSNR of 28.31, the highest among all compared methods.
• Efficiency:
  • Training Time: Reduced to ~2 minutes per frame (compared to hours for NeRF methods).
  • Rendering Speed: Achieved ~170 FPS (real-time), whereas NeRF methods are typically <1 FPS.
  • Resource Usage: Requires only ~3GB GPU memory, roughly one-tenth of the memory required by NeRF-based approaches.
• Ablation Studies: Removing the surface-aware reconstruction caused a significant drop in PSNR (from 38.16 to 33.60), proving the necessity of mesh constraints. Similarly, removing local or global rigidity constraints led to visible artifacts and reduced geometric consistency.

5. Significance

This work bridges the gap between high-fidelity 3D reconstruction and real-time performance, which is essential for Robot-Assisted Surgery (RAS).

• Clinical Impact: By providing smooth, artifact-free, and real-time 3D views of deformable tissues, the system can enhance surgeon situational awareness and improve the precision of robotic interventions.
• Technical Advancement: It demonstrates that 3D Gaussian Splatting, when combined with strong geometric priors (mesh constraints and semi-rigidity), can outperform implicit representations (NeRF) in dynamic, occluded, and soft-tissue environments.
• Hardware Accessibility: The drastic reduction in GPU memory and training time makes advanced 3D reconstruction feasible for deployment on standard surgical hardware without massive computational clusters.