ExtraGS: Enhancing Endoscopic View Extrapolation via Diffusion-Guided 3D Gaussian Splatting

Technical Summary: ExtraGS

Problem Statement
Robot-assisted minimally invasive surgery (MIS) relies heavily on endoscopic perception, yet conventional endoscopes suffer from a restricted field of view (FOV) due to hardware constraints and narrow anatomical spaces. This limitation creates visual blind spots, potentially leading to missed diagnoses and forcing surgeons to rely on spatial memory. While recent neural rendering techniques like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) enable novel view synthesis from endoscopic videos, they struggle significantly during view extrapolation. When a virtual camera moves far from the training trajectory into unobserved anatomical regions, the lack of actual image inputs prevents standard photometric losses and geometric priors from providing effective supervision. Consequently, these methods produce severe geometric distortions, chaotic artifacts, and structural incompleteness, rendering the extrapolated views clinically unreliable.

Methodology: ExtraGS Framework
The authors propose ExtraGS, a framework that enhances endoscopic view extrapolation by integrating diffusion-guided 3D Gaussian Splatting. The pipeline consists of four primary stages:

1. Scene Initialization and Feasible Region Estimation:
   The process begins by optimizing a coarse 3D Gaussian representation ($G_c$) using the sparse input endoscopic RGB and depth images. From this reconstruction, a surface mesh is extracted to define a bounding volume. An occupancy grid is then constructed to distinguish between occupied anatomical regions and free space, restricting subsequent virtual camera exploration to feasible, unobserved areas within the scene.

2. Uncertainty-Guided Virtual Camera Sampling:
   To actively explore blind spots and maximize information gain, the method employs a novel virtual camera sampling strategy within the feasible region. This strategy is tailored to endoscopic scenes and incorporates:

   • Illumination-aware exploration: It prioritizes directions aligned with the darkest regions of the current image, as these areas are often under-constrained and prone to reconstruction errors.
   • Fold-aware pose control: It penalizes abrupt viewpoint changes to ensure smooth camera motion, preventing instability in complex, folded anatomical structures.
   • Coverage gain: It tracks the number of newly observed Gaussian-direction pairs to ensure comprehensive scene coverage.

3. Diffusion-Guided View Enhancement:
   Initial renderings from the sampled virtual viewpoints often contain artifacts (e.g., holes, floaters, blur). Instead of relying solely on interpolation, the framework uses a 2D video diffusion model as a prior to refine these images. The diffusion model takes the coarse rendering and the nearest real input view (as a visual anchor) to generate high-fidelity pseudo-observations. This step effectively "hallucinates" plausible anatomical structures for the unobserved regions while preserving trustworthy cues from the real data.

4. Confidence-Weighted Joint Fine-Tuning:
   To prevent the generated pseudo-observations from degrading already well-reconstructed regions, the authors introduce a confidence-weighted optimization strategy. This involves two levels of weighting:

   • Image-level confidence: Based on Gaussian Intersection over Union (G-IoU), measuring the overlap between visible Gaussians in the virtual view versus the reference view. Lower overlap yields higher weight, emphasizing complementary information.
   • Pixel-level confidence: Based on perceptual differences (LPIPS) between the coarse rendering and the diffusion-enhanced image, focusing refinement on pixels where the diffusion model made significant corrections.
     The 3DGS model is then jointly fine-tuned using both the original real observations and the confidence-weighted pseudo-observations.

Key Contributions

• ExtraGS Framework: A novel integration of 2D diffusion priors into endoscopic 3DGS to resolve severe artifact issues during large-baseline view extrapolation.
• Uncertainty-Guided Sampling: A mechanism that actively targets anatomical blind spots using illumination and fold-aware strategies to maximize information gain.
• Confidence-Weighted Fine-Tuning: A dual-level weighting strategy that incorporates diffusion-generated content to fill unobserved areas while preserving the integrity of accurately reconstructed 3D structures.
• State-of-the-Art Performance: Comprehensive evaluations demonstrating superior performance in both novel view synthesis and large-baseline extrapolation tasks compared to existing methods.

Experimental Results
The method was evaluated on the C3VDv2 dataset, a colonoscopic dataset characterized by narrow tubular structures, repetitive folds, and self-occlusions. The authors compared ExtraGS against standard 3DGS, GaussianPancakes, and UCNeRF under two challenging settings: sparse view availability (Cecum t1) and distanced/complex trajectories (Ascending t2).

• Quantitative Performance: ExtraGS achieved the highest Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) across both scenarios. For instance, in the sparse Cecum t1 scenario, ExtraGS achieved a PSNR of 20.23 dB and SSIM of 0.769, outperforming UCNeRF (19.22 dB, 0.541) and standard 3DGS (18.74 dB, 0.673). In the distanced Ascending t2 scenario, it maintained a robust PSNR of 18.75 dB and SSIM of 0.857, whereas GaussianPancakes suffered a severe drop to 10.39 dB.
• Qualitative Performance: Visual comparisons showed that while baseline methods exhibited noisy artifacts, over-smoothing, or catastrophic blurring, ExtraGS successfully reconstructed complex tissue topologies, fine vascular networks, and specular highlights with high fidelity.
• Ablation Studies: Removing the illumination- and fold-aware components resulted in a slight performance degradation, confirming the utility of these specific design choices for endoscopic environments.

Significance and Claims
The paper claims that ExtraGS significantly reduces extrapolation artifacts and achieves state-of-the-art performance in endoscopic novel view synthesis. By leveraging generative priors to provide supervision in unobserved areas, the framework effectively expands the usable field of view and improves reconstruction reliability. The authors suggest this capability holds potential value for preoperative planning, intraoperative navigation, and retrospective diagnosis in robot-assisted minimally invasive surgery.

However, the authors remain modest regarding the current scope of the work. They explicitly state that the framework assumes relatively static scenes and has not yet been validated for real-time execution, dynamic lighting, instrument occlusion, or tissue deformation. Therefore, the current results demonstrate the potential of the approach rather than validated intraoperative reliability, with future work needed to address these dynamic factors for robust 4D reconstruction.