Technical Summary: CoIn - Comprehensive 2D-3D Inpainting with Gaussian Splatting Guidance

1. Problem Statement

3D scene inpainting is critical for reconstructing areas corrupted by occlusions, sensor noise, or limited viewpoints. While recent methods leverage Gaussian Splatting (3DGS) for efficient 3D editing, existing approaches face significant limitations:

• 3D-First Pipelines: Methods that reconstruct a 3D scene before inpainting (e.g., IMFine, 3DGIC) rely heavily on precise multi-view segmentation masks. Inconsistent masks across views lead to geometric misalignment, excessive removal of non-target regions, and incorrect 3D segmentation. Furthermore, these methods are often restricted to object removal tasks and struggle with object insertion.
• 2D-First Pipelines: Approaches that perform 2D inpainting first and then enforce 3D consistency (e.g., MVInpainter) often require fine-tuning on specific datasets or additional accurate 3D labels. Pure 2D inpainting without guidance results in severe inconsistencies across multi-view images, causing inaccurate geometries and blurriness in the final 3D reconstruction.
• General Limitations: Current solutions struggle to support arbitrary-shaped masks, handle diverse tasks (both removal and insertion), and maintain strict multi-view consistency without heavy computational costs or complex volumetric optimization.

2. Methodology

The authors propose CoIn, a novel framework that bridges 2D inpainting models and 3D Gaussian Splatting (3DGS) through a multi-stage consistency pipeline. Unlike traditional 3D-first approaches, CoIn adopts a 2D-first strategy enhanced by bidirectional information flow between 2D and 3D representations.

Pipeline Overview

The method operates in four main stages:

1. Initial 2D Inpainting: A pre-trained Stable Diffusion (SD) model generates initial inpainted images for all input views. This allows for the use of arbitrary-shaped masks and diverse tasks like object insertion, though the results lack cross-view consistency.
2. Coarse Stage (Reference Adaptive GS): A 3D Gaussian Splatting scene is reconstructed from the initial 2D results. To stabilize the scene and align it with a selected reference view, the authors introduce Reference Adaptive GS with Feature Attention (Ref-GS).
   • Adaptive Weights: Per-view weights are applied to the rendering loss, up-weighting the reference view and down-weighting inconsistent views.
   • Feature Attention: Anchor features within the inpainting region are updated via unidirectional attention from neighboring anchors, enhancing texture coherence.
   • Depth Regularization: A depth loss is applied using a monocular depth estimator to ensure geometric stability.
3. Guidance Stage (Consistency Loss Guidance): The reconstructed 3D scene guides the 2D inpainting process to enforce multi-view consistency.
   • GS-based Reference Feature Warping: Features from the reference view and current views are projected into a 3D point cloud and reprojected to an intermediate viewpoint.
   • Energy Function: A consistency loss is formulated using cosine similarity for high-frequency features (DINO) and L2 similarity for RGB features. This energy function is minimized during the diffusion denoising process (inspired by FreeDoM), ensuring the generated images align with the 3D geometry.
4. Fine Stage (Texture Enhancement): A Texture-Enhancing Discriminator (TE-D) is employed to refine the 3D scene. Trained adversarially on small image patches, TE-D transfers high-frequency texture details from the initial 2D inpainting output to the GS renderings while preserving the geometric structure established in the previous stage.

3. Key Contributions

The paper outlines four primary contributions:

1. CoIn Framework: A comprehensive pipeline that seamlessly integrates generative 2D synthesis with explicit 3D reconstruction, supporting arbitrary-shaped masks and diverse tasks (removal and insertion).
2. Reference Adaptive GS (Ref-GS): A novel module that assigns adaptive weights to each view to optimize the 3DGS scene toward a reference viewpoint, utilizing feature attention to maintain structural coherence.
3. Consistency Loss Guidance (CLG): A mechanism that leverages GS-based Reference Feature Warping to enforce 3D consistency during the diffusion denoising process, bridging the gap between 2D generation and 3D geometry.
4. Texture Enhancing Discriminator (TE-D): An adversarial component that mitigates blurriness in generated patches by learning the distribution of real image patches, significantly improving photometric realism.

4. Experimental Results

The authors evaluated CoIn on the SPIn-NeRF and IMFine datasets, comparing against state-of-the-art baselines including 3DGIC, GScream, MVIP-NeRF, and IMFine.

• Quantitative Performance: CoIn achieved state-of-the-art performance across most metrics (LPIPS, PSNR, FID) on both datasets. Notably, the performance gap between using segmentation masks and bounding-box masks was minimal for CoIn, demonstrating robustness to mask shapes.
• Object Insertion: In object insertion tasks, CoIn outperformed baselines (Gaussian Editor, Infusion) in semantic alignment (CLIPdir) and user preference studies regarding multi-view consistency, visual quality, and artifact absence.
• Ablation Studies: Removing any key component (Ref-GS, CLG, or TE-D) resulted in significant performance degradation. Specifically, the absence of TE-D led to overly smooth textures, while the lack of CLG caused inconsistent results and artifacts.
• Qualitative Analysis: Visual results showed that CoIn successfully preserved non-inpainted regions and generated high-fidelity content with strict multi-view consistency, whereas 3D-first methods often failed with inconsistent masks, and other 2D-first methods exhibited geometric instability.

5. Significance and Claims

The paper claims that CoIn represents a comprehensive solution to the challenges of 3D scene inpainting by effectively leveraging bidirectional information flow between 2D and 3D domains.

• Flexibility: Unlike 3D-first pipelines that require precise segmentation and are limited to removal, CoIn supports arbitrary-shaped masks and handles both removal and insertion.
• Consistency: By integrating 3D priors into the 2D generation process via Reference Feature Warping, CoIn overcomes the cross-view inconsistency issues common in pure 2D multi-view inpainting.
• Efficiency: The framework achieves high-quality results with an efficient optimization process (approx. 2.5 hours per scene), avoiding the heavy computation and long training times associated with implicit NeRF-based methods.

The authors conclude that CoIn effectively bridges the gap between the semantic flexibility of 2D generative models and the geometric rigor of 3D Gaussian Splatting, setting a new standard for 3D editing tasks.