In Depth We Trust: Reliable Monocular Depth Supervision for Gaussian Splatting

1. Problem Statement

While 3D Gaussian Splatting (3DGS) has achieved state-of-the-art results in novel view synthesis, it struggles with sparse training data and textureless surfaces, often leading to geometric artifacts (e.g., "floaters") and inaccurate depth maps.

• The Challenge: To mitigate these issues, researchers often use Monocular Depth Estimation (MDE) models as priors. However, foundation MDE models suffer from scale ambiguity (unknown absolute depth), multi-view inconsistency (different scales across views), and local geometric inaccuracies.
• The Consequence: Naively applying these noisy, scale-ambiguous depth priors to 3DGS training can corrupt well-reconstructed geometry, leading to worse rendering quality than training without depth supervision. Existing methods that rely on coarse scale alignment (e.g., matching to sparse SfM points) fail when SfM data is sparse or when the MDE model is weak.

2. Methodology

The authors propose a versatile training framework that integrates monocular depth priors into Gaussian Splatting optimization while actively filtering out unreliable cues. The framework consists of three core components:

A. Depth-Inconsistency Mask (DIM)

To prevent erroneous depth predictions from corrupting the scene geometry, the authors introduce a mechanism to identify and isolate "ill-posed" regions.

• Virtual Stereo Setup: During training, a virtual stereo pair is created using the current camera pose ($P_l$) and a pseudo-camera offset by a baseline $b$ ($P_r$).
• Consistency Check: The system renders depth maps for both views ($\hat{D}_l, \hat{D}_r$). It back-projects the right-view depth into 3D space and re-projects it onto the left view.
• Mask Generation: A pixel-wise difference is calculated between the rendered left depth ($\hat{D}_l$) and the re-projected depth ($\hat{D}_{l \leftarrow r}$). If the difference exceeds a threshold $\epsilon$ (or if the point is missing due to occlusion), a Depth-Inconsistency Mask (DIM) is set to 1.
• Function: This mask identifies regions where multi-view geometric consistency is broken (indicating poor reconstruction or unreliable priors). Supervision is selectively applied only to these inconsistent regions, protecting well-reconstructed areas from noisy MDE signals.

B. Scale-Invariant Depth Loss ($L_{sid}$) with Masking

Instead of applying depth supervision globally, the framework applies the standard scale-invariant depth loss ($L_1$) only where the DIM indicates inconsistency.

• Formula: $L_{abs} = L_1(M \odot D, M \odot \hat{D})$
• Goal: This forces the Gaussian primitives in inconsistent regions to align with the coarse absolute depth provided by the MDE model (after alignment with sparse SfM points), correcting large-scale geometric errors without disturbing accurate local structures.

C. Gradient-Alignment Loss (GAL)

To address the lack of fine-grained detail in MDE models and the smoothing effects of patch-based correlation losses (like NCC), the authors introduce a gradient-based loss.

• Mechanism: GAL computes the $L_1$ difference between the spatial gradients (first-order derivatives) of the rendered depth and the aligned monocular depth in both horizontal ($x$) and vertical ($y$) directions.
• Formula: $L_{rel} = L_1(\partial_x D, \partial_x \hat{D}) + L_1(\partial_y D, \partial_y \hat{D})$
• Goal: This loss encourages the network to learn relative depth variations and high-frequency geometric details, ensuring sharp edges and fine structures are preserved even when the absolute scale is ambiguous.

Total Supervision Loss:
The final depth supervision term is a weighted combination:
$$R = \alpha L_{abs} + \beta L_{rel}$$

3. Key Contributions

1. Reliable Training Framework: A novel approach that leverages readily available, imperfect monocular depth priors without degrading rendering quality, making 3DGS robust to scale ambiguity and noise.
2. Depth-Inconsistency Mask (DIM): A dynamic mechanism using a virtual stereo setup to detect multi-view inconsistent Gaussians, allowing for selective regularization that prevents error propagation.
3. Gradient-Alignment Loss (GAL): A new loss function that captures fine-grained geometric cues from relative depth variations, outperforming traditional patch-based correlation losses in preserving detail and computational efficiency.
4. Generalization: The framework is agnostic to the specific MDE backbone (tested with DepthAnything V2, MoGe-2, UniDepth V2, Marigold) and compatible with different GS variants (3DGS and 2DGS).

4. Experimental Results

The method was evaluated on three real-world datasets: ScanNet++, MipNeRF 360, and TanksAndTemples, under both low-data (sparse views) and moderate-data settings.

• Rendering Quality: The proposed method consistently outperformed the baseline (3DGS without depth) and state-of-the-art competitors (SparseGS, DNGaussian, FSGS).
  • In low-data settings, it achieved an average PSNR gain of >0.47 dB over the baseline.
  • In moderate-data settings, where other methods often degraded performance due to over-regularization, the proposed method continued to improve quality.
• Depth Accuracy: Significant improvements were observed in rendered depth metrics (Abs. Rel. and $\delta_1$ accuracy), even when using weaker MDE models (e.g., Marigold).
• Ablation Studies:
  • Removing the DIM resulted in performance drops, confirming that masking inconsistent regions is crucial.
  • Replacing GAL with NCC or Cosine similarity losses resulted in lower PSNR and higher training times, proving GAL's efficiency and effectiveness in preserving details.
• Robustness: The method remained effective across different MDE backbones, demonstrating that it can leverage future improvements in foundation depth models without architectural changes.

5. Significance

This paper addresses a critical bottleneck in 3D reconstruction: the trade-off between using helpful geometric priors and the risk of introducing noise.

• Practical Impact: It enables the use of cost-effective, single-image depth estimation models to enhance high-fidelity 3D scene reconstruction without requiring specialized RGB-D hardware.
• Theoretical Insight: It highlights that selective supervision (applying priors only where geometry is uncertain) is more effective than global supervision.
• Future Direction: By safely harnessing 2D foundation priors, this framework paves the way for robust, high-fidelity scene reconstruction in complex, real-world scenarios where data is sparse or noisy.