Reliability-Aware Monocular Depth Supervision for Sparse-View Neural Reconstruction

Technical Summary: Reliability-Aware Monocular Depth Supervision for Sparse-View Neural Reconstruction

Problem Statement
Sparse-view neural reconstruction is particularly challenging in outdoor driving scenes. Unlike object-centric datasets where cameras orbit a target, autonomous driving sequences involve cameras moving along narrow, forward-facing trajectories. This results in limited multi-view overlap, weak parallax for distant regions, and difficulties with dynamic objects, reflections, and sky regions. While Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) achieve high-quality rendering with sufficient views, they become under-constrained with sparse inputs, often leading to incorrect geometry and "floaters."

Although monocular depth estimators like Depth Anything V2 (DA-V2) offer dense geometric priors, their predictions are relative, noisy, and not uniformly reliable across all image regions. Applying these priors uniformly can introduce errors in unreliable areas (e.g., occlusion boundaries or reflective surfaces). The core question addressed is how to effectively leverage monocular depth supervision in sparse-view settings without compromising reconstruction quality due to noisy priors.

Methodology
The authors propose a reliability-aware monocular depth supervision pipeline that selectively applies depth supervision based on the reliability of the RGB reconstruction. The approach consists of three main stages:

1. Scale-Aligned Monocular Depth Prior:

   • DA-V2 is used to generate dense relative depth maps.
   • These maps are aligned to metric depth using a per-image scale-shift fitting process against valid reference anchors (LiDAR points for KITTI, COLMAP points for the Bicycle scene).
   • The aligned depth is clipped to a valid range (e.g., 80m for KITTI) to serve as a metric prior.

2. Photometric Reliability Masking:

   • An RGB-only baseline model is trained first to establish a reference for reconstruction quality.
   • Per-pixel photometric reconstruction error ($e(u)$) is computed by comparing the baseline's rendered image against the ground truth.
   • A binary mask is generated based on a threshold $\tau$. Pixels with low photometric error ($e(u) < \tau$) are deemed "reliable" and retained for depth supervision, while high-error regions (likely containing dynamic objects, reflections, or sky) are masked out.

3. Masked Depth Supervision Objective:

   • The reconstruction model (either Mip-NeRF-360 or Splatfacto) is retrained with a joint loss function: $L = L_{rgb} + \lambda_{depth} L_{depth}$.
   • The RGB loss is applied to the full image.
   • The depth loss ($L_{depth}$) is computed only on pixels where the effective supervision mask ($M_{eff}$) is active, defined as the intersection of the photometric reliability mask and the depth-validity mask.
   • This ensures that the noisy monocular prior only regularizes regions where the RGB reconstruction is already consistent.

Key Contributions

• Selective Supervision Strategy: The paper introduces a method to apply monocular depth priors selectively based on photometric reconstruction error, rather than uniformly across the image.
• Comparative Evaluation: The strategy is evaluated on two distinct scene representations: an implicit density field (Mip-NeRF-360) and an explicit Gaussian representation (Splatfacto).
• Ablation and Generalization: Through matched-ratio mask ablations and experiments on multiple KITTI sequences (02 and 05) and the Bicycle scene, the authors demonstrate that gains are derived from selecting reliable regions rather than simply reducing the number of supervised pixels.
• Backbone Sensitivity Analysis: The work highlights that the efficacy of depth supervision is highly dependent on the underlying representation (implicit vs. explicit) and the scene configuration (sparse forward-facing vs. object-centric).

Experimental Results
The evaluation was conducted on KITTISeq02 (every-2nd frame sparse setting), KITTISeq05, and the Mip-NeRF-360 Bicycle scene.

• Mip-NeRF-360 (Implicit):

  • On KITTISeq02, masked monocular depth supervision provided only marginal rendering gains (e.g., PSNR increased from 20.378 to 20.607 in the best case) and did not improve metric geometry (AbsRel and RMSE worsened compared to the RGB-only baseline).
  • On KITTISeq05, both global and masked depth supervision degraded both rendering quality and geometry compared to RGB-only training.
  • Conclusion: The implicit density field is sensitive to the noise in scale-aligned monocular depth, and the prior acts as a weak or destabilizing regularizer.

• Splatfacto (Explicit):

  • On KITTISeq02, Splatfacto benefited significantly. PSNR improved from 14.903 (RGB-only) to 15.932, and RMSE dropped drastically from 0.542 to 0.100.
  • Matched-ratio ablations confirmed that low-error masks outperformed high-error and random masks, proving the benefit comes from region selection.
  • On KITTISeq05, similar improvements in both rendering and geometry were observed.
  • Conclusion: The explicit Gaussian representation allows depth supervision to directly guide Gaussian placement, effectively reducing floaters and improving geometry in sparse regions.

• Bicycle Scene (Object-Centric):

  • In this scene with strong multi-view coverage, depth supervision improved geometry (RMSE dropped from 1.479 to 0.722) but generally degraded RGB rendering quality (PSNR, SSIM, LPIPS decreased).
  • This suggests a trade-off where depth priors can over-regularize the reconstruction when RGB constraints are already strong.

Significance and Claims
The paper concludes that monocular depth priors are a double-edged sword in sparse-view reconstruction. Their utility is not universal but depends heavily on the reconstruction backbone and scene constraints:

1. Backbone Dependence: They are highly effective for explicit Gaussian representations (Splatfacto) in sparse, forward-facing scenes but offer little to no benefit—and can be detrimental—for implicit NeRF-style representations (Mip-NeRF-360).
2. Reliability is Key: The gains are not merely due to adding more supervision data but stem from filtering out unreliable regions where the monocular prior conflicts with the RGB observations.
3. Trade-offs: In scenes with strong multi-view coverage, depth supervision may improve geometric metrics at the cost of photometric fidelity.

The authors advocate for the selective application of monocular depth priors with moderate weighting, particularly for explicit scene representations in under-constrained sparse-view scenarios. They suggest future work could focus on better scale alignment, uncertainty-aware masking, and adaptive loss weighting.