Technical Summary: Fine-Detail Monocular Geometry Estimation with Self-Guided Sparse Volumetric Refinement

1. Problem Statement

Monocular geometry estimation has recently achieved significant progress in zero-shot depth and point-map prediction across diverse scenes. However, state-of-the-art models still exhibit notable distortions in local 3D structures, particularly regarding fine details such as thin structures (e.g., fences, poles) and small objects. These artifacts often manifest as twisted or smeared geometries that are inconspicuous in 2D depth visualizations but become evident under viewpoint changes.

The authors attribute this limitation to a fundamental architectural mismatch: most current models decode 3D geometry within a 2D parameterization. Whether using 2D convolutional decoders, VAE-based decoding, or MLP heads over patch latents, feature interactions are governed by image-plane proximity rather than true 3D spatial relationships. This introduces a local smoothness prior that inadvertently mixes features from geometrically distant surfaces across depth discontinuities, resulting in over-smoothed geometry and a breakdown of local structural integrity.

2. Methodology

The proposed solution lifts monocular geometry modeling from the 2D image plane to 3D space via a framework consisting of two main components: a Base Model and a Self-Guided Sparse 3D Refiner (SSR).

2.1 Base Model

The system utilizes a foundation model (initialized from MoGe-2) to generate an initial dense point map $P$ and extract dense 2D features $F$ from a DINOv2 encoder. The base model predicts an affine-invariant point map, a validity mask, surface normals, and a global metric scale.

2.2 Self-Guided Sparse 3D Refiner (SSR)

The SSR module iteratively refines the initial geometry by operating directly on the occupied scene manifold using 3D sparse convolutions. The process involves three steps per iteration:

1. Self-Guided Voxelization:
   The current point map is transformed into a sparse 3D voxel shell. To normalize points with varying ranges, a factorized representation $q_{ij} = (u_{ij}, v_{ij}, \zeta_{ij})$ is used, where $u, v$ are image coordinates and $\zeta = \log Z$ is the log-depth. The voxel shell $V^{(k)}$ is constructed such that each pixel corresponds to exactly one voxel. This creates a "projection-aligned & scale-invariant shell" where the density remains invariant to metric scale transformations. Crucially, this construction ensures that two voxels are 3D-adjacent only if they are image-adjacent and their log-depths are within a specific threshold, effectively decoupling features across depth discontinuities.

2. Sparse 3D U-Net Processing:
   A sparse 3D U-Net processes the voxel set. Unlike standard 2D U-Nets, the downsampling and upsampling are geometry-dependent; the number and distribution of voxels at coarser levels vary with the scene geometry. This allows structures that collapse in image space to remain separated along the depth dimension, reducing foreground-background feature mixing.

   • Feature Injection: High-level 2D features $F$ from the base model are injected into the U-Net bottleneck. Voxels sharing an image location but differing in depth receive the same 2D feature, allowing the 3D convolutions to leverage rich appearance priors while maintaining 3D spatial locality.
   • Output: The network predicts a scalar log-depth residual $\Delta \zeta^{(k)}$ for each voxel.

3. Iterative Update:
   The predicted residuals are added to the current log-depth estimate to update the factorized representation $q^{(k+1)}$. This updated representation defines the voxel shell for the next iteration. After $K$ iterations, the final point map is recovered in Euclidean space.

2.3 Training Strategy

The model is trained in two stages:

• Stage 1 (Warm-up): Gradients from the base model features are detached from the refiner. The refiner learns strictly residual corrections starting from the known-good initialization of the base model. The refiner is trained only on synthetic data with pixel-accurate ground truth to learn sharp corrections.
• Stage 2 (Joint Fine-tuning): End-to-end training of the entire pipeline. The base model is supervised on a mixture of synthetic and real-world data to retain in-the-wild generalization.

3. Key Contributions

• Architectural Paradigm Shift: The paper proposes the first monocular geometry estimation method that lifts modeling from 2D image space to 3D space to address local structural distortions.
• Self-Guided Sparse 3D Refinement (SSR): A novel module that uses sparse 3D convolutions on a dynamically generated voxel shell. This aggregates features based on true 3D spatial locality, preventing the "bleeding" of features across depth discontinuities common in 2D approaches.
• Iterative Self-Guiding Mechanism: The refinement process is self-guided, where the voxel shell for each step is determined by the previous prediction, allowing the 3D convolutions to operate on increasingly precise surface boundaries.
• Efficient 3D Processing: By operating only on occupied regions (the thin manifold of the point map) rather than a full dense volume, the method achieves computational efficiency linear in point occupancy.

4. Experimental Results

Extensive experiments were conducted on 9 benchmarks (Synth4K, Spring, NYUv2, KITTI, ETH3D, DIODE, Sintel, iBims-1, HAMMER) covering synthetic and real-world, indoor and outdoor scenes.

• Quantitative Performance: The method achieves state-of-the-art performance across global and local metrics.
  • Global Metrics: The default ViT-L model outperforms the previous best (MoGe-2) in affine-invariant point maps, relative depth, and metric depth.
  • Local (Fine-Detail) Metrics: The advantage is most pronounced in local metrics (measuring accuracy on thin structures with strict thresholds like $\delta_{0.01}$), where the method substantially outperforms all baselines.
  • Boundary F1: While methods like InfiniDepth achieve high boundary sharpness via high-resolution 2D queries, the proposed method produces superior 3D structural fidelity, as evidenced by qualitative comparisons where prior methods show twisted structures despite sharp 2D edges.
• Ablation Studies:
  • 3D vs. 2D: Replacing the sparse 3D U-Net with a parameter-matched 2D U-Net fails to improve local geometry, confirming the necessity of the 3D inductive bias.
  • Feature Injection: Removing DINO feature injection degrades performance, indicating that semantic and contextual information is crucial for disambiguating local 3D geometry.
  • Iterations: Increasing refinement iterations ($K$) improves local metrics up to a saturation point ($K \approx 5$) without degrading global accuracy, demonstrating the stability of the learned residual corrections.
• Runtime: The method adds moderate overhead compared to the base model (e.g., 121ms vs 39ms for ViT-L on A100) but remains faster than other detail-enhancement methods while achieving higher accuracy.

5. Significance and Claims

The paper claims to bridge the gap between visually plausible 2D depth maps and faithful 3D reconstruction. By addressing the root cause of local distortions—the mismatch between 2D feature interaction and 3D geometry—the proposed method produces point maps that are both visually plausible and geometrically accurate, particularly in challenging regions with thin or elongated structures.

The authors position this work as a significant advancement in fine-detail monocular geometry estimation, offering the first approach that successfully lifts geometry modeling to 3D space to resolve structural artifacts inherent in 2D parameterizations.

Limitations: The authors acknowledge that as a regression-based paradigm, the method inherently struggles with pixel ambiguity along object boundaries, which can lead to "fly-points" and prevents the generation of perfectly sharp depth edges. They suggest that integrating generative formulations could address this in future work.