Technical Summary: DrivingDepth

Problem Statement: The Geometry–Scale Conflict

Autonomous driving requires depth estimation that is simultaneously dense, pixel-aligned, and metrically consistent. However, existing sensing modalities and models face a fundamental tension termed the geometry–scale conflict:

• Visual Foundation Models (e.g., DepthAnything3, Depth Pro) produce dense, pixel-aligned visual geometry with strong boundary consistency but lack reliable metric scale. They reason independently per image or rely on visual correspondences without absolute distance anchors.
• Projected LiDAR provides metric anchors but is inherently sparse, noisy, unevenly distributed, and often misaligned with image structures due to calibration drift and surface reflections.

The Limitation of Existing Methods: Current sparse-prompted approaches (e.g., MapAnything, PriorDA) attempt to incorporate LiDAR by regenerating depth from scratch or fusing features to predict absolute depth. This approach often overrides the coherent geometric priors of foundation models, leading to structural artifacts (holes, broken surfaces) on visually continuous objects where LiDAR evidence is sparse. Conversely, global post-hoc alignment can adjust average scale but fails to correct spatially varying metric errors across different surfaces.

Methodology: DrivingDepth

DrivingDepth proposes a minimal-intervention framework that resolves this conflict by treating sparse LiDAR not as dense supervision targets, but as geometric prompts for residual pixel-wise scale correction. The core insight is that foundation models already capture geometrically coherent relative depth; the model only needs to learn the per-pixel scale factor to map this relative geometry to metric coordinates.

Architecture

The framework is built upon a frozen DepthAnything3 (DA3) backbone and operates at two complementary levels:

1. Geometry-Preserving Feature Adapter (GPFA):

   • Function: Injects sparse-depth cues into the frozen DA3 representation without overwriting the backbone's geometry.
   • Mechanism: A lightweight CNN encodes multi-resolution LiDAR prompts into sparse-depth tokens. These tokens condition the image tokens via cross-attention.
   • Constrained Propagation: To respect the physical setup of surround-view cameras, self-attention is restricted to co-visible patterns: adjacent cameras at the same timestamp and the same camera across consecutive frames. This prevents geometric noise from unrelated views.
   • Output: Depth-aware features are concatenated with original backbone features, ensuring the scale head retains direct access to the unmodified geometric prior.

2. Sparse-Aware Pixel-Scale Head:

   • Function: Predicts a dense per-pixel scale map ($S_{pix}$) and a confidence map ($C$).
   • Input: Combines the depth-aware features from the adapter with a multi-resolution LiDAR prompt (constructed by downsampling sparse depth to increase effective density for lower-resolution decoder stages).
   • Initialization: The head is initialized neutrally (zero-initialized convolutions, scale initialized to 1) so the model reproduces the original DA3 prediction at the start of training.
   • Output: The final metric depth $\hat{D}_c$ is computed as $D_{prior} \odot S_{pix} \cdot s_g$, where $s_g$ is a global clip-level scalar derived via Robust Outlier Exclusion (ROE) alignment to ensure cross-view consistency.

Training Objective

The loss function balances metric anchoring against geometry preservation:

• Sparse Depth Alignment ($L_{depth}$): Combines multi-resolution supervision with a confidence-weighted regression. The learned confidence map automatically downweights unreliable LiDAR projections (e.g., misaligned points).
• Surface-Normal Regularization ($L_{norm}$): Penalizes angular deviations between surface normals of the corrected depth and the DA3 prior. This prevents the model from fitting noisy points in a way that breaks surface continuity.
• Scale Regularization ($L_{scale}$): Encourages the scale map to be smooth and close to unity, preventing overfitting to sparse noise.

Key Contributions

1. Problem Formulation: Identification of the geometry–scale conflict in driving depth and the proposal to resolve it via residual pixel-wise scale correction of a frozen foundation prior, rather than regenerating depth.
2. Minimal-Intervention Architecture: A dual-level design featuring a Geometry-Preserving Feature Adapter (injecting cues via constrained attention) and a Sparse-Aware Pixel-Scale Head (predicting corrections with learned confidence).
3. Training Strategy: A balanced objective using confidence-weighted alignment, surface-normal regularization, and scale smoothness to mediate between sparse metric anchoring and dense geometry preservation.
4. Empirical Validation: Demonstration of competitive metric accuracy with significantly superior geometric consistency compared to state-of-the-art methods.

Experimental Results

Experiments were conducted on nuScenes and DDAD datasets using 4-frame surround-view inputs.

• Metric Accuracy vs. Consistency: DrivingDepth achieves an AbsRel of 11.19 and an EdgeCR of 5.741 on nuScenes.
  • It outperforms MapAnything (AbsRel 11.99, EdgeCR 1.914), which sacrifices boundary alignment for metric accuracy.
  • It significantly improves upon DepthAnything3 (AbsRel 15.88), which lacks metric scale.
  • It avoids the structural artifacts seen in PriorDA (EdgeCR 2.524), which hallucinates depth discontinuities by overfitting to sparse LiDAR points.
• Robustness to Sparse Data: Under 10% LiDAR density, DrivingDepth maintains high performance (EdgeCR > 5.7), whereas MapAnything degrades sharply (EdgeCR drops to ~1.9). This validates the efficacy of treating LiDAR as geometric prompts rather than dense supervision.
• Qualitative Analysis: Visualizations show that DrivingDepth preserves sharp object boundaries and coherent 3D structures (e.g., moving trucks, building facades) while correcting scale errors, whereas other methods either lack scale or introduce geometric distortions.

Significance and Claims

The paper claims that DrivingDepth successfully bridges the gap between dense visual geometry and reliable metric scale by decomposing the problem: recognizing that foundation models already provide the necessary surface structure, and only the scale mapping requires learning.

• Efficiency: By freezing the backbone and learning only a lightweight scale head and adapter, the model fits on a single 8-GPU node, avoiding the cost of end-to-end retraining.
• Design Principle: The "minimal-intervention" principle ensures that dense visual geometry is preserved by construction, preventing the structural artifacts common in methods that regenerate depth from scratch.
• Limitation: The authors modestly note that the approach relies on the frozen foundation model's geometry being fundamentally correct. If the foundation model fails to capture structure (e.g., on highly reflective or transparent surfaces), the scale correction cannot recover the missing geometry.