Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces

Technical Summary: Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces

Problem Statement
Recent advances in Monocular Depth Estimation (MDE) have produced models with impressive zero-shot generalization capabilities in general scenes. However, these models frequently fail when encountering non-Lambertian surfaces, specifically transparent or mirror (ToM) regions. The unique reflective properties of these surfaces cause standard models to predict erroneous depth structures based on reflected content rather than the surface itself. Previous attempts to address this, such as Depth4ToM, rely on external segmentation masks to in-paint RGB images with random colors before processing them through a pre-trained MDE model. These approaches suffer from two main limitations: they are highly dependent on the accuracy of additional input masks, and the use of random colors during in-painting lacks robustness, requiring extensive hyperparameter tuning. Furthermore, lighting conditions significantly impact the recovery of depth in non-Lambertian surfaces; excessive lighting can overwhelm textures, while insufficient lighting introduces noise and distortion, particularly affecting transparent objects and mirror reflections.

Methodology
The authors propose a training framework designed to incrementally enable a baseline model (specifically Depth Anything V2) to directly learn the unique characteristics of non-Lambertian surfaces, eliminating the need for RGB in-painting. The method consists of three core components:

1. Random Tone-mapping Augmentation (RTA):
   To address the sensitivity of non-Lambertian depth estimation to lighting, the authors employ random tone-mapping during the training phase on the synthetic Hypersim dataset. By applying the transformation $I_{aug} = \alpha I_{ori}^{\gamma}$, where $\gamma$ is a standard gamma correction factor and $\alpha$ is a scale factor derived from random percentile intensity values (between the 70th and 99th percentiles), the model is exposed to a diverse range of lighting conditions. This aims to enhance the model's zero-shot generalization capability under varying illumination.

2. Non-Lambertian Surface Regional Guidance (NSRG):
   Instead of in-painting, the authors introduce a regional guidance mechanism that constrains the MDE model's predictions within the gradient domain. Based on the prior that most ToM objects are continuous planes, the method enforces consistency between the predicted depth gradient and the ground truth gradient within the masked non-Lambertian regions.
   The loss function, $\mathcal{L}_{ToM}$, utilizes a robust estimator to align the gradients of the predicted depth ($\nabla D$) and the ground truth ($\nabla D^\star$). It computes scale ($s$) and shift ($t$) coefficients to normalize the gradients, removing outliers by trimming the top 20% of residuals. The final loss minimizes the $L_1$ distance between the normalized gradients, ensuring the predicted depth plane remains smooth and consistent with the ground truth in ToM regions.

3. Diverse Lighting Images Fusion (DLIF) (Optional):
   For scenarios where multiple exposure images of the same scene are available, the authors propose an optional fusion module. This module utilizes a pre-trained Variational Autoencoder (VAE) from Stable Diffusion to encode multiple images with different lighting conditions into latent features. By averaging these latent features and decoding them, the system generates a single fused RGB image that theoretically possesses the optimal lighting conditions for depth estimation, leveraging the VAE's semantic priors.

Key Contributions

• Random Tone-mapping Augmentation: A data augmentation strategy that enriches the training dataset with varying lighting conditions, significantly improving the MDE model's robustness and zero-shot generalization to diverse lighting scenarios.
• Non-Lambertian Surface Regional Guidance: A novel training constraint that operates in the gradient domain. It directly guides the network to estimate correct depth planes for non-Lambertian surfaces using segmentation masks, avoiding the instability of RGB in-painting methods.
• Optional Image Fusion Module: A mechanism utilizing VAEs to fuse multi-exposure images, providing an advantageous input for depth estimation when multiple lighting conditions are available.

Experimental Results
The method was evaluated on the Booster dataset and the NYU Depth Dataset V2 (refined by Mirror3D), as well as the TRICKY2024 competition test set.

• Quantitative Performance: Compared to the Depth Anything V2 baseline, the proposed method achieved accuracy improvements of 33.39% on the Booster dataset and 5.21% on the Mirror3D dataset within ToM regions (measured by $\delta_{1.05}$).
• State-of-the-Art: On the TRICKY2024 competition test set, the method achieved a $\delta_{1.05}$ score of 90.75 within ToM regions, demonstrating state-of-the-art performance.
• Ablation Studies: Experiments confirmed that both random tone-mapping augmentation and non-Lambertian surface regional guidance independently contribute to performance gains, with the combination yielding the best results. The optional image fusion module further improved results when multi-exposure inputs were utilized.

Significance and Limitations
The paper claims that its primary significance lies in shifting the paradigm from post-processing in-painting to direct learning of non-Lambertian characteristics through a well-designed training framework. By focusing on gradient consistency and lighting robustness, the method simplifies the pipeline and enhances stability without relying on empirical adjustments for random colors.

The authors acknowledge limitations, noting that in cases of extremely overexposed images where textures of non-Lambertian surfaces (like glass) are completely lost, the network may still fail. They suggest that future work could integrate image quality optimization and semantic information fusion to further improve qualitative and quantitative results in such extreme conditions.