SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification

1. Problem Statement

Monocular depth estimation is a fundamental task for 3D reconstruction, robotics, and navigation. However, it faces severe challenges when applied to transparent objects (e.g., glass, water, plastic).

• The Core Issue: Standard depth estimation models rely on photometric assumptions (e.g., Lambertian reflectance, consistent texture) that are violated by transparent materials. Refraction, transmission, and reflection cause the appearance of transparent objects to decouple from their underlying geometry.
• Current Limitations:
  • Foundation Models: State-of-the-art (SOTA) general-purpose depth models (e.g., Depth Anything, Marigold) fail on transparent objects, often predicting them as translucent or merging them with the background.
  • Specialized Models: Existing methods tailored for transparent objects (e.g., Depth4ToM, DKT) require large-scale, paired RGB-D datasets specific to transparent materials. They are often model-dependent, requiring retraining whenever the backbone architecture changes, and struggle to generalize to "in-the-wild" scenarios.

2. Methodology: The SeeClear Framework

The authors propose SeeClear, a plug-and-play front-end framework that enables existing, off-the-shelf monocular depth estimators to handle transparent objects without retraining. The core idea is Generative Opacification: converting the refractive appearance of transparent objects into geometrically consistent opaque images before depth estimation.

The framework operates in a two-stage pipeline:

A. Transparent Object Localization

• A segmentation model (combining Trans4Trans for initial localization and SAM 3 for refinement) generates a binary mask ($M_{seg}$) identifying the transparent regions in the input image ($I_{tr}$).

B. Generative Opacification (Diffusion Module)

• Goal: Transform the refractive appearance within the mask into a diffuse, opaque appearance while preserving the object's geometry, boundaries, and shading.
• Architecture: A Conditional Latent Diffusion Model (based on Paint-by-Example).
  • Input: The noisy latent representation, the encoded input image ($I_{tr}$), and the downsampled segmentation mask ($M_{seg}$).
  • Conditioning: Uses CLIP image encoders (specifically the class token) to provide semantic guidance, ensuring the generated object retains its identity and shape.
  • Training Objective: Combines standard DDPM noise prediction loss ($L_{LDM}$) with a Perceptual Loss ($L_{LPIPS}$) computed in pixel space using VGG features. This ensures high-frequency texture fidelity and structural consistency, which $L_2$ loss alone cannot guarantee.
  • Output: A generated opaque image ($I_{pred}$) of the object.

C. Mask Refinement and Compositing

• Problem: Directly pasting the generated image can cause boundary halos or color drift due to VAE decoding artifacts or segmentation inaccuracies.
• Solution: A lightweight Mask Refinement Module (MRM) predicts a soft blending mask ($\hat{M}_{refine}$).
• Compositing: The final input for the depth estimator is created via alpha blending:
  $$I_{blend} = \hat{M}_{refine} \odot I_{pred} + (1 - \hat{M}_{refine}) \odot I_{tr}$$
  This seamlessly integrates the generated opaque object with the original background.

D. Depth Estimation

• The blended image ($I_{blend}$) is fed into a standard, pre-trained monocular depth estimator (e.g., Depth Anything V3 or MoGe-2) without any fine-tuning.

3. Key Contributions

1. Novel Framework (SeeClear): A plug-and-play approach that reframes transparent depth estimation as an appearance opacification problem. It allows foundation depth models to achieve high accuracy on transparent objects without architectural changes or retraining.
2. Generative Opacification Module: A diffusion-based model that learns to map refractive appearances to geometry-consistent opaque textures, preserving object boundaries and shading cues essential for depth inference.
3. SeeClear-396k Dataset: A large-scale synthetic dataset containing 396,000 paired transparent-opaque renderings.
   • Generated using Blender with 11 object types across 10 indoor scenes.
   • Includes aligned ground truth for depth, normals, and masks.
   • Designed to capture failure modes like refraction, textureless surfaces, and complex occlusions.
4. Comprehensive Evaluation: Extensive validation on both synthetic (TransPhy3D) and real-world (ClearGrasp) benchmarks, demonstrating superior performance over specialized and general baselines.

4. Experimental Results

The method was evaluated on ClearGrasp (real-world) and TransPhy3D (synthetic) datasets using metrics like AbsRel, RMSE, and $\delta < 1.25$.

• Performance on ClearGrasp (Real-World):

  • SeeClear significantly outperformed all baselines, including specialized models like DKT (which was fine-tuned on the test set) and general models like Depth Anything V3.
  • Example: Using Depth Anything V3 as the backbone, SeeClear reduced RMSE from 24.03 mm to 21.45 mm (10.7% improvement).
  • Using MoGe-2, the improvement was even more drastic, reducing RMSE from 56.83 mm to 26.62 mm.
  • It achieved the best results on non-transparent regions ("Other") as well, proving it does not degrade performance on standard scenes.

• Performance on TransPhy3D (Synthetic):

  • SeeClear with MoGe-2 achieved the best performance on 6 out of 8 metrics, surpassing even the fine-tuned DKT model in AbsRel (0.012 vs. 0.014).
  • It demonstrated robustness in handling complex occlusions and multi-layer layouts.

• Qualitative Results:

  • Baseline models often predicted transparent objects as "ghostly" or translucent, failing to capture surface geometry (e.g., bottle openings, liquid levels).
  • SeeClear produced sharp, geometrically accurate depth maps, correctly identifying hollow regions and surface contours.

5. Ablation Studies

The authors validated the necessity of each component:

• Localization: Using a precise segmentation mask ($M_{seg}$) was critical; bounding boxes or point heatmaps led to over/under-opacification and higher errors.
• Semantic Conditioning: CLIP class tokens provided better geometric consistency than full tokens or DINOv2 features.
• Loss Functions: The inclusion of Perceptual Loss ($L_{LPIPS}$) was vital for capturing structural details, outperforming $L_1$ and gradient-based losses.
• Compositing: The Mask Refinement Module (MRM) and alpha blending were essential to prevent boundary artifacts and background drift.
• Opacification Strategy: Replacing the generative module with simple solid-color inpainting resulted in the worst performance (RMSE 33.09 mm), proving that restoring plausible shading and texture is necessary for depth models to function.

6. Significance and Future Work

Significance:
SeeClear bridges the gap between the rapid evolution of foundation depth models and the specific needs of transparent object perception. By decoupling appearance correction from depth inference, it offers a scalable solution that does not require collecting massive new datasets or retraining heavy models for every new architecture.

Future Directions:
The authors plan to extend the framework to:

• Video-based depth estimation (incorporating temporal consistency).
• More complex non-Lambertian materials (specular, translucent, multi-layer objects).
• Improving robustness under heavy occlusions.