Technical Summary: Physically Grounded Monocular Depth via Nanophotonic Wavefront Encoding

1. Problem Statement

Monocular depth estimation has seen significant advances through Depth Foundation Models (DFMs), which leverage large-scale data to learn rich geometric priors. However, these models inherently lack physical depth cues from a single RGB capture, rendering metric depth estimation ill-posed. This leads to ambiguities in scale and inaccuracy in applications requiring precise metric depth. While recent approaches attempt to resolve this by integrating active sensors like LiDAR, such solutions increase system complexity, form factor, and power consumption, deviating from a strict monocular setting. The core challenge addressed by this work is how to ground DFMs in physics using only passive optics within a compact, single-sensor device to achieve accurate metric depth without active illumination or additional sensors.

2. Methodology

The proposed framework integrates nanophotonic hardware with deep learning software to create a physically grounded depth sensing system. The methodology consists of three primary components:

2.1 Birefringent Metalens Hardware

The system employs a custom-designed, ultra-thin birefringent metalens (3 mm diameter) fabricated from Titanium Dioxide (TiO2) nanopillars on a glass substrate.

• Polarization Multiplexing: The metalens independently modulates the phase of $x$- and $y$-polarized light.
• Rotating Point Spread Functions (PSFs): The metalens is engineered to create depth-dependent PSFs that rotate as a function of object distance. Specifically, the phase profile is partitioned into concentric rings with varying topological charges, causing the PSF to rotate by an angle $\Delta\phi_i(z)$ that is linearly related to $1/z$ (inverse depth).
• Spatial Separation: The focusing phase is designed to introduce opposite vertical deflections for the two polarization channels, projecting the two rotated PSF images onto separate halves of a single monochrome sensor in a single exposure.
• Depth Encoding: The relative positional shift (disparity) between the two conjugate PSFs encodes metric depth. Because the shifts occur in opposite directions for the two polarizations, the disparity vector's angle tracks the depth monotonically.

2.2 Input Adaptation and Model Fine-Tuning

To leverage pretrained DFMs without architectural changes, the authors propose a simple input adaptation strategy:

• Pseudo-RGB Construction: The two polarization channels ($I_x, I_y$) are combined into a three-channel input: $(I_x, I_y, (I_x + I_y)/2)$.
• Direct Fine-Tuning: This input format matches the expected input of standard monocular depth models (specifically Depth Anything V2). The model is fine-tuned directly on this adapted input, allowing it to utilize its robust learned visual priors while simultaneously recovering metric scale from the physical optical signals. No decoder-side fusion modules or auxiliary branches are required.

2.3 Simulation-to-Real Transfer Pipeline

To overcome the scarcity of real-world paired polarization-depth data, a comprehensive simulation pipeline was developed:

• Optical Forward Model: A physics-based simulator generates polarization images from RGB-D datasets by convolving scene slices with depth-dependent PSFs.
• Disocclusion-Aware Modeling: Standard linear convolution models fail at depth discontinuities, creating artifacts like bright occlusion edges and dark disocclusion gaps. The authors introduce a specialized module that detects depth discontinuities, performs edge extrapolation, and reconstructs missing background information using alpha compositing.
• Polarization-Aware Augmentation: To bridge the sim-to-real gap, the training data is augmented with global scaling, local brightness perturbations, noise injection, and Gaussian blur to simulate fabrication imperfections and environmental variations.
• Few-Shot Real Adaptation: A small number of manually annotated real-world samples are mixed into the training set to further align the model with physical reality.

3. Key Contributions

1. Physically Grounded Framework: Introduction of a novel approach for monocular metric depth estimation using a birefringent metalens that encodes depth into polarization-dependent PSF rotations, enabling direct fine-tuning of DFMs.
2. Disocclusion-Aware Simulator: Development of an optical forward model that accurately simulates asymmetric PSF formation and handles disocclusion artifacts, significantly reducing the simulation-to-real gap.
3. Integrated Hardware-Software System: Demonstration of a compact, passive depth sensing system that achieves high metric accuracy by synergizing physical optical cues with learned depth priors, outperforming both standard monocular estimators and depth-from-defocus baselines.

4. Experimental Results

The authors evaluated the system through both extensive simulations and physical experiments.

• Simulation Performance: On standard benchmarks (NYU Depth V2, MIT-CGH-4k, MPI Sintel, Hypersim), the method outperformed all LiDAR-free monocular metric depth baselines (including Depth Anything v2/v3, Depth Pro, Marigold, and UniDepth v2). Notably, it achieved performance comparable to PromptDA, a state-of-the-art method that relies on LiDAR as an auxiliary sensor.
• Physical Experiments: Using a prototype camera with a 3-mm metalens and a 20-MP sensor, the system was tested on 42 scenes. The method consistently outperformed LiDAR-free baselines and maintained a clear margin over them even after optimal scale/shift alignment. It demonstrated metric accuracy and preserved sharp object boundaries.
• Comparison with Depth-from-Defocus (DfD): The system significantly outperformed specialized DfD baselines (DeepDfD, Split-Aperture) across all metrics, attributed to the combination of the engineered rotating PSF and the powerful DFM backbone.
• Ablation Studies:
  • Pretrained Priors: Removing the pretrained DFM initialization caused a steep drop in accuracy, confirming that the learned priors are the primary driver of performance.
  • Optical Design: The specific metalens design provided superior physical grounding compared to standard DfD optics.
  • Input Adaptation: The simple input adaptation strategy proved more effective and efficient than decoder-side fusion modules.
  • Sim-to-Real: Removing the disocclusion handling module or data augmentations significantly degraded performance, validating the necessity of the proposed pipeline.

5. Significance and Claims

The paper claims that this work establishes an effective pathway for accurate monocular metric depth sensing by bridging the gap between nanophotonics and foundation models. By physically encoding missing metric cues via passive optics, the system resolves the inherent scale ambiguity of monocular capture without the need for active sensors or complex multi-sensor setups.

The authors position this technology as a viable solution for applications requiring compact, low-power 3D vision, specifically citing VR/AR, miniature robotics, medical endoscopy, and other embedded 3D vision systems. They emphasize that the approach leverages the "best of both worlds": the high-resolution, semantic understanding of large-scale pretrained models and the physical rigor of nanophotonic wavefront encoding.

The work is presented as a proof of concept for "physically grounding" foundation models, suggesting a future direction where optical hardware and AI models are co-designed to overcome the limitations of purely data-driven perception.