Target-depth sensing with metasurface-encoder integrated optoelectronic neural network

1. Problem Statement

Accurate, real-time 3D sensing is critical for emerging technologies like autonomous driving, robotics, and augmented reality. However, current state-of-the-art 3D sensing methods face significant limitations:

• Computational Burden: Traditional approaches (Stereo vision, Structured Light, LiDAR) rely on complex post-processing of multi-view images or point clouds, leading to high latency and power consumption.
• Hardware Constraints: These systems often require high-power active illumination and heavy electronic processing, which is unsuitable for resource-constrained platforms.
• The Gap: While "pre-sensor" optical computing (using diffractive optics or metasurfaces) has shown promise for 2D image processing, its application to 3D vision sensing (simultaneous depth and object recognition) remains underexplored.

2. Methodology

The authors propose a Metasurface-Encoder Integrated Optoelectronic Neural Network (MONN) architecture that shifts the computational load from the electronic domain to the optical domain.

A. Optical Encoding (The Metasurface Encoder)

• Mechanism: A custom-designed metasurface is placed in front of a standard monocular camera. It utilizes a Double-Helix Point Spread Function (DH-PSF).
• Function: The metasurface encodes depth information into the 2D image by rotating the orientation of two focal spots (a double-helix pattern) monotonically as the target distance changes.
• Design: The metasurface consists of silicon nanoposts on a quartz substrate, acting as a metalens-integrated phase mask. It was fabricated using electron-beam lithography (EBL) with a working wavelength of 785 nm.
• Range: The system is designed for a depth measurement range of 430 mm to 930 mm.

B. Electronic Decoding (The Neural Network)

• Architecture: A lightweight Shadow ResNet (adapted from ResNet-34) processes the encoded images.
• Dual-Task Learning: The network has two output heads:
  1. Classification Head: Identifies the target object (e.g., digit or vehicle type).
  2. Regression Head: Estimates the continuous depth value.
• Training Strategy:
  • Data Generation: Instead of using 3D models, the team generated training data by convolving 2D images (MNIST digits and Vehicle-Image dataset) with experimentally measured DH-PSF profiles at various depths.
  • Domain Adaptation: Initial training on purely simulated data failed due to discrepancies with real-world noise (illumination gradients, sensor noise). The team successfully retrained the network using experimentally recorded encoded images to bridge the sim-to-real gap.
  • Loss Function: A weighted sum of Cross-Entropy loss (classification) and Root-Mean-Square Error (RMSE) loss (depth regression).

3. Key Contributions

1. Novel Architecture: First demonstration of a fully integrated optoelectronic system that simultaneously performs target classification and absolute depth estimation using a single monocular camera and a passive metasurface encoder.
2. Efficiency: By encoding 3D information optically, the system eliminates the need for complex electronic 3D reconstruction algorithms, significantly reducing computational complexity and latency.
3. Experimental Validation: The system was validated not just via simulation but through rigorous physical experiments, including real-time tracking of moving targets.
4. Robustness Analysis: The study systematically evaluated performance under varying target sizes and partial occlusions, demonstrating the system's resilience.

4. Key Results

The system was tested on MNIST (handwritten digits) and Vehicle-Image datasets.

• Classification Accuracy:
  • MNIST: 99.9% accuracy.
  • Vehicles: 97.6% accuracy.
• Depth Estimation Precision:
  • MNIST: Mean Absolute Error (MAE) of 3.68 mm; RMSE of 4.53 mm.
  • Vehicles: MAE of 4.60 mm; RMSE of 6.13 mm.
  • Relative Error: Approximately 1% of the full measurement range (430–930 mm).
• Real-Time Performance:
  • The system achieved real-time tracking of moving targets.
  • While the camera frame rate was 100 FPS, the total processing cycle (including data readout and resizing) was ~100 ms. The neural network inference itself took only ~30 ms.
  • Depth trajectories of moving targets matched reference measurements (recorded by an iPhone) with high fidelity, even during constant velocity motion.
• Robustness:
  • Size Variation: Classification accuracy dropped slightly (to ~88%) for larger targets, but depth estimation remained robust.
  • Occlusion: The system maintained high accuracy with up to 35% occlusion. Performance degraded significantly only beyond 50% occlusion.

5. Significance and Impact

• Paradigm Shift: This work establishes a "meta-optic-encoder/electronic-decoder" paradigm. It proves that physics-based optical preprocessing can drastically reduce the complexity required for deep learning models, enabling high-performance 3D sensing on lightweight hardware.
• Energy and Latency: By offloading depth encoding to the metasurface, the system offers a pathway to low-latency, energy-efficient 3D perception, crucial for battery-powered autonomous systems and edge AI.
• Scalability: The framework is not limited to DH-PSF; it can be extended to other optical encoding strategies (e.g., angle-encoding) for multidimensional compression and detection.
• Practicality: The successful demonstration on real-world moving targets and under partial occlusion suggests immediate applicability in intelligent surveillance, robotic manipulation, and autonomous navigation.