Technical Summary: Seeing through boxes: Non-Line-of-Sight 3D Reconstruction from Radar Signals (GeRaF 2.0)

1. Problem Statement

Reconstructing 3D object geometry from Radio Frequency (RF) signals is fundamentally challenging due to the lensless nature of RF sensing, which results in low spatial resolution, high noise, and signal artifacts. While RF signals possess the unique ability to penetrate occlusions (Non-Line-of-Sight or NLoS), existing neural reconstruction methods struggle to produce accurate geometries.

Previous approaches, such as the authors' prior work GeRaF [31], attempted to reconstruct NLoS scenes by simply cropping out the occluding object (e.g., a box) and treating the scene as if the occlusion did not exist. This approach fails because it ignores the physical reality that wireless signals interacting with the Line-of-Sight (LoS) occluder (the box) are partially reflected and attenuated before reaching the hidden object. Consequently, these methods suffer from three primary limitations:

1. Inaccurate Reconstruction: Reflections from visible surfaces "leak" into the hidden region, creating noise and artifacts (e.g., a "hat" shape on a bunny caused by the box).
2. Unstable Optimization: Different LoS geometries alter the optimization landscape, leading to convergence failures.
3. Surface Ambiguity: The inability to normalize signal strength makes it difficult to determine the true surface location (the zero-level set of the Signed Distance Field or SDF), often resulting in surfaces being offset by several centimeters.

2. Methodology: GeRaF 2.0

The paper proposes GeRaF 2.0, a Unified LoS and NLoS (ULoS) neural geometry reconstruction framework. The core philosophy is to leverage stable, high-fidelity visual information from the visible (LoS) region to guide the reconstruction of the hidden (NLoS) region.

2.1 ULoS Scene Representation

The framework models the entire scene (both visible and hidden) as a series of nested, closed, and compact sets within a single ULoS Signed Distance Field (ULoS SDF).

• Unified Domain: The scene is defined as $\Omega_{\text{ULoS}} = \Omega_{\text{LoS}} \cup \Omega_{\text{NLoS}}$.
• Nested Sets: The space is represented by layers $M_1, \dots, M_n$, where $M_n \subset \text{int}(M_{n-1}) \dots \subset \text{int}(\Omega_{\text{ULoS}})$.
• Sign Semantics: Unlike traditional SDFs where "negative" implies inside an object, the ULoS SDF assigns signs based on radio frequency interaction strength. Regions with strong RF interaction (e.g., the box walls, the object) are assigned negative values, while regions with weak interaction (e.g., air) are positive. This allows the field to represent multiple surfaces (inner/outer box, object) within a single continuous field.

2.2 ULoS Lensless Rendering

To address the instability of training with noisy RF data, the authors introduce a rendering technique that incorporates a vision-pretrained SDF ($f_v$) as a prior.

• Shared Geometry: In the LoS region (outside the box), the RF-trained ULoS SDF ($f_r$) and the vision-trained SDF ($f_v$) are identical.
• Transmittance Adjustment: Instead of computing transmittance from scratch for every ray (which is computationally expensive and prone to error), the system uses the vision-pretrained SDF to adjust the accumulated transmittance of the primary rays. This provides a stable initialization for the RF training, ensuring that the optimization starts from a physically consistent geometric state.

2.3 Two-Stage Training Pipeline

To solve the surface ambiguity problem (where the zero-level set is incorrectly offset), the authors propose a two-stage optimization process:

• Stage 1: Initialization and Geometry Learning

  • The Reflectivity Network is frozen and initialized to a constant value (1.0).
  • The SDF Network is initialized using the vision-pretrained SDF.
  • Training minimizes the loss between the predicted and ground-truth Matched Filter (MF) power distributions, combined with Eikonal regularization to enforce a valid SDF.
  • The vision-pretrained SDF is used to adjust transmittance during rendering.

• Stage 2: RSDF Alignment

  • All networks are trained jointly.
  • A new loss term, RSDF Alignment, is introduced. The authors define a Relative SDF (RSDF), $g_r$, which is the RF-trained SDF offset by an unknown constant.
  • Theoretical Basis: If the gradients of two scalar fields are identical and the fields match on a closed surface, the fields are identical everywhere.
  • Implementation: The system aligns the RF-trained RSDF ($g_r$) with the vision-trained SDF ($f_v$) on the outer surface of the box ($\partial M_1$). This is achieved by minimizing the difference in expected depth along primary rays between the vision and RF modalities.
  • This alignment forces the RF-trained SDF to converge to the correct zero-level set, effectively transferring the geometric accuracy of the vision model to the hidden NLoS region.

3. Key Contributions

1. Unified Representation: Introduction of the ULoS SDF, which unifies LoS and NLoS regions into a single neural field with physically meaningful sign semantics for RF propagation.
2. ULoS Lensless Rendering: A rendering technique that leverages vision-pretrained SDFs to provide stable initialization and transmittance adjustment, mitigating LoS-induced artifacts in NLoS reconstruction.
3. RSDF Alignment: A novel two-stage training strategy that uses the known geometry of the LoS region (the box) to align the RF-trained SDF, resolving the surface ambiguity problem and enabling the extraction of accurate zero-level sets without manual thresholding.
4. State-of-the-Art Performance: The framework achieves stable training and physically consistent reconstruction of both visible and hidden geometry, outperforming previous RF-based methods.

4. Experimental Results

The authors evaluated GeRaF 2.0 using a 77 GHz mmWave radar mounted on a robotic arm, imaging various real-world objects (bunny, elephant, deer, chicken, boat, ball) inside different boxes.

• Quantitative Performance: GeRaF 2.0 significantly outperformed baselines (Matched Filter imaging, GeRaF [31], and vision-based NeuS [48] on occluded scenes) in terms of F1-score and Chamfer Distance (CD). For example, on the "Bunny" dataset, GeRaF 2.0 achieved an F1-score of 0.962 and a CD of 0.15 mm, compared to GeRaF's 0.822 and 4.24 mm.
• Qualitative Improvements: The method produced clean, accurate surfaces with correct zero-level sets. Unlike previous methods that required manual selection of surface levels (e.g., $g_r = 40$ mm), GeRaF 2.0 correctly identified the surface at $g_r = 0$ mm, preserving fine details like the chicken's comb and the elephant's tusks.
• Robustness: The system maintained high-fidelity reconstruction even when the box was filled with bubble wrap (introducing scattering layers) or when reconstructing nested boxes.
• Ablation Studies:
  • Removing ULoS Lensless Rendering resulted in slower convergence and less accurate shape formation.
  • Removing RSDF alignment ($\lambda_{\text{RSDF}} = 0$) resulted in surfaces significantly offset from the zero level, confirming the necessity of the alignment stage for resolving surface ambiguity.

5. Significance

The paper claims to present the first work that utilizes line-of-sight modalities to boost non-line-of-sight 3D reconstruction, setting a new state-of-the-art in RF-based geometry reconstruction. By bridging the gap between visible and hidden regions, GeRaF 2.0 addresses the fundamental instability and ambiguity inherent in RF sensing. The framework enables robots and smart devices to accurately perceive hidden objects behind occlusions, moving beyond the limitations of previous methods that treated occlusions as non-existent or failed to model their physical interaction with RF signals. The work demonstrates that integrating visual priors into neural RF reconstruction is a viable path toward robust, high-resolution 3D sensing in challenging visibility conditions.