🔬 optics

Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering

This paper proposes a novel non-line-of-sight imaging pipeline that utilizes 3D Gaussian primitives and differentiable transient rendering to achieve state-of-the-art reconstruction of hidden scenes under spatially limited, sparsely sampled conditions with arbitrary relay surface geometries, eliminating the restrictive planar-wall assumptions of existing methods.

Original authors: Yi Wang, Ziyu Zhan, Yuran Wang, Hao Wang, Qiang Liu, Zuoqiang Shi, Lingyun Qiu, Xing Fu

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yi Wang, Ziyu Zhan, Yuran Wang, Hao Wang, Qiang Liu, Zuoqiang Shi, Lingyun Qiu, Xing Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Seeing Around Corners Without a Flat Wall

Imagine you are trying to see a person hiding behind a corner (like a pedestrian stepping out from behind a parked car). You can't see them directly, but you can see the side of the car. If you shine a super-fast laser at the car, the light bounces off the car, hits the hidden person, bounces back to the car, and then reflects into your camera. By measuring exactly how long that tiny trip takes, you can figure out where the person is. This is called Non-Line-of-Sight (NLOS) imaging.

The Problem:
Most previous methods for doing this magic trick had a strict rule: the surface you bounce the light off of (the "relay surface") had to be a large, flat wall.

Real Life Reality: In the real world, we rarely have giant flat walls. We have curved car hoods, bumpy sidewalks, or even the backs of other people.
The Failure: If you try to use old methods on a curved or weirdly shaped surface, the math breaks, and the image comes out blurry or disappears entirely. It's like trying to play a piano with a hammer; the tool just isn't built for the shape of the keys.

The Solution: "3D Gaussian Transient Rendering" (3D-GTR)

The authors propose a new way to solve this that doesn't care what shape the wall is. They call their method 3D-GTR. Here is how it works, broken down into three simple steps:

1. The "Smart Flashlight" (LOS-Guided)

Before trying to see the hidden object, the system first takes a quick look at the surface it can see (the "Line of Sight" or LOS).

Analogy: Imagine you are in a dark room trying to find a hidden object. Before you start guessing, you shine a flashlight on the table in front of you to see exactly how the table is shaped. Is it flat? Is it bumpy? Is it curved?
What it does: The system scans the visible surface to create a 3D map of its shape and angles. This map tells the computer exactly how light will bounce off that specific, weird surface.

2. The "Cloud of Dots" (3D Gaussian Primitives)

Instead of trying to build a solid 3D model of the hidden object (like a digital statue), the system represents the hidden scene as a cloud of fuzzy, glowing dots.

Analogy: Think of a cloud of smoke. You can't see individual water droplets, but you can see the shape of the cloud. The system uses thousands of these "fuzzy dots" (called 3D Gaussians) to represent the hidden person or object. Each dot has a position, a size, an orientation, and a brightness.
Why this helps: Because these dots are flexible, they can mold themselves to fit any shape, whether the hidden object is a straight line or a curvy statue.

3. The "Virtual Simulator" (Differentiable Rendering)

This is the magic engine. The system runs a simulation in reverse.

The Process:
1. It guesses where the hidden dots are.
2. It simulates the laser light hitting the visible surface, bouncing to the hidden dots, and coming back.
3. It compares its simulation to the actual data the camera collected.
4. If the simulation doesn't match the real data, it automatically tweaks the position, size, and brightness of the fuzzy dots.
The Result: It keeps adjusting the dots over and over (in a split second) until the simulation perfectly matches the real-world measurements. Once the dots are in the right place, the computer can "see" the hidden object clearly.

Why This Paper is a Big Deal

The authors tested this on two types of setups:

Standard Flat Walls: Even on flat walls, their method was faster and sharper than existing technologies, especially when the data was sparse (like having only a few dots of light instead of a full scan).
Weird Shapes (The Real Test): They tested it on curved surfaces and even used the backs of two mannequins (people dummies) as the relay surface.
- The Result: Old methods failed completely on the mannequins. The new method successfully reconstructed the hidden object, proving it works even when the "wall" is curved and irregular.

Summary in One Sentence

This paper introduces a new camera trick that uses a "cloud of fuzzy dots" and a smart simulation engine to see hidden objects, allowing it to work perfectly even when the surface it bounces light off of is curved, bumpy, or shaped like a person, rather than requiring a perfect flat wall.

Technical Summary: Non-line-of-sight imaging with arbitrary relay surface geometries via 3D Gaussian Transient Rendering

Problem Statement
Non-line-of-sight (NLOS) imaging aims to reconstruct hidden objects by analyzing light scattered from a visible relay surface. While Time-of-Flight (ToF) based NLOS imaging has advanced significantly, its real-world deployment is hindered by restrictive assumptions in existing methods. Most state-of-the-art techniques assume a large, continuous, and planar relay wall with dense, uniform sampling. In practical scenarios—such as autonomous driving or robotic perception—measurements are often acquired over spatially limited regions with arbitrary, non-planar geometries (e.g., the backs of pedestrians or irregular walls). Current approaches, including model-based iterative frameworks and learning-based neural fields, struggle under these conditions, often requiring prolonged optimization times, lacking robustness to complex geometries, or failing to resolve targets when the relay surface is unconstrained.

Methodology
The authors propose a LOS-guided NLOS imaging pipeline that eliminates geometric assumptions regarding the relay surface. The core of the method is 3D Gaussian Transient Rendering (3D-GTR), which integrates the following components:

LOS-Guided Geometry Recovery: The system first utilizes Line-of-Sight (LOS) ToF measurements to reconstruct a point cloud of the relay surface. This point cloud is post-processed to estimate local surface normals, creating a "virtual relay surface" that accurately reflects the arbitrary geometry of the real-world measurement environment.
3D Gaussian Representation: The hidden scene is represented not by a volumetric grid or implicit neural field, but by a set of 3D Gaussian primitives. Each primitive is parameterized by a mean position ( $\mathbf{x}_g$ ), a covariance matrix ( $\mathbf{\Sigma}$ ) defining spatial spread, and an albedo ( $\rho$ ) controlling scattering strength.
Differentiable Transient Rendering: A physics-based, differentiable rendering model maps these 3D Gaussian primitives to 1D temporal Gaussian kernels. This model explicitly accounts for:
- Light Transport Geometry: The optical path length from the illumination point to the primitive and then to the detector is linearized around the primitive's mean, inducing a Gaussian distribution in the path-length domain.
- Radiometric Weighting: The model incorporates the solid angle subtended by the primitive relative to the illumination and detection points, as well as the bidirectional reflectance distribution function (BRDF) effects via surface normals ( $\mathbf{n}_i, \mathbf{n}_d$ ). This results in a physically grounded radiometric weight that follows the inverse fourth-power distance dependence typical of NLOS forward models.
End-to-End Optimization: The system optimizes the parameters of the 3D Gaussian primitives (position, scale, orientation, and albedo) via backpropagation to minimize the $L_2$ loss between the rendered transients and the measured transient signals. The process supports both confocal and non-confocal acquisition configurations.
Synthesis and Reconstruction: Once optimized, the 3D Gaussian representation is used to synthesize dense, physically consistent transient measurements on a regular virtual grid. These synthesized transients are then fed into conventional solvers (e.g., Light-Cone Transform or RSD) to produce the final volumetric reconstruction.

Key Contributions

Geometry-Agnostic Pipeline: The proposed framework imposes no prior assumptions on the relay surface geometry, naturally supporting confocal and non-confocal acquisitions with arbitrary illumination-detection pair configurations.
3D Gaussian Transient Rendering (3D-GTR): The authors introduce an efficient, fully differentiable, physics-based transient rendering model built on lightweight 3D Gaussian primitives. By explicitly incorporating physics-aware light transport and surface normals, the model achieves reliable performance across diverse acquisition setups.
Robustness and Efficiency: The method is validated on real-world measurements from both public datasets and a custom-built capture system. It demonstrates significantly improved robustness and reconstruction fidelity compared to existing methods (e.g., NeTF, CC-SOCR, FBP), particularly under spatially limited, sparsely sampled, and arbitrarily shaped relay surface conditions.

Experimental Results
The paper evaluates the method across three challenging sampling regimes:

Region-Limited: In scenarios with sparse spatial sampling (e.g., "NLOS" shaped masks or random 3,000 points), 3D-GTR preserves multi-object geometry and maintains high fidelity where other methods fail.
Aperture-Limited: When the measurement aperture is severely restricted (smaller than the hidden object), the method reconstructs clearer shapes than prior art.
Resolution-Limited: The method exhibits a super-resolution effect, successfully resolving 2 cm features even when the sampling pitch (4 cm) exceeds the target resolution, outperforming competing methods that degrade under undersampling.
Non-Confocal & Arbitrary Geometries: In non-confocal settings using curved relay surfaces (e.g., mannequin torsos), 3D-GTR recovers target shapes more clearly than baselines. The inclusion of surface normals in the rendering model is shown via ablation studies to be critical for handling varying surface orientations.
Runtime: The method demonstrates significant efficiency advantages, with training converging substantially faster than NeTF and inference times orders of magnitude faster than iterative optimization methods, leveraging the parallel nature of Gaussian primitives on GPUs.

Significance
The paper claims that this work establishes a novel NLOS imaging framework that shifts the paradigm from geometry-assumed pipelines to a measurement-driven framework. By leveraging LOS measurements to recover relay surface geometry and utilizing 3D Gaussian primitives for efficient, differentiable rendering, the approach bridges the gap between theoretical NLOS imaging and practical deployment in complex, real-world environments. The authors position this as a concrete step toward making NLOS imaging viable for applications such as autonomous navigation and public safety, where relay surfaces are rarely planar or dense. The work acknowledges that while the method is faster than existing baselines, it is not yet real-time, but suggests that the physically grounded, differentiable nature of the representation offers a promising pathway for future integration with video reconstruction techniques.