💻 computer science

Imaging Hidden Objects with Consumer LiDAR via Motion Induced Sampling

This paper presents a multi-frame fusion strategy based on a motion-induced aperture sampling model that enables non-line-of-sight imaging, including 3D reconstruction and object tracking, using affordable, off-the-shelf consumer LiDAR devices without requiring extensive setup.

Original authors: Siddharth Somasundaram, Aaron Young, Akshat Dave, Adithya Pediredla, Ramesh Raskar

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Siddharth Somasundaram, Aaron Young, Akshat Dave, Adithya Pediredla, Ramesh Raskar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a hallway, and there is a large box blocking your view of a room behind you. You can't see the room, but you know there's a wall right next to you. If you shine a flashlight at that wall, the light bounces off, hits the hidden room, bounces back to the wall, and then returns to your eyes.

Normally, this "echo" of light is too faint and messy for a standard camera to make sense of. It's like trying to hear a whisper in a noisy stadium. However, this paper shows how to turn a cheap, everyday smartphone LiDAR (the sensor used for AR games and depth sensing) into a device that can "see" around corners.

Here is how they did it, explained through simple analogies:

1. The Problem: The "Faint Whisper"

Consumer LiDARs are small and safe for our eyes, which means they use very weak lasers. When light bounces off a hidden object and comes back, the signal is incredibly weak (low signal-to-noise ratio) and blurry. If you take just one snapshot, it's like trying to guess what a song sounds like by listening to a single, static-filled second of it. You can't tell what's going on.

2. The Solution: "Burst Photography" and the "Virtual Mirror"

The researchers realized that while one snapshot is useless, a video of many snapshots is powerful. They used two main tricks:

The Virtual Mirror: They treat the wall in front of the camera as a giant, invisible mirror. Even though the wall is rough, the math treats it as if it were a perfect mirror reflecting the hidden room.
Burst Photography: Just like how photographers take a rapid burst of photos to combine them into one super-clear image, this system takes many rapid measurements. By combining them, the "noise" cancels out, and the "whisper" becomes a clear voice.

3. The Secret Sauce: The "Motion-Induced Aperture"

This is the paper's biggest innovation. Imagine you are trying to hear a conversation in a dark room.

If you stand still: You only hear the sound from one angle.
If you walk around: You hear the sound from many different angles. This gives you a much better idea of where the speakers are and what they look like.

The paper calls this Motion-Induced Aperture Sampling.

The Camera Moving: When you hold the phone and move it, you are essentially creating a "synthetic aperture" (a giant virtual lens) by combining all the different angles you captured.
The Object Moving: If the hidden object is moving (like a person walking), the system uses that movement to its advantage, gathering more data points over time.

They created a mathematical model that treats the shape of the object, the movement of the object, and the movement of the camera as a single, unified puzzle.

4. What They Actually Achieved

Using this method on a standard smartphone LiDAR (costing less than $100), they demonstrated three specific things:

3D Reconstruction (The "Ghost Sculptor"): If you move your phone around a hidden, static object (like a mannequin), the system can build a 3D model of it, even though you never saw it directly. It's like sculpting a statue by feeling the air currents around it rather than touching the statue itself.
Tracking (The "Invisible Chaser"): If a hidden object is moving (like a ball rolling behind a wall), the system can track its path in real-time. They even showed it tracking a hand moving behind a wall, predicting where the hand is even when it's out of sight.
Camera Localization (The "Blind Navigator"): If you are in a featureless white room where normal cameras get lost (because there are no corners or textures to grab onto), this system can use hidden objects as landmarks to tell you exactly where you are. It's like navigating a cave by listening to the echo of a hidden bell rather than looking at the walls.

5. The "Plug-and-Play" Revolution

Previously, doing this kind of "Non-Line-of-Sight" (NLOS) imaging required massive, expensive lab equipment that took days to set up and calibrate.

This paper claims to have democratized the technology. They showed that you can do this with off-the-shelf hardware, no complex setup, and in real-time. It turns a $100 sensor into a tool that can see the invisible, making it possible for robots to avoid collisions around corners or for AR apps to understand the full 3D space around a user, not just what is directly in front of the lens.

In short: They figured out how to turn the "noise" of a cheap phone sensor into a clear picture of the hidden world by mathematically combining many shaky, blurry snapshots taken while moving.

Technical Summary: Imaging Hidden Objects with Consumer LiDAR via Motion Induced Sampling

Problem Statement

Non-line-of-sight (NLOS) imaging, which enables the visualization of objects hidden from a camera's direct field of view, has traditionally been restricted to bulky, expensive research-grade hardware requiring extensive calibration. While time-of-flight (ToF) sensors are increasingly integrated into consumer devices (smartphones, AR headsets, robots), applying NLOS imaging to these platforms presents three primary challenges:

Low Signal-to-Noise Ratio (SNR): Caused by eye-safety constraints limiting laser power and short exposure times required for dynamic scenes.
Aperture vs. Resolution Trade-off: Consumer sensors have low spatial resolution and limited bandwidth, forcing a compromise between the size of the virtual aperture and the density of sampling.
Motion Artifacts: The joint motion of the camera and the hidden object introduces motion blur, which degrades reconstruction quality if not explicitly modeled.

Existing NLOS methods often assume idealized, static conditions or single-shot reconstruction, which are insufficient for the noisy, low-resolution, and dynamic data captured by consumer LiDARs.

Methodology

The authors propose a Motion-Induced Aperture Sampling (MAS) model and a multi-frame fusion strategy to address these constraints.

1. Motion-Induced Aperture Sampling (MAS) Model

The core theoretical contribution is the MAS model, which unifies object shape, object motion, and camera motion into a single measurement framework.

Light-Cone Transform (LCT): The model utilizes the LCT to transform the space-time impulse response (STIR) of the hidden scene. In this transformed space, the relationship between the object's volumetric albedo and the measured STIR becomes a 3D convolution.
Decomposition: The model decomposes the time-varying STIR into a canonical STIR (representing the object's shape, which is time-invariant) and a time-dependent translation (representing the object's position).
Sampling: The camera acts as a spatial sampling function. The measured signal at any time $t$ is modeled as the canonical STIR, translated by the object's motion, and sampled by the camera's pose. This allows the system to treat the problem as recovering a single unknown variable (shape, object position, or camera position) while treating the others as known or constrained.

2. Multi-Frame Fusion via Particle Filtering

To overcome low SNR and resolution limits, the authors employ particle filtering rather than single-shot reconstruction. This approach leverages:

Burst Photography Principles: Exploiting viewpoint redundancy across multiple frames to improve SNR.
Synthetic Aperture Radar Principles: Exploiting viewpoint diversity to improve reconstructed resolution.
Sequential Estimation: The algorithm estimates the posterior distribution of the unknown variable (e.g., object position) by:
1. Propagation: Predicting the state based on a motion prior (e.g., constant velocity).
2. Evaluation: Scoring particles by rendering the expected measurement using the MAS model and comparing it to the actual noisy measurement via a normalized dot product.
3. Resampling: Eliminating inconsistent particles and retaining those that align with the data, effectively modeling uncertainty and ambiguity.

3. Specific Applications

The framework is applied to three distinct tasks by fixing two of the three unknowns (shape, object position, camera pose):

3D Reconstruction: Assumes known camera pose and static object; recovers the object's shape by fusing measurements from handheld camera motion.
Object Tracking: Assumes known object shape; recovers the position of single or multiple hidden objects in real-time.
Camera Localization: Assumes known static hidden object; recovers the camera's 2D translation (and potentially 6D pose) using the hidden object as a visual landmark, particularly useful in textureless environments where standard visual odometry fails.

Key Results

The authors demonstrate these capabilities on a smartphone-grade LiDAR (approx. 100 pixels, ~$100 hardware cost) with no additional calibration beyond standard camera calibration.

3D Reconstruction: Successful reconstruction of static hidden objects (including mannequins and diffuse objects) using natural handheld motion.
Tracking: Real-time tracking of single and multiple hidden objects, including human hands. The system handles multi-object tracking by clustering particle distributions.
Localization: Accurate camera localization in environments with planar, textureless walls where traditional methods (ICP, SfM) fail due to lack of features.
Diffuse Objects: While the model assumes retroreflective properties for ideal confocal imaging, empirical results show the system works on diffuse objects, albeit with lower SNR and higher ambiguity.
Hardware Agnosticism: The method was successfully demonstrated on a commercially available ST VL53L8CX sensor with minimal setup.

Significance and Claims

The paper claims to represent a significant shift from "research-grade" NLOS imaging to plug-and-play consumer NLOS imaging.

Democratization: The work demonstrates that NLOS capabilities, previously requiring tens of thousands of dollars and hours of calibration, can be achieved with off-the-shelf hardware (<$100) and seconds of setup.
Practicality: By addressing the specific constraints of consumer sensors (low SNR, motion, resolution), the authors enable practical applications in robotics (navigation around corners), augmented reality (robust body pose mapping), and localization in featureless environments.
Modest Limitations: The authors acknowledge that the current model assumes rigid-body translation and does not fully account for complex reflectance properties (BRDFs) or non-rigid deformations. They note that ambiguity increases with object distance from the virtual aperture and that solving for all three unknowns (shape, object motion, camera motion) simultaneously remains a challenge due to degenerate cases.

In summary, the paper establishes that consumer LiDARs, when paired with a motion-induced sampling model and multi-frame fusion, can effectively image hidden objects, marking a transition toward accessible, real-time NLOS imaging for everyday applications.