💻 computer science

DENALI: A Dataset Enabling Non-Line-of-Sight Spatial Reasoning with Low-Cost LiDARs

This paper introduces DENALI, the first large-scale real-world dataset of time-resolved LiDAR histograms from low-cost sensors, demonstrating that data-driven inference can enable accurate non-line-of-sight perception of hidden objects despite severe hardware limitations.

Original authors: Nikhil Behari, Diego Rivero, Luke Apostolides, Suman Ghosh, Paul Pu Liang, Ramesh Raskar

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikhil Behari, Diego Rivero, Luke Apostolides, Suman Ghosh, Paul Pu Liang, Ramesh Raskar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are standing in a dark room, and there is a secret object hidden behind a wall. You can't see it, and you can't touch it. But, if you shout, the sound bounces off the wall, hits the hidden object, bounces back to the wall, and then returns to your ear. Even though you never saw the object, the echo tells you exactly what shape it is and where it is.

This paper, titled DENALI, is about teaching our phones and robots to "hear" these echoes using light instead of sound.

Here is the story of how they did it, explained simply:

1. The Problem: The "Lazy" Flashlight

Most modern smartphones and robots have a special sensor called a LiDAR. Think of it as a super-fast, silent flashlight that shoots out a grid of laser beams to measure how far away things are.

How it usually works: The sensor shoots a laser, waits for the light to bounce back, and says, "Okay, that wall is 3 feet away." It throws away everything else. It's like taking a photo and only keeping the average brightness of the whole picture, throwing away all the details.
The hidden treasure: Inside that sensor, there is actually a much richer story. The light doesn't just bounce once; it bounces off the wall, hits the hidden object, bounces back to the wall, and then comes to the sensor. These "triple-bounce" light particles carry a secret code about the hidden object. But because the sensors are cheap and fast, they usually ignore this code and just give us a simple distance number.

2. The Solution: The "DENALI" Dataset

The researchers at MIT and Berlin wanted to see if we could teach computers to read that secret code using cheap, everyday sensors. To do this, they built a massive library of data called DENALI.

The Experiment: They set up a room with a "relay wall" (like a whiteboard). They placed a hidden object behind the wall (out of the camera's direct sight). They used a cheap LiDAR sensor (the kind found in iPhones) to look at the wall.
The Magic: Even though the sensor couldn't see the object directly, the light hit the wall, bounced to the object, and came back. The sensor recorded the exact timing of every single photon (particle of light) that returned.
The Scale: They didn't just do this once. They did it 72,000 times! They used 60 different shapes (letters, numbers, weird blobs), moved them to 100 different spots, and changed the lighting. They created a "digital twin" (a perfect 3D computer simulation) for every single photo to compare against the real world.

3. The Results: Teaching the Computer to "See" the Invisible

Once they had this massive library of "echoes," they taught an AI (a type of computer brain) to look at the data and guess what was hidden.

What the AI learned: The AI could look at the light pattern bouncing off the wall and say:
- "That's a hidden letter 'A'."
- "That object is 8 inches tall."
- "It is located 2 feet to the left."
The Surprise: They found that even with a cheap, low-resolution sensor, the AI could do this surprisingly well! It's like teaching a dog to recognize a specific person just by the sound of their footsteps, even if the dog can't see them.

4. The Catch: It's Not Perfect Yet

The paper also points out where the system struggles, which is very important for the future:

Size Matters: The AI is great at spotting big objects but struggles with tiny ones. It's like trying to hear a whisper from a giant vs. a mouse.
Lighting Confusion: The AI sometimes gets confused by the room's lighting. It's like trying to hear a secret conversation in a noisy room; the background noise makes it hard to separate the signal.
Simulation Gap: When they tried to teach the AI using only computer simulations, it didn't work as well as using real data. The real world is messy, and computers are too perfect.

Why Does This Matter?

This research is a big step toward superpowers for everyday devices.

Imagine your phone could:

See a person hiding behind a corner so you don't walk into them.
Help a self-driving car "see" a child running behind a parked bus.
Let a robot navigate a dark, cluttered room without bumping into things.

DENALI proves that we don't need million-dollar, lab-grade equipment to do this. We can use the cheap sensors already in our pockets. By collecting this massive dataset and teaching AI how to interpret the "echoes" of light, the researchers are paving the way for a future where our devices can see the invisible world around us.

1. Problem Statement

The Gap: Consumer-grade LiDARs (found in smartphones and robots) typically output a single depth value per pixel by identifying the dominant peak in a photon-arrival histogram. However, these sensors internally record full time-resolved histograms containing multi-bounce light returns. These late-arriving photons encode rich Non-Line-of-Sight (NLOS) information about hidden objects, which is currently discarded.
The Challenge: Existing NLOS reconstruction methods rely on expensive, lab-grade scanning LiDARs with high temporal/spatial resolution and controlled environments. Directly applying these methods to low-cost, flash-based consumer LiDARs (e.g., ams TMF8828) fails due to severe hardware limitations: coarse spatial resolution, low photon counts, noise, crosstalk, and flood illumination rather than collimated beams.
The Goal: To determine if data-driven inference can unlock NLOS perception capabilities on low-cost hardware, bypassing the need for complex physical reconstruction algorithms.

2. Methodology: The DENALI Dataset

The authors introduce DENALI, the first large-scale real-world dataset designed to characterize NLOS capabilities on consumer LiDARs.

A. Data Acquisition Setup

Sensor: ams TMF8828 (a $10 consumer-grade flash dToF sensor) operating in 3×3 and 8×8 pixel modes with 128 time bins.
Scene Configuration:
- Three-Bounce Path: The setup uses a "relay wall" to facilitate the canonical NLOS path: Laser $\to$ Wall $\to$ Hidden Object $\to$ Wall $\to$ Detector.
- Objects: 3D-printed objects (letters, numbers, shapes) in two sizes (4-inch and 8-inch) coated with retroreflective tape to maximize return signals.
- Variations: 60 unique objects, 100 spatial positions (outside the direct Line-of-Sight), two lighting conditions (lights on/off), and repeated samples.
Ground Truth: A synchronized tracking system (Intel RealSense cameras + AprilTags) captures precise 6-DoF poses of the LiDAR, wall, and hidden objects.
Digital Twins: Every real-world capture is paired with a physically-based simulation (rendered in Mitsuba 3) to create a "real-sim" pair for analyzing simulation fidelity.

B. Dataset Scale

Total Captures: 72,000 scenes.
Data Volume: 2.6 million full-histogram pixels; 336 million time-of-flight bin measurements.

3. Key Contributions

DENALI Dataset: A large-scale, real-world dataset of time-resolved histograms from low-cost LiDARs capturing hidden objects, paired with digital twins.
Data-Driven NLOS Demonstration: Proof that consumer LiDARs can perform accurate NLOS perception (localization, classification, size estimation) using deep learning, despite hardware limitations.
Simulation Fidelity Analysis: A framework to quantify the "sim-to-real" gap, identifying specific modeling discrepancies (pulse width, noise, jitter) that hinder current simulation-based approaches.
Sensor Design Insights: Analysis of how temporal jitter and resolution affect different NLOS tasks, guiding future hardware design for NLOS applications.

4. Experimental Results & Analysis

The authors evaluated three tasks using four model architectures (MLP, 1D CNN, 3D CNN, Transformer) on the 3×3 resolution data:

Task 1: NLOS Object Localization (Predicting $x, y$ $x, y$ position)
- Best Performance: 3D CNN achieved an RMSE of 0.046m.
- Insight: Convolutional architectures outperformed MLPs and Transformers, suggesting that local spatiotemporal structure is critical.
Task 2: Object Classification (Identifying shape among 30 categories)
- Best Performance: 3D CNN achieved a Macro-F1 of 0.44 (Top-1 accuracy ~35%).
- Insight: Performance is highly dependent on object size and distance from the relay wall.
Task 3: Size Classification (4-inch vs. 8-inch)
- Best Performance: 1D CNN achieved 94.7% accuracy.
- Insight: Size is the easiest feature to distinguish, even with low-resolution sensors.

Key Limitations Identified:

Scene Factors: Performance degrades significantly for smaller objects and those further from the relay wall. Objects too close to the wall suffer from overlapping first-bounce and third-bounce returns.
Modeling Factors: Models struggle to disentangle object properties from ambient lighting conditions, leading to distinct spatial error patterns under different lighting.
Sim-to-Real Gap: Simulated histograms fail to capture real-world noise, pulse width, and jitter. While adding real data improves transfer, high-fidelity simulation is required to minimize the gap.

5. Significance and Impact

Democratizing NLOS Vision: This work shifts the paradigm from "reconstructing hidden scenes with expensive hardware" to "learning to perceive hidden scenes with ubiquitous, low-cost sensors."
Scalability: By leveraging data-driven methods, the approach bypasses the need for perfect physical modeling of complex light transport, making NLOS feasible for mobile devices and robots.
Future Directions: The dataset provides a benchmark for:
- Improving simulation fidelity for low-cost sensors.
- Designing task-specific sensors (e.g., determining necessary timing jitter tolerances).
- Developing robust models that factorize geometry, material, and lighting.

In conclusion, DENALI demonstrates that while low-cost LiDARs cannot yet reconstruct high-fidelity 3D hidden scenes using traditional physics-based methods, they possess sufficient signal for data-driven spatial reasoning, opening the door for scalable, deployable NLOS perception in everyday technology.