💻 computer science

Bootstrap Perception Under Hardware Depth Failure for Indoor Robot Navigation

This paper presents a bootstrap perception system that leverages surviving Time-of-Flight pixels to self-calibrate monocular depth and selectively fuse it with 2D LiDAR, enabling robust indoor robot navigation with significantly improved obstacle coverage and zero collisions even when hardware depth sensors suffer up to 78% pixel loss.

Original authors: Nishant Pushparaju, Vivek Mattam, Aliasghar Arab

Published 2026-04-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nishant Pushparaju, Vivek Mattam, Aliasghar Arab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a robot through a busy office building. To navigate safely, the robot needs to "see" obstacles like chairs, people, and glass walls. It has two main eyes:

The Laser Scanner (LiDAR): This is like a very reliable flashlight that sweeps a single horizontal line across the room. It's great at seeing walls and legs, but it can't see anything above or below that line (like a person's head or a hanging sign).
The Depth Camera (ToF): This is like a pair of 3D glasses that gives the robot a full 3D view. However, it has a major flaw: it hates shiny things. If the robot looks at a polished floor, a glass door, or a mirror, the camera gets confused and goes blind, returning "zero" or "infinity" for those spots.

In many modern offices and warehouses, these shiny surfaces are everywhere. The paper describes a situation where the 3D camera goes blind up to 78% of the time in a single hallway. The robot is essentially driving with one eye closed and the other one hallucinating.

The Problem: The "Blind Spot" Crisis

If the robot relies only on the laser scanner, it might walk right into a person's torso because the scanner only sees their legs. If it relies only on the broken 3D camera, it might crash into a glass wall because the camera thinks the wall isn't there.

The Solution: The "Bootstrap" System

The authors created a clever system they call "Bootstrap Perception." Think of it like a student taking a test who forgets the answers but has a few clues left on the page.

Here is how it works, step-by-step:

1. The "Survival Instinct" (Self-Calibration)
Even when the 3D camera is 78% broken, it still works for the remaining 22% of the image (like a non-shiny wall or a dark chair). The system uses these few "good" pixels as a ruler. It says, "Okay, the camera is broken here, but it's working there. Let's use the working part to figure out the scale for the broken part."

2. The "AI Filler" (Monocular Depth)
The robot uses a smart AI (trained on millions of photos) that can guess the depth of a scene just by looking at a normal 2D picture. Usually, this AI is a bit vague about exact distances. But because the robot has that "ruler" from the few working pixels of the broken camera, it can calibrate the AI's guess. It turns a vague "that looks far away" into a precise "that is exactly 2 meters away."

3. The "Smart Switch" (Selective Fusion)
This is the most important part. The robot doesn't blindly trust the AI. It uses a hierarchy:

If the 3D camera sees something clearly: Use the camera's data (it's the most accurate).
If the 3D camera sees "nothing" (because of a reflection): Switch to the AI's calibrated guess to fill in the gap.
The Laser Scanner: Always stays in the background as the "safety anchor" to make sure the robot doesn't get lost.

The Results: Filling the Gaps

The team tested this in two scenarios:

The Shiny Hallway: In a corridor with glass and polished floors, the 3D camera failed constantly. By using this "Bootstrap" system, the robot's map of obstacles grew by 55%. It suddenly "saw" chairs and people behind glass walls that the camera alone missed.
The Moving Crowd: In a lobby with walking people, the laser scanner only saw their legs. The AI filled in the rest of their bodies (heads and torsos), increasing the obstacle map by 110%. This prevented the robot from thinking a person was just a pair of legs and walking right into them.

The "Tiny Brain" (Efficiency)

Usually, these smart AI models are huge and require powerful, expensive computers. The authors also trained a "distilled student"—a tiny, lightweight version of the AI.

The Big Model: Runs at 22 frames per second (slow for a robot).
The Tiny Student: Runs at 218 frames per second (super fast!) and uses very little memory.
The Trade-off: The tiny student is great for specific, known hallways but isn't as safe for completely new, unpredictable environments. However, in a controlled simulation, it navigated perfectly without hitting anything.

The Big Picture

This paper solves a common robot problem: What do you do when your best sensor breaks?

Instead of giving up or relying on a backup that isn't good enough, the robot uses its own "surviving" data to teach its AI how to fill in the blanks. It's like a painter who loses half their paintbrushes but uses the remaining ones to mix the perfect colors to finish the masterpiece. The robot becomes more robust, seeing obstacles it was previously blind to, all while running on a small, affordable computer.

1. Problem Statement

Indoor robots typically rely on a combination of 2D LiDAR and Time-of-Flight (ToF) cameras for navigation. However, this setup faces a critical failure mode in environments with reflective surfaces (polished floors, glass doors, metal walls), which are common in warehouses and logistics centers.

ToF Limitations: ToF cameras return invalid measurements (zero or out-of-range) on reflective surfaces. In the authors' dataset, this resulted in up to 78% of depth pixels being invalid in specific high-reflectivity segments.
LiDAR Limitations: 2D LiDARs provide stable planar geometry but cannot detect obstacles above their scan plane (e.g., the torso of a pedestrian or objects on shelves).
The Gap: When the ToF sensor fails, the robot loses vertical depth information. Replacing it with a 3D LiDAR is often cost-prohibitive for compact platforms. Existing solutions either assume sensors are healthy or replace the primary sensor entirely, lacking a mechanism to selectively fill gaps using learned models while retaining reliable hardware data.

2. Methodology: Bootstrap Perception System

The authors propose a self-referential, failure-aware sensing hierarchy that uses the surviving valid pixels of a failing ToF sensor to calibrate a learned monocular depth model in real-time.

A. Core Architecture

The system operates as a conservative hierarchy:

LiDAR: Remains the primary geometric anchor for stable planar structure.
Hardware Depth (ToF): Used only where valid (confidence > threshold).
Learned Monocular Depth: Activates only in regions where the ToF sensor returns invalid data.
Semantics (Optional): Refines obstacle inflation radii based on object class (e.g., wider margins for pedestrians).

B. The "Bootstrap" Mechanism

The system's novelty lies in its self-calibration process:

Calibration: Even when 78% of pixels are invalid, the remaining 22% (valid pixels) provide a metric scale. The system computes a per-frame scale factor ( $s$ ) by aligning the median of the learned relative depth predictions with the valid ToF measurements.
Fusion Rule:
$d_{fused} = \begin{cases} d_{tof} & \text{if } c_{tof} \geq \tau \text{ and within range} \\ s \cdot \hat{d}_{monocular} & \text{otherwise} \end{cases}$
This allows the system to "fill its own gaps" without external ground truth or offline calibration data.

C. Embedded Implementation & Distillation

To meet the constraints of embedded hardware (Jetson Orin Nano, 8GB RAM):

Teacher-Student Distillation: A large foundation model (DA3-Large) acts as a "teacher" for offline training. A compact student model (EfficientViT-B1) is trained to mimic the teacher.
Training Objective: The student learns to predict depth and segmentation simultaneously. The training target is a hybrid: it uses ground truth where available, but crucially, in the "corridor" domain, it uses the ToF sensor's valid pixels as ground truth and the teacher's output for invalid pixels, mirroring the runtime fusion logic.
Performance: The distilled student runs at 218 FPS using only 2.7 GB of RAM, compared to the foundation model's 21.9 FPS and 5.1 GB usage.

3. Key Contributions

Bootstrap Perception Architecture: A novel hierarchy where a degraded sensor's surviving pixels calibrate learned depth at runtime, enabling the system to fill its own dead pixels without external data.
Selective Fusion Strategy: Unlike previous works that replace LiDAR or assume healthy sensors, this system keeps LiDAR as the anchor and inserts learned depth only where hardware fails, ensuring conservative behavior when sensors work and recovery when they fail.
Distilled Student for Edge Deployment: Demonstrates that a compact, specialized student model can replace heavy foundation models for specific environments (corridors), achieving high inference speeds (218 FPS) on embedded hardware.

4. Experimental Results

A. Costmap Coverage Improvement

Corridor Scenario (High Reflectivity):
- LiDAR alone: Baseline.
- LiDAR + Learned Depth: Increased occupied obstacle cells by 55% (from 2,295 to 3,546 cells).
- The system successfully detected obstacles behind glass and on polished floors where the ToF failed.
Dynamic Scenario (LILocBench - Pedestrians):
- LiDAR alone: Misses upper body of pedestrians.
- LiDAR + Learned Depth: Increased obstacle coverage by 110% (from 957 to 2,000 cells), capturing the full vertical extent of moving people.
Stability: Adding learned depth reduced costmap "flicker" (normalized occupancy jitter) from 0.61% to 0.43%, as the dense prediction smooths out the intermittent noise of failing ToF pixels.

B. Student Model Performance

General vs. Specialized: A general-purpose student (V4) struggled in the specific corridor domain. A corridor-specialized student (V9), fine-tuned on domain data, achieved a 0.382 m RMSE on the LILocBench benchmark, outperforming the general DA3-Small baseline (0.596 m).
Trade-off: Specialization led to "catastrophic forgetting" of general indoor performance (NYU RMSE degraded), confirming the student is best suited for fixed-route deployments.
Near-Range Safety: The student showed lower accuracy in the critical 0.3–1.0m range compared to the foundation model (8.83% of pixels within 25% error vs. 70.7% for the foundation). This suggests the student requires a safety supervisor for unsupervised general deployment.

C. Closed-Loop Simulation

In a Gazebo simulation of a narrow corridor, the corridor-specialized student achieved a 9/10 success rate with zero collisions, matching the performance of a ground-truth depth baseline.
The single failure in both cases was due to a planner timeout, not depth perception errors, proving the student's depth is sufficient for navigation in controlled, known environments.

5. Significance and Conclusion

This paper addresses a critical reliability gap in indoor robotics: the failure of active depth sensors on reflective surfaces.

Practical Impact: It enables compact, low-cost robots to navigate complex industrial environments (glass doors, polished floors) without expensive 3D LiDARs.
Self-Sufficiency: The "bootstrap" approach eliminates the need for external calibration data or perfect ground truth, making the system deployable in real-world, unstructured settings.
Efficiency: By proving that a distilled student can replace a heavy foundation model for specific tasks, the work paves the way for high-performance, low-latency perception on resource-constrained edge devices.

The authors conclude that while the distilled student is highly effective for fixed-route navigation, the general foundation model (DA3-Small) remains the safer choice for unsupervised, general-purpose deployment due to superior near-range accuracy. Future work will focus on closed-loop hardware deployment in industrial settings.