💻 computer science

EfficientPENet: Real-Time Depth Completion from Sparse LiDAR via Lightweight Multi-Modal Fusion

EfficientPENet is a lightweight, real-time depth completion network that leverages a ConvNeXt backbone, sparsity-invariant convolutions, and position-aware test-time augmentation to achieve competitive accuracy on the KITTI benchmark with significantly reduced parameters and latency compared to existing methods, making it suitable for resource-constrained edge platforms.

Original authors: Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, Kendall N. Niles

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Johny J. Lopez, Md Meftahul Ferdaus, Mahdi Abdelguerfi, Anton Netchaev, Steven Sloan, Ken Pathak, Kendall N. Niles

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Seeing the Invisible

Imagine you are a robot inspector sent down into a dark, damp sewer pipe to check for cracks or holes. You have two tools:

A Camera: It sees the walls clearly, but it's "blind" to distance. It can't tell if a stain on the wall is just a smudge or a deep, dangerous hole.
A Laser Scanner (LiDAR): It measures distance perfectly, but it's like a flashlight in a foggy room—it only hits a few spots on the wall, leaving 95% of the image as empty black space.

The Problem: To navigate safely, the robot needs a complete 3D map. It needs to fill in all those black holes in the laser data using the camera's picture as a guide. This is called "Depth Completion."

The Catch: The best AI models that do this job are like heavy, gas-guzzling trucks. They are incredibly accurate, but they are too big and slow to fit inside a small, battery-powered robot. They would drain the battery in minutes or move so slowly the robot would crash.

The Solution: The authors built EfficientPENet. Think of this as a sleek, electric sports car. It's much lighter and faster, but it still drives just as well as the heavy truck.

How They Built the "Sports Car" (The 3 Key Upgrades)

The researchers took an existing, heavy design and gave it three major makeovers to make it fast and light without losing accuracy.

1. Swapping the Engine: From a V8 to a Modern Hybrid

The Old Way: Previous models used a "ResNet" engine. It's like an old V8 engine: powerful but heavy, hot, and inefficient.
The New Way: They swapped it for a ConvNeXt backbone.
- The Analogy: Imagine replacing a clunky, heavy brick wall with a modern, lightweight carbon-fiber frame. It uses the same materials (math) but arranges them smarter.
- The Result: It uses 3.7 times fewer parts (parameters) and runs 23 times faster than the old heavy models, yet it still understands the scene perfectly.

2. The "Don't Eat the Empty Space" Rule

The Problem: When the laser scanner misses a spot, it sends a "0" (zero). Standard AI gets confused. It thinks, "Oh, the distance is zero! That means the wall is right here!" But actually, it just means "I didn't see anything."
The Fix: They added a special filter called Sparsity-Invariant Convolution.
- The Analogy: Imagine a chef tasting a soup. If the spoon hits an empty spot in the bowl, a normal chef might think the soup is missing. This new chef has a rule: "Ignore the empty spots; only taste the actual food."
- The Result: The AI stops getting confused by the missing laser dots and focuses only on the real data.

3. The "Edge Sharpening" Polish

The Problem: AI predictions often look like a blurry watercolor painting. The edges of a pipe or a crack look fuzzy.
The Fix: They added a final step called CSPN (Convolutional Spatial Propagation Network).
- The Analogy: Think of the AI's first guess as a rough sketch. The CSPN is like a sharpie marker that goes over the sketch. It looks at the camera image, sees where the edges are, and says, "Hey, this line is sharp! Let's make the depth map sharp here too, and smooth here."
- The Result: The final 3D map has crisp, clean edges, so the robot knows exactly where the wall ends and the hole begins.

The Secret Weapon: The "Mirror Trick"

The researchers also found a clever trick to make the AI even smarter without teaching it anything new.

The Trick: They take the image, flip it horizontally (like looking in a mirror), run the AI on it, and then flip the result back.
The Catch: Usually, flipping an image confuses the AI because the "coordinates" (left vs. right) get messed up.
The Fix: They built a Position-Aware wrapper. It's like a smart mirror that not only flips the picture but also rewrites the address labels on the wall so the AI knows it's looking at the mirror image.
The Result: By averaging the "normal" view and the "mirror" view, the AI makes fewer mistakes. It's like asking two people to look at the same thing from different angles to get a better consensus.

Why Does This Matter? (The Real-World Impact)

1. Speed:
The new system runs at 48 frames per second. That's smoother than a movie! The old heavy systems were so slow they could barely manage 2 or 3 frames per second. This means the robot can move fast without crashing.

2. Size:
It fits on a NVIDIA Jetson, which is a computer the size of a deck of cards. This is crucial for robots that need to crawl into tiny pipes where a laptop-sized computer won't fit.

3. Accuracy:
On standard tests (the KITTI benchmark), it made the fewest catastrophic errors.

The Trade-off: It makes slightly more "average" small errors than the super-heavy models, but it makes zero "disaster" errors (like thinking a wall is 100 feet away when it's 1 foot away).
Why this is good: For a robot in a sewer, you care more about not hitting a wall right in front of you than you care about measuring a wall 50 feet away perfectly.

Summary

EfficientPENet is a breakthrough because it proves you don't need a supercomputer to see in 3D. By using a smarter engine (ConvNeXt), ignoring empty data (Sparsity-Invariant), and sharpening the edges (CSPN), they created a system that is fast, small, and accurate enough to power the next generation of autonomous robots inspecting our aging underground infrastructure.

1. Problem Statement

Context: Robotic systems for infrastructure inspection (e.g., underground sewers) require accurate 3D perception to assess structural conditions. While 2D cameras provide texture, they lack geometric depth. LiDAR sensors provide depth but are often sparse (covering <5% of the image) when using low-cost, lightweight sensors suitable for battery-powered crawlers.
**Challenge:** Existing depth completion methods (recovering dense depth from sparse LiDAR + RGB) achieve high accuracy on benchmarks like KITTI but rely on heavy backbone architectures (e.g., ResNet-101/152, Swin Transformers) with >90M parameters. These models are too computationally expensive for real-time deployment on resource-constrained edge hardware (e.g., NVIDIA Jetson AGX), which has strict limits on power, memory, and thermal dissipation.
Goal: Develop a depth completion architecture that bridges the gap between benchmark accuracy and real-time deployability on edge devices, specifically tailored for environments with unique challenges like zero lighting and repetitive cylindrical geometry.

2. Methodology: EfficientPENet Architecture

The authors propose EfficientPENet, a two-branch encoder-decoder network that modernizes the established PENet framework through three key architectural innovations:

A. ConvNeXt-Based RGB Encoder

Replacement: The standard ResNet-34 encoder is replaced with a ConvNeXt-Base backbone.
Key Modifications:
- Layer Normalization: Replaces Batch Normalization to ensure stability with small micro-batch sizes (batch size 4) common in memory-constrained training.
- 7×7 Depthwise Convolutions: Replaces standard 3×3 stacks to enlarge the receptive field early in the network, crucial for capturing the cylindrical geometry of sewer pipes.
- Stochastic Depth: Applied as a regularization technique to prevent overfitting and improve generalization.
- Input Handling: The network accepts 6 input channels (3 RGB + 1 sparse depth + 2 position encodings). Pretrained ImageNet weights are transferred via a structured weight replication strategy.

B. Sparsity-Invariant Depth Encoding

Problem: Standard convolutions treat missing LiDAR points (zeros) as valid data, corrupting feature maps.
Solution: The depth branch employs sparsity-invariant convolutions. A parallel binary validity mask tracks valid LiDAR returns. The convolution output is normalized by the count of valid pixels within the kernel window, ensuring features are invariant to the density of the input point cloud.

C. Late Fusion and Multi-Scale Deep Supervision

Late Fusion: Instead of concatenating inputs early, RGB and depth branches encode features independently. Features are merged at the network bottleneck using an attention-based gating mechanism.
Deep Supervision: Intermediate depth predictions are generated at multiple scales (1/8, 1/4, 1/2 resolution) and supervised against downsampled ground truth. This provides short gradient paths to deep layers, accelerating convergence and mitigating vanishing gradients.
Confidence Weighting: Final depth is a confidence-weighted combination of the RGB and depth branch predictions.

D. Geometric Refinement (CSPN)

A Convolutional Spatial Propagation Network (CSPN) is applied as a post-processing step. It learns an affinity matrix from RGB features to propagate depth information locally, preserving sharp object boundaries and preventing "bleeding" of depth values across edges.

E. Position-Aware Test-Time Augmentation (TTA)

Standard horizontal flipping for TTA breaks consistency when the network uses explicit pixel position encodings.
Innovation: The authors introduce a wrapper that inverts the horizontal coordinate values ( $x' = 1.0 - x$ ) during the flip. This maintains geometric consistency, yielding a consistent ~12 mm RMSE reduction without retraining.

3. Key Contributions

Architectural Modernization: Successfully adapted the ConvNeXt backbone for multi-modal depth completion, demonstrating that modernized convolutional designs can outperform heavy ResNet/Transformer backbones in efficiency.
Sparsity Handling: Integrated sparsity-invariant convolutions to robustly handle the extreme sparsity of low-cost LiDAR data.
Position-Aware TTA: Introduced a novel coordinate correction strategy for test-time augmentation that specifically addresses the needs of networks using explicit position encodings.
Edge Optimization: Demonstrated a practical solution for real-time inference on embedded platforms, achieving a massive reduction in parameters and latency while maintaining competitive accuracy.

4. Experimental Results

The model was evaluated on the KITTI Depth Completion Benchmark:

Accuracy: Achieved an RMSE of 631.94 mm, the lowest among all compared methods (outperforming BP-Net by ~53 mm and DMD3C by ~46 mm).
Efficiency:
- Parameters: 36.24M (a 3.7× reduction vs. PENet and 2.5× vs. BP-Net).
- Latency: 20.51 ms per frame (48.76 FPS).
- Speedup: Represents a 23× speedup in latency compared to BP-Net.
Trade-off Analysis: While EfficientPENet has a higher Mean Absolute Error (MAE) than heavier models (320.37 mm vs. ~194 mm), it significantly reduces catastrophic outliers (low RMSE). This profile is ideal for the target application (underground inspection) where near-field accuracy is critical, and far-field precision is less relevant.

5. Significance and Application

Real-World Deployment: The paper validates that high-accuracy 3D perception is feasible on edge devices (NVIDIA Jetson) without cloud connectivity, a critical requirement for autonomous robotic inspection in environments like sewers.
Domain Adaptation: The authors demonstrate a transfer learning strategy (KITTI pre-training + synthetic sewer fine-tuning) to adapt the model to underground environments characterized by zero lighting, wet reflective surfaces, and cylindrical geometry.
System Integration: The model was successfully integrated into a LIO-SAM (LiDAR-Inertial Odometry and Mapping) pipeline, generating dense, geometrically coherent 3D point clouds in real-time, significantly improving upon traditional Delaunay triangulation methods.

Conclusion: EfficientPENet establishes a new state-of-the-art for the accuracy-efficiency trade-off in depth completion, proving that lightweight, modernized architectures can replace heavy backbones for real-time robotic perception on resource-constrained hardware.