💻 computer science

ESAM++: Efficient Online 3D Perception on the Edge

The paper proposes ESAM++, a lightweight and scalable framework for online 3D scene perception on edge devices that replaces ESAM's computationally expensive 3D sparse UNet with an efficient 3D Sparse Feature Pyramid Network (SFPN), achieving competitive accuracy with up to three times faster inference and half the model size.

Original authors: Qin Liu, Lavisha Aggarwal, Saptarashmi Bandyopadhyay, Vikas Bahirwani, Marc Niethammer, Ehsan Adeli, Andrea Colaco

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qin Liu, Lavisha Aggarwal, Saptarashmi Bandyopadhyay, Vikas Bahirwani, Marc Niethammer, Ehsan Adeli, Andrea Colaco

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a pair of smart glasses that need to understand the 3D world around you in real-time. You walk into a room, and the glasses need to instantly figure out: "That's a chair, that's a table, and that's a person." They need to do this without lag, without needing a massive supercomputer in your pocket, and without sending your private video feed to the cloud.

This is the problem ESAM++ solves.

The Problem: The "Heavy Backpack"

The paper starts by looking at a previous, very smart system called ESAM. Think of ESAM as a brilliant detective who can identify objects perfectly. However, to do its job, this detective carries a giant, heavy backpack filled with complex tools (specifically, a "3D sparse UNet").

While the detective is great at solving cases, the backpack is so heavy that:

It takes a long time to put on and take off (slow processing).
It's too heavy for a regular person (like a phone or a robot on the edge) to carry around.
It uses up all the detective's energy, leaving little room for anything else.

The researchers found that this "backpack" was the main reason the system was too slow and too big for everyday devices like phones or robots that don't have powerful graphics cards (GPUs).

The Solution: The "Lightweight Backpack" (ESAM++)

The authors created ESAM++, which is like giving that same brilliant detective a brand new, ultra-lightweight backpack.

Instead of carrying one giant, clunky tool, the new backpack uses a 3D Sparse Feature Pyramid Network (SFPN). Here is how that works in plain English:

The Old Way (The Pyramid): Imagine trying to build a house by only looking at the very top floor. You miss all the details of the foundation and the middle rooms. The old system only looked at the "top layer" of the 3D data, which required a huge amount of energy to process.
The New Way (The Feature Pyramid): The new system looks at the house from all angles at once. It checks the foundation, the middle floors, and the roof simultaneously. It gathers small, efficient clues from every level and combines them.

By doing this, the system doesn't need to carry the heavy "top-floor-only" tools anymore. It can grab the right information quickly from any level of detail.

The Results: Faster, Smaller, Just as Smart

The paper claims that by swapping the heavy backpack for this new, smart one, they achieved three major wins:

3x Faster: The new system processes the world three times faster than the old one. If the old system took 3 seconds to recognize a room, the new one does it in 1 second.
2x Smaller: The "backpack" (the computer model) is half the size. This means it fits easily on devices with limited memory, like a smartphone or a small robot.
Just as Accurate: Even though it's lighter and faster, it doesn't make more mistakes. It identifies objects just as well as the heavy, slow version.

Real-World Testing

The researchers tested this on four different "training grounds" (datasets) filled with messy, real-world 3D scans of rooms.

The "Noisy" Test: They even tested what happens if the camera shakes or the data is a bit blurry (like walking quickly). The new system stayed surprisingly steady, only getting confused when the noise was extreme.
The "Edge" Test: They ran it on an iPhone 15 (a standard phone). The phone could handle the task smoothly, using very little battery power, proving that you don't need a supercomputer to do this anymore.

The Bottom Line

ESAM++ is like upgrading a clumsy, slow-moving robot with a heavy backpack into a nimble, fast-moving one with a smart, lightweight vest. It allows robots, AR glasses, and phones to understand 3D spaces instantly and privately, right where they are, without needing to connect to a powerful server.

The paper concludes that this makes real-time 3D vision practical for the devices we actually use every day, rather than just for research labs with expensive equipment.

Technical Summary: ESAM++: Efficient Online 3D Perception on the Edge

Problem Statement

Real-time online 3D scene perception is critical for robotics, AR/VR, and autonomous systems, particularly in edge computing environments where computational resources are limited and data privacy is paramount. While recent state-of-the-art methods like EmbodiedSAM (ESAM) have demonstrated the potential of online 3D perception by leveraging the Segment Anything Model (SAM) for fine-grained instance segmentation, they face significant deployment barriers.

The primary bottleneck identified in ESAM is its reliance on a 3D sparse UNet for point cloud feature extraction. This component accounts for the majority of the 3D inference time and model size, rendering it impractical for resource-constrained edge devices lacking GPU acceleration. The 3D sparse UNet suffers from specific inefficiencies:

High Computational Cost: It produces final outputs exclusively at the last layer operating on high-resolution data, incurring substantial costs when input resolution or output channels are large.
Model Size: As the encoder downsamples, channel counts increase, leading to a large number of parameters in lower-resolution layers.
Underutilized Features: The decoder progressively upsamples hierarchical features, yet predictions are made only at the highest resolution, leaving intermediate multi-scale features underutilized.

Methodology: ESAM++

The authors propose ESAM++, a lightweight and scalable framework designed to replace the computationally expensive 3D sparse UNet with a 3D Sparse Feature Pyramid Network (SFPN). The system maintains the core online processing pipeline of ESAM but optimizes the backbone for edge deployment.

Core Architecture: 3D Sparse Feature Pyramid Network (SFPN)

The SFPN is a lightweight encoder-decoder architecture specifically designed for efficient multi-scale feature extraction from sparse 3D point clouds.

Encoder: Gradually downsamples the input through sparse 3D convolutions and residual blocks, capturing hierarchical features across four spatial resolutions. Feature channels increase from $C_1$ to $C_4$ .
Decoder: Mirrors the encoder using sparse transposed convolutions (ConvTr) and residual blocks to progressively upsample features, with channels decreasing from $C_4$ to $C_8$ .
Multi-Scale Aggregation: Unlike the standard UNet which predicts only from the topmost layer, SFPN harnesses the decoder's multi-scale outputs. It upsamples features from all decoder stages to the original resolution, concatenates them into a unified high-resolution feature map, and processes them through a Multi-Layer Perceptron (MLP) to generate final point-wise features.
Optimization: The architecture limits the number of output channels in high-resolution layers to reduce computational cost and removes intermediate layers with excessive channel counts to reduce model size.

Integration with Instance Segmentation

ESAM++ retains ESAM's mechanism for 3D instance segmentation:

Query Lifting: 2D masks generated by a Visual Foundation Model (VFM, specifically FastSAM in the experiments) are lifted into 3D superpoints by pooling features from the SFPN output.
Refinement: These superpoints initialize 3D instance queries, which are refined via transformer-based decoder layers to predict instance masks.
Merging: Current frame masks are merged with previously accumulated masks using pairwise similarity calculations (matrix multiplication) to maintain temporal consistency.

Key Contributions

Bottleneck Analysis: The paper provides a detailed computational analysis identifying the 3D sparse UNet as the primary efficiency bottleneck in online 3D perception, specifically highlighting the trade-offs between resolution, channel count, and latency.
Novel Architecture (SFPN): The introduction of the 3D Sparse Feature Pyramid Network, which enables efficient multi-scale representation and prediction, specifically tailored for edge devices without GPU acceleration.
Performance Gains: Demonstration that replacing the UNet with SFPN yields a model that is 2× smaller in size and achieves up to 3× faster inference on CPUs while maintaining competitive accuracy.
Robustness and Generalization: Evidence that the model performs well in low-data regimes and remains robust to camera pose noise (up to 5%), suggesting potential for integration with RGB-only online reconstruction models.

Experimental Results

The authors evaluated ESAM++ on four challenging benchmarks: ScanNet, ScanNet200, SceneNN, and 3RScan.

Class-Agnostic Segmentation (ScanNet200):
- Ours-Large achieved an AP of 43.7, surpassing the previous online state-of-the-art (ESAM-E at 43.4) and offline methods.
- Inference Speed: On an Intel Xeon Silver 4314 CPU, the Ours-Large model processed frames in 326ms, compared to ESAM's 934ms (a ~3× speedup).
- Model Size: The Ours-Large model has 41.2M parameters, compared to ESAM's 44.6M. The Ours-Small variant is only 14.1M parameters with 211ms latency.
Class-Aware Segmentation (ScanNet & SceneNN):
- ESAM++ achieved state-of-the-art performance among online methods, with Ours-Large reaching 43.7 AP on ScanNet and 34.1 AP on SceneNN.
Cross-Dataset Generalization:
- The model demonstrated strong transferability to SceneNN and 3RScan, outperforming zero-shot methods and offline approaches that struggle with the noisy, fast-motion data of 3RScan.
Edge Device Evaluation:
- Tested on an iPhone 15 (A16 Bionic), the small model achieved 190ms latency per frame with 4.5W power consumption, validating its feasibility for mobile deployment.
Ablation Studies:
- Removing the upsampled fusion, the pyramid structure, or skip connections resulted in significant drops in accuracy (e.g., removing the pyramid dropped AP from 43.7 to 33.8), confirming the necessity of the multi-scale design.
Robustness to Noise:
- The method remained robust with up to 5% camera pose noise (AP dropped only slightly from 43.7 to 42.4), though performance degraded significantly at 20% noise.

Significance and Claims

The paper claims that ESAM++ represents a practical solution for deploying online 3D scene perception on edge devices without GPU acceleration. By addressing the computational inefficiencies of the 3D sparse UNet, the authors enable real-time, fine-grained 3D instance segmentation that balances speed, model size, and accuracy.

The authors emphasize that while the current work focuses on optimizing the point cloud backbone, the framework is compatible with existing Visual Foundation Models. They modestly suggest that the demonstrated robustness to camera noise highlights a promising direction for future work: integrating ESAM++ with RGB-only online reconstruction models (such as CUT3R) to enable fully online 3D perception using only RGB inputs, further expanding its applicability to resource-constrained scenarios.