🤖 machine learning

Filling the Pareto-Optimal Front for Affordance Segmentation on Embedded Devices Using RGB-D Cameras

This paper proposes two hardware-aware approaches to integrate RGB-D data into compact deep networks for affordance segmentation on embedded devices, successfully achieving real-time performance within smartphone-compatible energy budgets while identifying Pareto-optimal solutions that balance generalization accuracy with hardware constraints.

Original authors: Edoardo Ragusa, Giovanni Paolo Canuti, Simone Lugani, Rodolfo Zunino, Paolo Gastaldo

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Edoardo Ragusa, Giovanni Paolo Canuti, Simone Lugani, Rodolfo Zunino, Paolo Gastaldo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot hand how to pick up a coffee mug. You can't just tell it "grab the mug"; you have to tell it where to grab it. If it grabs the handle, it might tip over; if it grabs the bottom, it might slip. This is the job of "affordance segmentation": a fancy term for teaching a computer to look at an object and highlight exactly which parts are safe to touch and which parts are not.

For a long time, robots have relied on standard cameras (RGB) to do this, seeing the world in color. But color can be tricky. If a blue mug sits on a blue table, a color camera might get confused and think the mug is part of the table. This is where depth sensors come in. Think of a depth sensor as a pair of eyes that can also measure distance, creating a 3D map of the world. It doesn't care if the mug is blue or the table is blue; it knows the mug is closer to the camera. The big challenge, however, is that robots often need to be small, portable, and battery-powered. Running complex 3D vision software on a tiny chip is like trying to run a supercomputer inside a smartwatch—it usually overheats or drains the battery instantly. Scientists have been trying to figure out how to make these "3D-aware" brains small enough to fit on a wearable robot without eating up all its energy.

This paper tackles that exact puzzle. The researchers wanted to see if they could build a tiny, efficient robot brain that uses both color (RGB) and depth (D) data to figure out how to grab things, all while running on a portable device. They didn't just guess; they built two different "architects" to design these brains. The first approach was a "Hardware-Aware Neural Architecture Search" (HW-NAS). Imagine this as a super-fast, automated robot architect that tries thousands of different brain designs, testing each one to see which fits best on a specific tiny chip while still being smart enough to grab a mug. The second approach was a "fine-tuning" method, which is like taking a brain that was already very good at seeing in 2D (color only) and gently teaching it how to understand 3D depth, rather than building a new brain from scratch.

The team tested these ideas on real-world datasets containing thousands of images of everyday objects like spoons, hammers, and mugs. They found that adding depth information almost always made the robot smarter. For example, when trying to identify a "scoop," the model using depth data was 31.6% more accurate than the one using only color. This was especially helpful when lighting was bad or when objects had colors that blended into the background. The researchers also proved that their new designs sit on the "Pareto optimal front." In plain English, this means they found the perfect sweet spot: these models are as smart as they can possibly be for the amount of energy and memory they use. You can't get a smarter model without making it much bigger or slower, and you can't make it smaller without making it dumber.

Finally, they put their best models to the test on a real prototype: a Jetson Nano board (a small computer often used in robotics) connected to a RealSense RGB-D camera. They measured how fast it could think and how much battery it used. The results were promising. Even with the extra work of processing 3D depth data, the system could process images in real-time (up to 16 frames per second with optimization) and run on a power budget compatible with standard smartphone batteries. The paper suggests that by using these smart, depth-aware designs, we can make wearable robots that are not only more accurate at grabbing objects but also light and energy-efficient enough to be worn by people in their daily lives.

Technical Summary: Filling the Pareto Optimal Front for Affordance Segmentation on Embedded Devices

Problem Statement
Wearable robots require semi-autonomous control capabilities, particularly for grasping and manipulation tasks, to relieve users of strenuous or hazardous chores. A critical component of this control pipeline is Affordance Segmentation (AS), which identifies functional parts of an object to enable fine-grained control. While Deep Neural Networks (DNNs) are the standard for AS, deploying them on portable, energy-constrained embedded devices presents a significant challenge. Existing solutions often rely on either "tiny" models optimized for microcontrollers (which may lack robustness) or high-performance unconstrained models (which are too heavy for portable hardware). Furthermore, while depth (D) sensors provide essential geometric information for contactless grasping, their integration into lightweight DNNs for AS remains underexplored. Naive integration of RGB and Depth data often yields marginal improvements or increases computational costs prohibitively. The core problem is designing a system that effectively leverages RGB-D data to improve generalization while strictly adhering to the energy, size, and real-time constraints of embedded hardware.

Methodology
The paper proposes two distinct design strategies to generate hardware-efficient DNNs for RGB-D affordance segmentation, aiming to identify solutions on the Pareto optimal front (balancing accuracy against hardware requirements like FLOPs and parameter count).

Hardware-Aware Neural Architecture Search (HW-NAS):
- The authors reformulated HW-NAS with a novel search space specifically designed for RGB-D inputs.
- The architecture employs a dual-branch structure where separate encoders process RGB and Depth streams independently before merging into a shared backbone.
- An Aggregator network combines the backbone output with intermediate representations from multiple encoder layers, enabling multi-resolution feature extraction.
- The search space allows for flexible connections, including skip connections from early layers to the aggregator, and optimizes parameters such as kernel sizes, expansion factors, and strides.
- The optimization uses an evolutionary algorithm to select architectures that minimize validation loss while satisfying constraints on inference time and memory.
Fine-Tuning Approach:
- As a computationally cheaper alternative, the paper proposes adapting pre-trained RGB-only networks (specifically MobileNetV3) to handle RGB-D inputs.
- A dedicated preprocessing layer is inserted at the input to merge the Depth channel with the RGB channels, creating a 4-channel tensor ( $W \times H \times 4$ ).
- A single convolutional layer (3 filters, stride 1) reduces this back to a 3-channel tensor, allowing the rest of the pre-trained architecture to process the data "as is."
- This method leverages the structural similarities between depth maps and RGB images, allowing the network to adjust low-level features without significantly altering the pre-existing weight configuration.

Experimental Setup
The methods were evaluated on two real-world datasets:

UMD: 28,843 RGB-D images across 7 object categories with varied framing.
IIT: 8,835 images with variations in lighting, occlusion, and resolution.
The task was defined as a three-class pixel-wise classification: "grasp," "do not grasp," and "background." The models were compared against state-of-the-art baselines, including tiny models (HW-0.5s, HW-1s), custom MobileNetV3 variants, transformer-based models (SegFormer), and large CNNs (EfficientNet, VGG16). Deployment was tested on a Jetson Nano board interfaced with a RealSense RGB-D camera.

Key Results

Performance Improvement: Both proposed approaches demonstrated that integrating depth information improves generalization performance, particularly for the "grasp" and "do not grasp" classes. On the UMD dataset, the HW-NAS approach with RGB-D showed accuracy gains of up to 2.8% over RGB-only counterparts.
Pareto Optimality: The generated models consistently located on the Pareto optimal front when plotting generalization performance against model size and FLOPs. They outperformed existing hardware-efficient approaches (tiny models) while maintaining significantly lower resource requirements than large unconstrained models.
Hardware Efficiency:
- Inference Time: Optimized models (using TensorRT float16 quantization) achieved real-time performance on the Jetson Nano, with frame rates ranging from 8–16 FPS depending on the model and power mode (5W vs. 10W).
- Energy Consumption: The system operated within an energy budget compatible with standard smartphone batteries (approx. 7–11W total system power), allowing for roughly 3.5 hours of operation on a 5000 mAh battery.
- Memory: The NAS-generated models were more memory-efficient than the MobileNetV3-based models, utilizing approximately 300 MB of RAM compared to ~400 MB for the latter.
Robustness: Depth information proved crucial in scenarios with suboptimal lighting or when object colors matched the background (e.g., a blue scoop against a blue foreground), significantly enhancing segmentation accuracy.

Significance and Claims
The paper claims that its primary contribution is a design strategy that successfully fills the gap between "tiny" models and high-performance accelerators, specifically for wearable robotics. By introducing a dedicated HW-NAS search space for RGB-D data and a lightweight fine-tuning mechanism, the authors demonstrate that it is possible to:

Extract high-level semantic information directly from RGB-D cameras on smart devices with limited power.
Generate architectures that are Pareto-optimal, offering the best trade-off between accuracy and hardware constraints.
Achieve real-time inference on portable hardware (Jetson Nano) with energy profiles suitable for battery-powered wearable robots.

The authors emphasize that while depth processing adds computational complexity, their methods scale effectively to manage these costs, proving that mid-range, application-specific DNNs are a viable and effective solution for semi-autonomous wearable control pipelines. Future work is identified as incorporating object localization to automate framing and extending these techniques to transformer-based models.

Technical Summary: Filling the Pareto Optimal Front for Affordance Segmentation on Embedded Devices

More like this