🤖 AI

SemanticFeels: Semantic Labeling during In-Hand Manipulation

The paper presents SemanticFeels, a framework that integrates semantic material classification from high-resolution tactile data with neural implicit shape representation to enable robots to simultaneously perceive object geometry and continuous material regions during in-hand manipulation.

Original authors: Anas Al Shikh Khalil, Haozhi Qi, Roberto Calandra

Published 2026-02-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anas Al Shikh Khalil, Haozhi Qi, Roberto Calandra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot hand trying to figure out what it's holding. Right now, most robots are like people trying to identify an object while wearing thick, fuzzy winter gloves and blindfolds. They can feel the shape (is it round? is it flat?), but they struggle to tell the difference between a soft piece of fabric, a cold piece of metal, or a rough piece of wood just by touching it.

This paper introduces SemanticFeels, a new "superpower" for robot hands that lets them not only feel the shape of an object but also "feel" what it's made of in real-time.

Here is how it works, broken down with some everyday analogies:

1. The Problem: The Robot is "Tactilely Blind"

Think of a robot hand as a detective. Previously, the detective had a magnifying glass (cameras) to see the object's shape, but when the object was hidden or in the dark, the detective had to rely on touch. However, the robot's "touch" was just a generic "bump." It knew where the bump was, but not what the bump felt like.

2. The Solution: Giving the Robot "Fingertip Eyes"

The researchers equipped a robot hand (called the Allegro Hand) with special sensors on its fingertips called Digit sensors.

The Analogy: Imagine if your fingertips had tiny, high-definition cameras built right into your skin. When you touch a table, your finger doesn't just feel "hard"; it takes a microscopic photo of the wood grain.
The Tech: These sensors take high-resolution photos of whatever the robot is touching.

3. The Brain: The "Material Detective"

Once the sensors take these microscopic photos, the robot needs to figure out what they mean.

The Analogy: Think of this like a seasoned chef tasting a sauce. The chef doesn't just taste "salty"; they identify specific ingredients like "oregano," "basil," or "garlic."
The Tech: The robot uses a pre-trained AI brain (a neural network) to look at the photos from the fingertips. It instantly classifies the texture: "This is Fabric," "This is Plastic," "This is Wood."

4. The Magic Map: Painting the Object

This is the coolest part. The robot doesn't just say "I am touching wood." It builds a 3D map of the object in its mind, but it paints that map with colors representing materials.

The Analogy: Imagine the robot is holding a multi-colored stress ball. As it rotates the ball in its hand, it's like a painter slowly revealing a hidden mural. One side is painted blue (plastic), and as it turns, it reveals a red patch (fabric). The robot is building a "material map" in real-time, knowing exactly where the plastic ends and the fabric begins.
The Tech: They combined the shape-mapping technology (which tells the robot the object's geometry) with the material-detecting AI. The result is a single, unified 3D model that knows both the shape and the material of every part of the object.

5. The Results: How Good Is It?

The team tested this on objects made of one material (like a wooden candle holder) and objects made of mixed materials (like a plastic toy with a fabric patch).

The Score: On the mixed objects, the robot successfully identified the correct material in about 80% of the spots it touched.
The Quirk: The robot's "thumb" wasn't as good at this as its other fingers.
- Why? The robot's movement policy (how it rotates the object) sometimes made the thumb slip or touch the object loosely. It's like trying to take a photo with a shaky hand; the picture comes out blurry, and the AI can't tell if it's wood or metal. The other fingers held the object steady, so they got perfect scores.

Why Does This Matter?

In the future, robots won't just be able to pick up a cup; they will know if the cup is glass (so they don't crush it) or plastic (so they can squeeze it). They will know if a handle is slippery metal or grippy rubber.

SemanticFeels is the first step toward robots that don't just "see" the world, but truly "feel" and understand the texture and nature of the things they interact with, making them safer and smarter helpers in our daily lives.

1. Problem Statement

Robotic in-hand manipulation requires not only tracking an object's pose and reconstructing its geometry but also understanding its semantic properties, such as material composition (e.g., plastic, wood, metal, fabric). While vision-based methods (RGB-D cameras) are effective for global shape and pose estimation, they fail when objects are occluded or lighting is poor. Conversely, while tactile sensing provides direct information about texture and stiffness, existing frameworks (like NeuralFeels) primarily focus on geometric reconstruction and pose tracking, lacking the capability to perform real-time semantic labeling (material classification) and integrate it into a unified 3D map.

The core challenge addressed is: How can a robot simultaneously reconstruct the 3D geometry of an object and assign semantic material labels to specific regions during dynamic, in-hand manipulation using only tactile and visual data?

2. Methodology

The authors propose SemanticFeels, an extension of the NeuralFeels framework that integrates a material classification module with a neural implicit shape representation.

A. Hardware Setup

Robot: An Allegro Hand (4 fingers, 16 joints) equipped with four Digit sensors (high-resolution vision-based tactile sensors) on the fingertips.
Vision: An Intel RealSense D435i RGB-D camera (specifically the front-left view) provides global context and point clouds.
Control: The hand is controlled by HORA (In-Hand Object Rotation via Rapid Motor Adaptation), a reinforcement learning-based controller trained in simulation to rotate objects using proprioceptive feedback.

B. System Architecture

The framework consists of two main pipelines that run in parallel and fuse data:

Material Classification Pipeline:
- Input: High-resolution tactile images captured by the Digit sensors.
- Model: A fine-tuned EfficientNet-B0 Convolutional Neural Network (CNN).
- Task: Classifies the local material under the sensor into one of four classes: Plastic, Metal, Fabric, Wood.
- Training: Trained on a dataset of ~20,749 tactile images (balanced across materials) with data augmentation (rotation, lighting).
Semantic Neural Implicit Representation (The "SemanticFeels" Core):
- Base: Extends the original NeuralFeels Signed Distance Field (SDF) network.
- Architecture: A dual-branch Multi-Layer Perceptron (MLP).
  - Geometry Branch: Maps 3D coordinates ( $x$ ) to a signed distance value (SDF).
  - Material Branch: Maps the same 3D coordinates (concatenated with features from the geometry branch) to a 4-dimensional vector representing material class logits.
- Encoding: Uses HashGrid encoding for efficient spatial representation and Spherical Harmonics for the material branch.
- Output: A unified 3D map that simultaneously predicts the object's shape and assigns a continuous material label to every point in space.

C. Workflow

The robot manipulates the object while the HORA controller rotates it.
Tactile sensors capture images; the CNN predicts the local material.
The 3D position of the contact point and the predicted material label are fed into the neural SDF network.
The network updates the global 3D map, refining both the geometry and the semantic material distribution over time.

3. Key Contributions

Semantic Extension of NeuralFeels: First work to integrate real-time semantic material labeling into a neural implicit representation framework for in-hand manipulation.
Visuo-Tactile Fusion: Demonstrates a pipeline that fuses sparse, localized tactile data (for texture/material) with global visual data (for pose/shape) to create a semantically rich 3D map.
Dual-Branch Neural Network: Proposes a modified SDF network architecture that jointly learns geometry and material properties, allowing the system to "learn" where different materials exist on an object's surface.
Real-Time Implementation: Successfully deploys the system on a physical robotic hand, processing tactile images and updating the 3D map in real-time.

4. Experimental Results

A. Material Classification Performance

Offline Dataset: The EfficientNet-B0 model achieved 99.04% validation accuracy and 98.88% test accuracy on the hand-collected dataset.
Sensor Variance: Performance varied by finger. The Ring and Index fingers achieved the highest accuracy (up to 99.60%) due to stable contact. The Thumb performed lower (97.42%) due to inconsistent contact during the rotation policy.
Material Difficulty:
- Plastic: Near-perfect accuracy (~100%).
- Fabric: High accuracy (95% on Ring).
- Metal & Wood: Lower accuracy (ranging 23%–75%), particularly on the thumb and middle fingers, due to texture similarity or contact instability.

B. Multi-Material Object Mapping

Experiment: The system manipulated a custom object composed of both plastic and fabric regions.
Metric: Matching percentage between the predicted material map and the ground truth map.
Results: Across 4 manipulation runs, the system showed progressive improvement as more tactile data was gathered.
- Average Matching Accuracy: 79.87% (Standard Deviation: 4.41%).
- Progression: Accuracy improved from ~43% at the start of a run to >85% by the end (e.g., Run 4 reached 85.12%).
Visualization: The system successfully reconstructed the 3D shape and correctly localized the fabric and plastic regions, though some boundary errors occurred due to contact inconsistencies.

5. Significance and Limitations

Significance

Adaptive Manipulation: By knowing the material of an object in real-time, robots can adapt their grip force and manipulation strategies (e.g., avoiding slipping on fabric or crushing fragile wood).
Occlusion Robustness: The system can infer material properties even when parts of the object are occluded from the camera, relying on tactile feedback.
Foundation for Future Tasks: This approach lays the groundwork for complex tasks like object sorting, tool use, and assembly where material properties are critical.

Limitations & Future Work

Contact Stability: The current rotation policy does not guarantee firm contact for all fingers (especially the thumb), leading to noisy tactile data and lower classification accuracy for certain materials.
Material Diversity: The study was limited to four materials. Future work needs to expand to a wider variety of textures and stiffness levels.
Dynamic Sensitivity: Tactile data is highly sensitive to contact angle and pressure, which vary unpredictably during dynamic motion.
Future Directions:
- Developing adaptive manipulation policies that adjust contact points based on sensor feedback.
- Integrating online learning to adapt to new materials during deployment.
- Fusing vision and force sensing to complement tactile data and improve robustness.

In conclusion, SemanticFeels successfully bridges the gap between geometric reconstruction and semantic understanding in robotic manipulation, proving that tactile sensing can be effectively used to build rich, material-aware 3D maps of objects in real-time.