💻 computer science

ShapeGrasp: Simultaneous Visuo-Haptic Shape Completion and Grasping for Improved Robot Manipulation

ShapeGrasp is a novel robotic system that iteratively combines visual estimation with tactile feedback during grasping to simultaneously refine 3D object shape reconstruction and improve grasp success rates on unfamiliar objects.

Original authors: Lukas Rustler, Matej Hoffmann

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Rustler, Matej Hoffmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a mysterious object it has never seen before, like a weirdly shaped tool or a crumpled box. If the robot only uses its eyes (a camera), it sees a flat, incomplete picture. It might guess the back of the object is smooth, but in reality, there's a handle sticking out. If the robot tries to grab it based on that guess, it might miss, drop the object, or crush it.

ShapeGrasp is a new method that teaches robots to learn by doing, much like how a human would. Instead of just looking and guessing, the robot "feels" its way to a solution.

Here is how it works, broken down into simple steps:

1. The "Blindfolded" Start

The robot starts with a single snapshot from a camera. It uses a smart computer program to guess what the rest of the object looks like. Think of this like looking at a puzzle piece and trying to imagine the whole picture. The robot creates a 3D model, but it's just a guess.

2. The "Trial and Error" Grab

The robot tries to grab the object based on that guess.

If it succeeds: Great! It lifts the object.
If it fails: The object might slip, or the robot might realize, "Wait, my guess was wrong; the object is heavier or shaped differently than I thought."

3. The "Sensory Update"

This is the magic part. Every time the robot tries to grab the object, it learns two new things:

Touch: If the robot's fingers actually touch the object, its sensors tell it, "Here is the exact surface of the object." It's like the robot running its fingers over a statue to feel the bumps.
Empty Space: Even if the robot misses, its fingers take up space. The robot knows, "My fingers couldn't go here because the object must be in the way." It uses this "empty space" data to refine its mental map.

4. The "Do-Over"

If the first grab failed, the robot doesn't just try the same thing again. It takes all the new touch and space information, updates its 3D model of the object to make it more accurate, and then tries a new grab based on this better map. It keeps doing this loop—grab, feel, update, re-grab—until it succeeds.

Why is this special?

Most robots either try to build a perfect 3D model before they touch anything, or they just grab blindly. ShapeGrasp is the first to update its 3D model in the real world after a physical grab.

Think of it like a sculptor. A traditional robot is like someone who tries to carve a statue based on a blurry photo and hopes for the best. ShapeGrasp is like a sculptor who chisels a little, steps back, feels the stone, realizes they chiseled too much, and then adjusts their plan before chiseling again.

The Results

The researchers tested this on two different robot arms with different types of "hands" (grippers):

A two-fingered hand (like a pincer).
A three-fingered hand (more like a human hand).

They found that ShapeGrasp was much better at picking up objects than previous methods.

The two-fingered hand succeeded 91% of the time.
The three-fingered hand succeeded 84% of the time.

Crucially, every time the robot tried to grab the object, the 3D model of that object became more accurate and detailed. By the end, the robot knew exactly what the object looked like, which helps it handle the object carefully later on (for example, if it needs to hand the object to a human).

The Trade-off

The paper notes that this process takes a bit of time (about 5 to 9 seconds per object) because the robot has to think, simulate, and move carefully. However, the trade-off is worth it because it is much more reliable than guessing.

In short, ShapeGrasp turns a robot from a "guess-and-hope" machine into a "learn-and-adapt" machine, using its own mistakes to build a better understanding of the world.

Technical Summary of ShapeGrasp: Simultaneous Visuo-Haptic Shape Completion and Grasping for Improved Robot Manipulation

Problem Statement
Robust grasping of a priori unknown objects remains challenging when objects are only partially observed via a single RGB-D view, resulting in large occluded regions. While shape completion methods can infer missing geometry, the resulting models often contain uncertainty or hallucinated geometry, leading to grasp planning failures, particularly with compliant objects. Conversely, grasping is an informative interaction that provides tactile feedback and constraints on the space occupied by the gripper (free space). Existing approaches often treat shape completion and grasping as separate stages or rely on dedicated haptic exploration loops that are time-consuming. The authors address the need for a system that simultaneously reconstructs a detailed 3D model and executes grasps, utilizing the feedback from grasp attempts (success or failure) to iteratively refine the object model and re-plan.

Methodology
The authors propose ShapeGrasp, an iterative visuo-haptic pipeline that couples implicit-surface shape completion with physics-based grasp planning. The system operates in a closed loop:

Object Representation: The system represents the object as an implicit surface defined by a Signed Distance Function (SDF), $O = \{x \in \mathbb{R}^3 \mid f(x; \theta, z_r) = 0\}$ . The input is an aggregated point cloud $X_r$ containing the initial visual observation, tactile contacts, and free space constraints.
Shape Completion: The core component is a deep neural network based on Implicit Geometric Regularization for Learning Shapes (IGR). It optimizes a latent vector $z_r$ for the specific object instance to approximate the SDF. The network is trained to minimize the distance to observed points and align with surface normals, while an Eikonal term regularizes the gradient magnitude.
Data Fusion:
- Visual: Initial RGB-D data from a fixed table camera and optional wrist camera data.
- Tactile: When gripper sensors contact the object, their positions are computed via forward kinematics and added as point cloud data.
- Free Space: The volume occupied by the gripper body during the approach is projected as a point cloud. These points are treated as constraints that must be outside the object (positive signed distance), preventing the network from hallucinating geometry in the gripper's path.
Pose Estimation: Between iterations, the object's 6D pose is re-estimated using Iterative Closest Point (ICP) alignment between the current segmented point cloud and the initial reference frame, allowing new data to be fused correctly.
Grasp Generation: A physics-based grasp planner (using PyBullet) generates candidates from the reconstructed triangular mesh. It samples surface points, computes approach vectors, and simulates the grasp (closing, lifting, and rotating) to evaluate stability. A quality score $q$ penalizes slippage, object movement, and lack of contact.
Iterative Loop: The system attempts the highest-quality feasible grasp. If the grasp fails (object slips or falls), the system detects the failure, re-estimates the pose, updates the shape model with the new tactile and free space data, and generates new grasp candidates. The loop continues until a successful grasp is achieved or the maximum iteration count is reached.

Key Contributions

Iterative Grasp-and-Complete Pipeline: The authors present the first approach, to their knowledge, that updates shape representations in the real world following a grasp attempt. The pipeline integrates tactile contacts and gripper-derived free space constraints to refine the implicit object model and triggers regrasping upon failure.
Physics-Based Grasp Proposal Generator: A novel generator that consumes reconstructed triangular meshes and supports multi-finger grippers via URDF models. It utilizes rigid-body simulation to evaluate grasp stability, tightly integrating with the shape completion output.
Real-World Evaluation: Extensive experiments were conducted on two different robots (Kinova Gen3 and KUKA iiwa) with two distinct grippers (Robotiq 2F-85 and Barrett Hand) equipped with tactile sensors. The study includes an ablation analysis of information sources (visual-only, free space, tactile, and combined).
Open Source: The code and data are made publicly available.

Results
The method was evaluated on 9 objects from the YCB dataset across 10 runs per object.

Grasp Success Rate (GSR): ShapeGrasp achieved superior success rates compared to baselines (HyperPCR, 3DSGrasp, VQDIF, VISHAC, etc.). It achieved an 84% success rate with the three-finger Barrett Hand and 91% with the two-finger Robotiq 2F-85 gripper.
Shape Completion Quality: The method outperformed baselines in Chamfer Distance (CD) and Hausdorff Distance (HD) metrics. For the Robotiq gripper, ShapeGrasp achieved a CD of 18.64 mm compared to 23.85 mm for the best baseline (HyperPCR). The inclusion of tactile data provided a statistically significant improvement (approx. 5.5% average improvement across metrics) over visual-only completion.
Ablation Study:
- Tactile Data: Significantly improved reconstruction quality and grasp success.
- Free Space: Results were mixed; it slightly improved performance for the Robotiq gripper but decreased performance for the Barrett Hand, likely because the powerful Barrett Hand could crush compliant objects, placing "free space" constraints inside the object volume.
- Wrist Camera: Provided the best reconstruction metrics but is not always available; tactile data serves as a robust alternative.
Efficiency: While the method is slower than some learning-based baselines (approx. 5s for completion + planning vs. <1s for some baselines), it offers a unified pipeline that does not require pre-trained models for specific object categories and handles failures dynamically.

Significance and Claims
The authors claim that ShapeGrasp demonstrates a "symbiotic pipeline" where shape completion and grasping mutually reinforce each other. The primary significance lies in the ability to improve the 3D shape representation after a real-world grasp, a capability the authors state is unique to their approach. By treating grasping failures as informative events rather than terminal errors, the system adapts its internal model to reduce uncertainty. The authors note that while the method currently prioritizes recall over precision (leading to slightly "inflated" shapes), it successfully achieves high grasp success rates and better reconstruction quality than state-of-the-art methods that rely solely on visual data or dedicated haptic exploration loops. The work is positioned as a step toward more robust autonomous manipulation in scenarios requiring precise knowledge of object properties, such as robot-human handovers or sorting tasks.