💻 computer science

TacRefineNet: Goal-Conditioned Tactile Grasp Refinement for Edge-Prominent Objects

TacRefineNet is a tactile-only, goal-conditioned framework that uses a Siamese policy network to iteratively predict corrective wrist pose increments for refining grasps on edge-prominent objects, achieving high success rates in real-world zero-shot deployment after training entirely in simulation.

Original authors: Shuaijun Wang, Haoran Zhou, Diyun Xiang, Yangwei You

Published 2026-07-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shuaijun Wang, Haoran Zhou, Diyun Xiang, Yangwei You

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a delicate, thin piece of glass or a long, slippery rod. To a human, this feels easy, but to a robot, it's a nightmare. Why? Because robots usually rely on their eyes (cameras) to see what they are doing. But when a robot's fingers wrap around a thin edge, the camera often can't see the contact point anymore—it's hidden behind the fingers or the object is just too thin to show up clearly on a screen. It's like trying to thread a needle while wearing thick winter gloves and looking through a foggy window. This is where "tactile sensing" comes in. Instead of just looking, the robot learns to "feel." Just as you can tell if a coin is flat or round by running your thumb over it, even with your eyes closed, robots can use special sensors on their fingertips to feel the shape and position of an object. The big question scientists are asking is: Can a robot use this feeling alone to fix a bad grip and slide an object into the perfect position, without needing to see it or having a pre-made map of what the object looks like?

This paper introduces a clever new system called TacRefineNet that answers "yes" to that question, specifically for tricky, edge-heavy objects like thin plates, flat discs, and slender rods. Think of the robot's hand as a clumsy student trying to hold a slippery pencil. The robot grabs the pencil, but it's slightly crooked. Instead of letting go and trying to guess where to grab it again, TacRefineNet acts like a super-sensitive coach. It looks at the "feeling" of the current grip and compares it to a "feeling" of the perfect grip (which could be a memory from a human demonstration). Using a special AI brain, it calculates exactly how much to twist or slide the wrist to fix the grip. Then, the hand opens, moves, and grabs again, repeating this process until the "feeling" matches the target perfectly.

The researchers built this system entirely in a computer simulation first, training it on 156,007 fake "touch" samples of 15 different objects. They taught the AI a "Siamese" strategy, which is like having two identical twins (one looking at the current grip, one at the target grip) who compare notes to figure out the difference. The AI learned to ignore the parts of the object it couldn't feel (like spinning a perfectly round coin) and focus only on the parts it could sense, such as sliding up and down or tilting. When they tested this on a real robot hand with 11 moving parts and 5 fingers equipped with pressure sensors, the results were impressive. For objects the robot had seen before, it successfully fixed the grip 80.7% of the time for fixed targets and 59.3% for random targets, getting the object within about 5 millimeters and 3.5 degrees of the perfect spot after just five tries.

However, the paper is careful to note that this isn't a magic wand for every situation. The system works best when the object has distinct edges that create unique "fingerprints" for the sensors. If the object is a perfectly smooth, symmetrical disc, the sensors get confused because the feeling is the same no matter how you rotate it, and the success rate drops significantly. Also, while the robot learned entirely in a video game-like simulation, it still struggled a bit more when facing completely new objects it had never "felt" before, showing that while the technology is promising, it still has a gap to cross to become perfectly reliable in the real world. The authors conclude that this method is a powerful step forward for robots that need to handle delicate, edge-heavy items using only their sense of touch, but it works best within the specific range of movements it was trained on.

Technical Summary: TacRefineNet

Problem Statement
Accurate final grasp alignment remains a significant challenge for edge-prominent objects, such as thin plates, discs, and slender rods. For these objects, stable grasping and downstream task execution (e.g., insertion, assembly) depend on precise contact with thin, flat, or elongated edges. Visual feedback is often unreliable in these scenarios because contact regions are frequently occluded by fingers, and depth sensing struggles with reflective or thin structures. Consequently, small perception and control errors accumulate, leading to pose deviations that cause downstream failures. While existing methods utilize tactile sensing for grasp quality prediction or slip detection, there is a need for a framework that uses tactile feedback to directly control and iteratively refine the grasp pose without relying on vision, object models, or target-specific retraining.

Methodology: TacRefineNet
The authors propose TacRefineNet, a goal-conditioned, tactile-only framework for local grasp refinement. The system operates as an external-dexterity tactile servoing loop:

Input: The policy takes as input current and target multi-finger tactile images (from piezoresistive sensors) and their corresponding hand-joint configurations (proprioception).
Architecture: It employs a Siamese network architecture with weight-shared encoders (Vision Transformer-based) for both current and target tactile observations. Learnable positional embeddings preserve finger identity, and a cross-attention mechanism allows the current tactile features to attend to the target features. This is fused with proprioceptive states to predict corrective wrist-pose increments.
Output: The network directly regresses a 6-DoF wrist-pose increment ( $\Delta x$ ), encoded as a concatenated vector of a 9D rotation matrix and a 3D translation vector.
Execution Loop: The hand grasps the object, observes tactile feedback, predicts a corrective wrist motion, opens, moves to the new wrist pose, and regrasps. This process repeats until the tactile observation matches the target.
Training Strategy: The policy is trained entirely in a physics-based MuJoCo simulator using a cross-combination strategy. Current and target samples are randomly paired from the dataset to form training pairs, allowing the system to specify targets within the sampled pose range without retraining. The dataset comprises 156,007 simulated samples across 15 objects (plates, discs, rods).
Observability: The method refines only pose dimensions that are tactilely observable based on object geometry (e.g., $z$ -translation and roll for rods; $z$ , roll, and pitch for plates), avoiding noise injection from unobservable dimensions.

Key Contributions

Goal-Conditioned Tactile Servoing: A formulation that treats fine grasp execution as a tactile servoing problem, iteratively aligning current multi-finger tactile observations with a target observation via an external-dexterity regrasp loop. This eliminates the need for vision, object models, or target-specific retraining.
Siamese Multi-Finger Tactile Policy: The development of TacRefineNet, which jointly encodes paired tactile observations and proprioceptive states to directly regress relative pose increments. This avoids intermediate absolute-pose estimation, which is ill-conditioned for sparse contacts, and supports different targets within the local pose range.
Extensive Sim-to-Real Validation: The policy is trained entirely in simulation and deployed zero-shot on a real 11-DoF five-fingered hand with custom piezoresistive sensors. The evaluation covers seen objects, unseen objects within the same geometry class, and long-horizon dynamic perturbations.

Experimental Results

Performance on Seen Objects: On the real system, TacRefineNet achieved success rates of 80.7% (fixed targets) and 59.3% (random targets) under the $10^\circ/10$ mm criterion. After five refinement steps, the mean errors were approximately 5.2 mm (position) and 3.5 $^\circ$ (orientation).
Generalization: The method transfers to unseen objects within the same geometry category, though performance degrades when tactile contact patterns are less discriminative (e.g., symmetric discs). Rods showed the best transfer performance (25.0% success under random targets at $10^\circ/10$ mm) compared to discs and plates.
Ablation Studies: Removing tactile input caused a catastrophic drop in performance (success rates near 0%), confirming tactile sensing as the primary information source. Removing finger fusion or proprioception also degraded performance. Increasing the number of tactile fingers from two to three yielded the largest gains, with diminishing returns beyond three fingers.
Comparison: TacRefineNet significantly outperformed a baseline that estimated absolute 6-DoF poses independently and computed the relative transformation (85.9% vs. 54.1% success under $10^\circ/10$ mm in simulation).
Robustness: The system demonstrated continuous correction capabilities under long-horizon object perturbations, maintaining high-precision refinement toward a specified target pose.

Significance and Limitations
The paper claims that TacRefineNet provides a robust solution for the final stage of robotic grasping, particularly for edge-prominent objects where vision fails. By leveraging goal-conditioned tactile servoing, it achieves millimeter-level refinement accuracy without external perception.

However, the authors acknowledge specific limitations:

Dependence on Discriminative Contacts: Performance relies on informative tactile patterns; objects with symmetric or weakly discriminative contacts (like discs) remain challenging.
Local Scope: The framework is designed for local pose adjustments within the sampled dataset range, not for global grasp planning or large-scale corrections.
Sim-to-Real Gap: While zero-shot transfer is achieved, a residual gap affects performance, particularly on unseen objects with novel tactile features.

The authors suggest that future work could combine this tactile refinement with visual perception for global localization and coarse guidance.

Technical Summary: TacRefineNet

More like this