💻 computer science

HT-Bench: Benchmarking and Learning Dexterous Full-Hand Tactile Representations with Egocentric Vision

This paper introduces HT-Bench, a large-scale multi-task benchmark for dexterous full-hand tactile sensing, and proposes HandTouch, a vector-quantized vision-tactile encoder that significantly outperforms existing baselines in learning tactile representations through egocentric vision and full-hand tactile data.

Original authors: Yuzhe Huang, Jiaping Wu, Jiaming Jiang, Hezhe Lin, Aikebaier Aierken, Yunlong Wang, Kun Cheng, Ziyuan Jiao, Yuanxin Zhong

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuzhe Huang, Jiaping Wu, Jiaming Jiang, Hezhe Lin, Aikebaier Aierken, Yunlong Wang, Kun Cheng, Ziyuan Jiao, Yuanxin Zhong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to teach a robot hand to "feel" the world. For a long time, scientists have struggled to create a standard test for this, much like trying to compare the taste of apples from 200 different orchards when every orchard uses a different scale, a different type of fruit, and a different way of measuring sweetness.

This paper, HT-Bench, tackles that problem by changing the game. Instead of trying to force every robot hand to be the same, the authors created a massive, standardized "gym" specifically for robots that use eyes on the wrist (egocentric vision) and sensitive skin all over the hand (full-hand tactile sensing).

Here is the breakdown of their work using simple analogies:

1. The Problem: A Messy Kitchen of Sensors

Currently, robot tactile sensors are like a kitchen where everyone uses different measuring cups. Some sensors measure pressure, others measure slip, and they all come in different shapes. Because of this mess, it's hard to know if a robot is actually "learning" to feel or if it's just memorizing a specific trick for one specific sensor.

2. The Solution: HT-Bench (The "Gym")

The authors built HT-Bench, a giant dataset containing 10 million video frames and 7.8 million tactile frames. Think of this as a massive library of "movies" where a robot hand is doing 226 different tasks (like picking up a cup or turning a knob), and we can see both what the robot sees and exactly how its skin feels every single moment.

To test if a robot's "brain" is good at feeling, they don't just ask it to do one job. They give it four different types of exams:

The "Memory Match" Game (Retrieval): The robot is shown a picture of a hand touching something and asked, "Which of these 20 other hand-touches feels exactly like this one?" It's like a memory game where you have to match the feeling of a texture, not just the look.
The "Fill-in-the-Blanks" Puzzle (Inpainting): Imagine a hand touching a ball, but half the sensors on the hand suddenly go blind (like a patch of dead pixels on a screen). The robot has to guess what the missing sensors should be feeling based on the rest of the hand and what it sees with its eyes.
The "Mind Reader" Test (Vision-to-Tactile): The robot sees a picture of an object (like a sponge or a rock) but hasn't touched it yet. It has to predict exactly how the pressure will feel on its skin just by looking at the object. This is like seeing a picture of a lemon and instinctively knowing it will feel sour and bumpy before you even touch it.
The "Crystal Ball" Test (Prediction): The robot watches a hand moving and touching things over time. It has to predict what the hand will feel next second, based on what it saw and felt just a moment ago.

3. The New Brain: HandTouch

To pass these exams, the authors built a new AI model called HandTouch. They trained it in three progressive stages, like a student learning a new language:

Stage 1 (Learning the Alphabet): The robot learns to look at a full map of its hand's touch and reconstruct it perfectly. It learns the "shape" of how pressure spreads across the hand.
Stage 2 (Learning to Read Context): The robot is given a broken map (with missing parts) and a picture of the scene. It learns to use the picture to fill in the missing parts of the touch map. It learns that "if I see a soft pillow, my hand should feel soft, even if my sensors are broken."
Stage 3 (Learning the Story): The robot learns to watch a sequence of events and predict the next feeling. It understands that if a hand is sliding down a ramp, the feeling will change in a specific way.

4. The Results: A New Champion

When they put HandTouch through the HT-Bench gym, it crushed the competition.

Memory Match: It got better at matching similar feelings than any previous model (improving from about 75% to 85% accuracy).
Fill-in-the-Blanks: It made far fewer mistakes when guessing missing touch data (reducing errors by more than half).
Mind Reader: It got much better at predicting how an object feels just by looking at it.

The Bottom Line

The paper concludes that while we can't yet build a single test for every robot hand in existence, we can build a massive, high-quality standard for robots that use eyes and full-hand sensors. HT-Bench provides the playground, and HandTouch proves that with the right training, robots can learn to "feel" the world in a way that is smart, adaptable, and generalizes to new tasks they've never seen before.

What they did NOT claim:

They did not say this robot can now perform surgery or handle delicate clinical tasks.
They did not claim this works on every type of robot sensor (like just a fingertip sensor or a force sensor on a leg).
They did not say this is ready for immediate commercial use in homes; it is a research benchmark to help future robots get there.

Technical Summary: HT-Bench and HandTouch

Problem Statement

Establishing a universal benchmark for tactile representation learning in robotic manipulation is currently hindered by the intrinsic heterogeneity of tactile sensing. This heterogeneity arises from diverse sensor designs (hardware principles, spatial layouts, signal modalities), varying robot embodiments, and inconsistent data formats. Unlike visual perception, which benefits from standardized datasets and mature evaluation protocols, tactile learning remains constrained by task-specific pipelines and sensor-dependent processing. Consequently, it is difficult to determine whether learned features represent transferable tactile representations or merely task-specific adaptations. The authors argue that seeking a single universal benchmark to resolve all sensor discrepancies is impractical. Instead, they propose exploring a scalable, promising direction: egocentric vision paired with full-hand tactile data. This setting captures interaction-centric visual context alongside distributed contact information during dexterous manipulation, providing a basis to evaluate whether tactile encoders can encode contact geometry, align with visual information, and generalize to unseen tasks.

Methodology

1. HT-Bench: A Large-Scale Multi-Task Benchmark

The authors introduce HT-Bench, a benchmark comprising approximately 10 million RGB frames and 7.8 million tactile frames collected across 226 diverse tasks in various environments (e.g., home, electronics workbench, chemistry lab). The dataset is synchronized between egocentric vision and full-hand tactile sensors.

To evaluate generalization, HT-Bench employs a task-level Out-of-Distribution (OOD) split strategy: one interaction task is held out for OOD evaluation, while the remaining tasks are split 9:1 for training and in-distribution testing.

HT-Bench evaluates tactile representations through four complementary tasks:

Fine-Grained Tactile Similarity Retrieval: A 1-vs-20 retrieval task where encoders must rank candidate tactile maps based on Structural Similarity Index Measure (SSIM) against a query, testing the preservation of fine-grained contact geometry.
Masked Tactile Inpainting: The model reconstructs missing regions of a tactile map using remaining tactile observations and synchronized visual cues. This tests spatial reasoning and the ability to handle partial information loss (simulating sensor failures).
Vision-to-Tactile Synthesis (RGB-to-Tactile): Given a single RGB frame, the model predicts the corresponding tactile pressure distribution. This evaluates cross-modal alignment and the ability to infer tactile responses from visual object geometry.
Multimodal Tactile Frame Prediction: The model predicts the tactile distribution at time $T$ ( $t_T$ ) using recent visual observations ( $v_{T-2}$ ) and past tactile trajectories ( $t_{T-2}$ ). This assesses temporal contact modeling and dynamic interaction understanding.

2. HandTouch: Vector-Quantized Vision–Tactile Encoder

To address the evaluation requirements of HT-Bench, the authors propose HandTouch, a vector-quantized vision–tactile encoder trained via a progressive three-stage pipeline:

Stage 1: Vector-Quantized Tactile Reconstruction (Spatial Topology):
- A convolutional layer tokenizes normalized tactile maps into patches, which are processed by an 8-layer Vision Transformer (ViT) encoder.
- A factorized vector quantizer with a shared codebook ( $K=2048$ ) projects continuous latent features into a discrete token space.
- To prevent codebook collapse, a codebook restart mechanism reinitializes underutilized entries using active features from the current batch.
- The objective is to reconstruct the input tactile map from quantized tokens, learning a structurally discriminative spatial topology.
Stage 2: Cross-Modal Masked Tactile Inpainting (Vision–Touch Alignment):
- The model learns to reconstruct corrupted tactile maps using both remaining tactile data and synchronized visual cues.
- A curriculum-based dual masking strategy is employed:
  - Regional Random Masking: Masks specific hand parts (thumb, fingers, palm) to simulate partial loss.
  - Complete Masking: Masks the entire tactile map, forcing the model to infer tactile responses primarily from visual context. The probability of complete masking increases over training.
- Visual features (from a frozen pre-trained ViT) are injected into the tactile decoder via cross-attention. A learnable hand-specific token ( $t_{hand}$ ) is added to resolve geometric ambiguities between left and right hands.
Stage 3: Multimodal Tactile Frame Prediction (Temporal Dynamics):
- The model integrates recent visual observations and past tactile histories to predict the future tactile distribution.
- This stage fine-tunes the prediction modules, shared codebook, and decoder to capture the temporal evolution of contact states.

Key Contributions

HT-Bench: The introduction of a large-scale, multi-task benchmark specifically designed for dexterous full-hand tactile sensing, featuring synchronized egocentric vision and tactile data, and task-level OOD splits to rigorously evaluate generalization.
HandTouch: A novel vector-quantized vision–tactile encoder that learns transferable representations through progressive spatial, cross-modal, and temporal training objectives.
Comprehensive Evaluation: A multi-perspective evaluation framework assessing contact geometry encoding, visual alignment, and generalization to unseen tasks, moving beyond single downstream objectives.

Experimental Results

Experiments on HT-Bench demonstrate that HandTouch consistently outperforms representative baselines (CNN-based, ResNet-18, VQ-VAE, and ViT-based encoders):

Fine-Grained Retrieval: HandTouch improved Recall@5 from 74.65% to 85.23% and Hit@1 from 94.27% to 99.27% compared to the strongest baseline (ViT-based).
Masked Tactile Inpainting: On the standard test split, HandTouch reduced full-image RMSE from 0.022 to 0.010 and increased cIoU from 0.762 to 0.911. It maintained superior performance on OOD splits as well.
Vision-to-Tactile Synthesis: HandTouch improved OOD cIoU from 0.628 to 0.705, indicating stronger cross-modal generation and generalization. While VQ-VAE showed slightly lower RMSE in OOD settings, HandTouch achieved significantly higher cIoU, suggesting better capture of sharp, localized contact patterns.
Temporal Prediction: HandTouch demonstrated accurate prediction of tactile distributions in both in-distribution and OOD settings, validating its ability to model temporal contact dynamics.

Significance and Claims

The paper claims that HT-Bench provides a scalable and practical basis for evaluating and advancing tactile representation learning in dexterous manipulation, specifically within the regime of egocentric vision paired with full-hand sensing. By shifting focus from a universal (but currently impractical) benchmark to a scalable, data-driven approach, the authors aim to answer the prerequisite question of what constitutes an effective representation backbone for full-hand tactile perception.

The authors assert that HandTouch demonstrates the effectiveness of progressive training (spatial, cross-modal, temporal) and vector quantization in learning representations that are structurally discriminative, visually grounded, and temporally aware. The results suggest that large-scale egocentric full-hand tactile data can serve as a foundation for general tactile representation learning.

The paper remains modest regarding its scope, acknowledging that HT-Bench does not cover all tactile sensing paradigms (e.g., fingertip-only sensors, force/torque sensing, or non-hand embodiments) and that current evaluations focus on representation-level capabilities rather than direct downstream robotic performance. Future work is identified as necessary to extend the benchmark to broader hardware, conduct deeper ablation studies, and validate performance in real-world closed-loop manipulation tasks.