💻 computer science

Heterogeneous Tactile Transformer

The paper introduces the Heterogeneous Tactile Transformer (HTT), a framework leveraging a novel 1.6M-frame paired dataset to learn shared tactile representations across diverse sensor types, thereby enabling scalable, transferable contact-rich manipulation policies.

Original authors: Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot hand how to handle delicate objects, like picking up a ripe tomato or tightening a tiny screw. To do this safely, the robot needs "feel." But here's the problem: just like humans have different types of touch (some people are sensitive to texture, others to pressure), robots have different types of "touch sensors," and they all speak different languages.

Some sensors are like high-resolution cameras that take pictures of a squishy surface to see how it deforms (great for seeing shapes, but maybe a bit slow). Others are like arrays of pressure buttons that tell you exactly how hard something is pushing back, but they don't give you a clear picture of what the object looks like.

Because these sensors speak different "languages," a robot trained on one type of sensor usually can't understand data from another. It's like trying to teach a dog to speak French by only using English books; the dog just doesn't get it.

The Solution: The "Universal Translator" for Robot Touch

The authors of this paper created a new system called the Heterogeneous Tactile Transformer (HTT). Think of HTT as a universal translator that allows different types of robot touch sensors to understand each other and learn together.

Here is how they built it, using a simple analogy:

1. The "Language Exchange" Dataset (HPT)

To teach the translator, you need students who speak different languages but are talking about the same thing at the same time. The researchers built a massive dataset called HPT (Heterogeneous Paired Tactile).

The Setup: They put two different types of sensors on opposite sides of a robot gripper.
The Action: They made the robot touch, twist, and slide against 1.6 million different objects (like pressing a sponge, twisting a screw, or sliding a block).
The Result: Because the sensors were touching the exact same thing at the exact same time, the system could learn that "this squishy picture from the camera sensor" matches "this specific pressure pattern from the button sensor."

2. The Training Method: "Fill in the Blanks"

The system uses a clever training game called Masked Reconstruction (similar to a "fill-in-the-blanks" test).

The Game: The system takes a sensor's data, hides (masks) a big chunk of it, and asks the AI to guess what was missing.
The Twist: It doesn't just guess based on the same sensor. It looks at the other sensor's data to help fill in the blanks.
The Goal: By doing this, the AI learns a shared "inner language" (a latent space) where the concept of "holding a screw" looks the same, whether it's coming from a camera sensor or a pressure sensor.

3. The Results: Does it Work?

The researchers tested this "translator" in three ways:

Recognizing Objects: They asked the AI to identify objects just by feeling them. The HTT system was much better at this than previous methods, especially when switching between different sensor types. It learned that a "soft" feeling is a "soft" feeling, regardless of which sensor reported it.
Detecting Slips and Force: They tested if the AI could tell if an object was slipping or how hard it was being squeezed. The HTT system was excellent at this because it combined the "visual" detail of the camera sensors with the "force" detail of the pressure sensors.
Real-World Robot Tasks (The Big Test): This is the most impressive part. They put the HTT system on a real robot arm (a Franka arm with a Sharpa hand) to do two tricky tasks:
1. Tightening a Toy Screw: The robot had to grip and rotate a screw repeatedly.
2. Lifting Tofu: The robot had to pick up a piece of soft tofu without crushing it or dropping it.

The Outcome:

Without the HTT "translator," the robot failed miserably (it dropped the tofu or couldn't turn the screw).
With the HTT system, the robot became much more successful. It could feel the subtle shifts in contact needed to tighten the screw and knew exactly how much pressure to apply to the tofu so it wouldn't squish.

Why This Matters

The paper claims that this approach allows robots to learn from many different types of sensors at once. Instead of training a new robot brain for every new sensor you buy, you can train one "universal brain" (HTT) that understands all of them.

In short, the authors built a system that teaches robots to "speak touch" fluently, no matter which brand or type of sensor they are using, making them much better at handling delicate, real-world objects.

Technical Summary: Heterogeneous Tactile Transformer (HTT)

1. Problem Statement

Tactile sensing is essential for contact-rich robot manipulation (e.g., grasping fragile objects, precision assembly), yet scaling tactile learning remains a significant bottleneck. Unlike cameras, tactile sensors are inherently heterogeneous: they differ fundamentally in transduction mechanisms, data structures, and output formats.

Optical sensors (e.g., GelSight, 9DTact) capture rich spatial and geometric information via elastomer deformation but are often limited by camera frame rates.
Array-based sensors (e.g., Xela, Tac-02) provide high-rate temporal signals and direct force/pressure measurements but offer lower spatial resolution.

Current representation learning methods largely focus on optical sensors or treat sensors in isolation. Models trained on one modality cannot directly ingest data from another, preventing the creation of unified, sensor-agnostic manipulation policies. There is a lack of large-scale, synchronized datasets pairing these distinct modalities, and no existing framework effectively learns shared representations across them.

2. Methodology

2.1 The Heterogeneous Paired Tactile (HPT) Dataset

To address the data scarcity, the authors constructed the Heterogeneous Paired Tactile (HPT) dataset using a Universal Manipulation Interface (UMI) setup.

Scale: 1.6 million synchronized paired frames.
Sensors: Four distinct sensors: two optical (GelSight Mini, 9DTact) and two array-based (Xela, Tac-02).
Collection: Sensors were mounted on opposing sides of gripper fingers to capture simultaneous contact properties during interactions with daily-life objects.
Components: The dataset includes:
- Unlabeled Interaction Data: For self-supervised pretraining (press, twist, slide motions).
- Object Classification: 20 distinct objects with varied geometries and textures.
- Force Estimation: Synchronized 6-D force labels collected via a probing rig with an external F/T sensor.
- Slip Detection: Labels derived from friction-coefficient time series, categorized into static, incipient, and gross slip.

2.2 Heterogeneous Tactile Transformer (HTT) Architecture

HTT is a self-supervised framework designed to learn a shared latent space across heterogeneous sensors. It adopts a Masked Autoencoder (MAE) design with the following components:

Sensor-Specific Encoders ( $E_i$ ):
- For optical sensors: Vision Transformer (ViT) encoders processing spatial patches.
- For array sensors: Self-attention transformers processing temporal patches of multi-dimensional time series.
- Inputs are patchified and normalized before encoding.
Shared Transformer Trunk ( $T$ ): A unified backbone that processes token embeddings from all sensors into a common latent space.
Sensor-Specific Decoders ( $D_i$ ): Used during pretraining to reconstruct masked input tokens.
Cross-Modal Predictors ( $P_{ij}$ ): Cross-attention transformers that predict masked target sensor embeddings using source sensor embeddings and visible target embeddings.

2.3 Pretraining Objective

The model is trained using a joint loss function combining two objectives:

Per-Modality Masked Reconstruction ( $L_{MAE}$ ): Encourages each sensor to extract robust structural and physical features by reconstructing its own masked input.
Cross-Modal Alignment ( $L_{Align}$ ): Trains the model to predict masked target sensor embeddings from the source sensor's full embedding and the target's visible embeddings. This aligns disparate modalities in the shared latent space.
- A stop-gradient is applied to the target to prevent representation collapse.
- The alignment loss is weighted by a time-dependent coefficient ( $\alpha_t$ ), starting at 0 (warmup) to allow sensor-specific feature development before ramping up.

During downstream tasks, the decoders and predictors are discarded; only the sensor encoders and the shared trunk are used.

3. Key Contributions

HPT Dataset: A large-scale (1.6M frames), synchronized dataset pairing optical and array-based tactile sensors, enabling the study of heterogeneous tactile representation learning.
HTT Framework: A self-supervised architecture that combines MAE-style reconstruction with bidirectional cross-sensor prediction to learn transferable representations across heterogeneous sensor types.
Comprehensive Evaluation: Extensive validation across perception tasks (classification, force estimation, slip detection) and manipulation tasks (simulated and real-world), demonstrating the framework's ability to generalize to unseen sensors and tasks.

4. Experimental Results

4.1 Tactile Perception Tasks

Object Classification: HTT outperformed the "Scratch" baseline on all four sensors. On optical sensors, HTT surpassed the strongest optical-only baselines (T3, SITR) by significant margins (e.g., +13.5% on 9DTact, +17% on GSMini). Cross-modal alignment improved performance on optical sensors but showed mixed results on array sensors (slight regression on Xela), attributed to information imbalance between modalities.
Force Estimation & Slip Detection: HTT substantially outperformed baselines on all sensors. Optical-only pretraining methods (T3, SITR) often failed to transfer to force-sensitive tasks, whereas HTT maintained robust performance. The cross-modal alignment was particularly effective for slip detection, improving macro-F1 scores significantly (e.g., +12.0% on TAC-02) by binding complementary force and spatial cues.

4.2 Robot Manipulation Tasks

Real-World Experiments (Sharpa Hand on Franka Arm):
- Tasks: "Toy Screw" (tightening a screw) and "Grasp Tofu" (lifting soft tofu without crushing).
- Setup: Zero-shot adaptation using HTT embeddings from a sensor (9DTact encoder) not seen during pretraining.
- Results: Policies using HTT embeddings significantly outperformed those using raw 6-D wrench vectors or joint positions alone.
  - Toy Screw: HTT achieved 95% success vs. 50% for wrench-based policies. The authors attribute this to HTT's ability to represent spatial contact features (contact location and shift) required for cyclic regrip, which raw force vectors cannot capture.
  - Grasp Tofu: HTT achieved 55% success vs. 35% for wrench-based policies, reducing slip rates by one-third and eliminating crushing failures.
Simulated Tasks (ManiFeel): HTT embeddings improved success rates in Peg Insertion and Bulb Installation tasks compared to baselines, confirming the utility of sensor-agnostic contact embeddings.

5. Significance and Claims

The paper claims that paired heterogeneous pretraining is a practical path toward general, sensor-agnostic tactile representation learning.

Transferability: HTT learns representations that adapt to new tasks and previously unseen sensors without requiring sensor-specific pretraining.
Policy Learning: The framework provides richer contact representations than raw force readouts, directly boosting the performance of contact-rich manipulation policies.
Scalability: By bridging the gap between optical and array-based sensors, HTT addresses a critical bottleneck in scaling tactile learning, moving beyond the limitations of single-modality datasets.

The authors note limitations, including the current focus on only optical and array families, the lack of same-family pairing studies, and the absence of explicit geometric space alignment between sensor patches. Future work aims to extend alignment to other sensing modalities and refine cross-modal supervision.