How to Train your Tactile Model: Tactile Perception with Multi-fingered Robot Hands

1. Problem Statement

The rapid deployment of vision-based tactile sensors (VBTS), such as the TacTip, in multi-fingered robot hands is hindered by the limitations of current deep learning approaches.

• Sensor-Specific Dependency: Current state-of-the-art methods rely on Convolutional Neural Networks (CNNs). While effective on a single, known sensor, CNNs struggle to generalize to new sensors of the same design.
• Domain Shift: Minor variations in manufacturing (lens properties, illumination, sensor wear) create distribution shifts that significantly degrade CNN performance.
• Scalability Bottleneck: To deploy a new tactile sensor on a multi-fingered hand, engineers must currently collect thousands of new domain-specific data points and retrain a model for every individual sensor. This makes large-scale, "plug-and-play" tactile sensing impractical.

2. Methodology: TacViT

The authors propose TacViT, a novel tactile perception model based on Vision Transformers (ViTs) designed to generalize across unseen sensor domains without extensive retraining.

• Architecture:
  • Backbone: Utilizes a pre-trained ViT (Google's vit-base-patch16-224 trained on ImageNet) to extract feature embeddings from tactile images.
  • Mechanism: Unlike CNNs, which rely on local spatial hierarchies, ViTs use self-attention mechanisms to model long-range dependencies and global features across the entire image. This allows the model to be robust against local artifacts (e.g., glare, lens distortion) specific to individual sensors.
  • Regression Head: The extracted embeddings are passed through a series of fully connected (FC) layers to predict contact pose ($x, y, z, R_x, R_y$) and contact forces ($F_x, F_y, F_z$).
• Training Strategy:
  • Fine-tuning with LoRA: To improve computational efficiency and stability, the authors employ Low-Rank Adaptation (LoRA). Instead of updating all weights, low-rank matrices are inserted into the self-attention layers.
  • Gradual Unfreezing: The training process begins by training only the regression head while keeping the transformer frozen. Selected transformer layers are unfrozen in later epochs to prevent catastrophic forgetting.
  • Loss Function: Mean Squared Error (MSE) is used to minimize the difference between predicted and ground-truth values.

3. Experimental Setup

The model was evaluated on a five-fingered robot hand equipped with five distinct TacTip-style sensors. Three specific experimental scenarios were designed to test generalization:

1. Tr1–Te1 (Baseline): Train and test on the same single sensor.
2. Tr5–Te1 (Multi-Source): Train on data from all five sensors and test on one sensor seen during training.
3. Tr4–TeU (Unseen Generalization): Train on data from four sensors and test on the fifth, unseen sensor.

The performance of TacViT was compared against a standard CNN architecture using Mean Absolute Error (MAE) for pose and force variables.

4. Key Results

The results demonstrate a clear superiority of TacViT in generalization tasks compared to CNNs.

• Tr1–Te1 & Tr5–Te1 (Known Sensors): Both TacViT and CNN performed well with low MAE and high consistency. TacViT was slightly less accurate than CNNs in these controlled, same-domain scenarios but remained within acceptable error margins for robotics ($z_{MAE} < 0.1$ mm, $R_{MAE} < 2.5^\circ$).
• Tr4–TeU (Unseen Sensor): This was the critical differentiator.
  • CNN Performance: Degraded significantly. The mean MAE increased by an order of magnitude (e.g., $z$-error jumped from ~0.08mm to ~0.21mm). The error distribution showed a wide spread (up to 2.2 standard deviations), indicating high instability.
  • TacViT Performance: Maintained robust performance. While accuracy decreased slightly compared to the Tr5-Te1 baseline, the error remained within acceptable operational limits. Crucially, the standard deviation remained low, indicating consistent predictions across different sensor domains.
• Visual Analysis: Scatter plots of predicted vs. ground truth values showed that CNN predictions for unseen sensors became noisy and often failed to predict values in the correct range (e.g., predicting $z$ starting at 2mm instead of 1mm). TacViT predictions remained clustered around the true values.

5. Key Contributions

1. Novel Architecture: Introduction of TacViT, the first ViT-based model specifically designed for tactile perception that generalizes to new, unseen sensors.
2. Scalability: Demonstration that TacViT eliminates the need for sensor-specific data collection and retraining, enabling faster deployment of new tactile sensors in multi-fingered hands.
3. Empirical Validation: Comprehensive validation across five distinct sensors on a robot hand, proving that ViTs outperform CNNs in cross-sensor generalization tasks.

6. Significance and Future Work

• Impact: This work addresses a major bottleneck in robotic manipulation: the inability to easily swap or add tactile sensors without retraining. TacViT moves the field toward plug-and-play tactile sensing, which is critical for complex, multi-fingered robotic hands where sensor interchangeability is vital.
• Mechanism Insight: The success of ViTs is attributed to their global self-attention, which integrates information across the entire image, making them less susceptible to local sensor-specific artifacts (glare, wear) that confuse CNNs.
• Future Directions:
  • Training ViTs from scratch on large-scale tactile datasets (rather than fine-tuning from ImageNet) to potentially further improve performance.
  • Tailoring architecture designs to tactile-specific priors (e.g., symmetry, surface topology).
  • Testing TacViT on other types of vision-based tactile sensors beyond the TacTip family.

In conclusion, the paper establishes that Vision Transformers offer a robust, scalable solution for tactile perception, significantly reducing the data and computational overhead required to deploy new tactile sensors in real-world robotic applications.