💻 computer science

VQ-Touch: A Data-Efficient Tactile Generation Framework Across Sensors and Scenarios

VQ-Touch is a data-efficient tactile generation framework that leverages a novel DM-VQGAN representation learner and a discrete diffusion decoder with few-shot mixed training to synthesize high-fidelity tactile data across diverse sensors and scenarios, thereby overcoming the limitations of existing methods in generalization and data dependency.

Original authors: Kailin Lyu, Long Xiao, Jianing Zeng, Di Wu, Lin Shu, Jie Hao

Published 2026-07-17

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Kailin Lyu, Long Xiao, Jianing Zeng, Di Wu, Lin Shu, Jie Hao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot trying to pick up a ripe strawberry. It has great eyes, but eyes can't feel if the fruit is soft or if the stem is about to snap. To fix this, engineers give robots "skin"—special sensors that squish and stretch when they touch things, turning pressure and texture into pictures. These are called vision-based tactile sensors. They are like tiny, high-tech cameras looking at a squishy rubber sheet; when you press the sheet, the wrinkles tell the robot what it's touching. The problem is, these sensors are fragile, expensive, and collecting real-world "touch photos" is a slow, messy job. It's like trying to learn how to play the piano by only listening to one person practice in a noisy room; you need a lot of data to get good, but getting that data is a pain. Scientists have been trying to use computers to "dream up" or synthesize these touch pictures instead of taking them all in real life, but previous attempts were like trying to teach a robot to feel using only one specific type of glove. If the robot got a different glove, the computer got confused.

Enter VQ-Touch, a new framework that acts like a universal translator for robot skin. The researchers behind this project realized that instead of teaching a robot to feel with every single sensor type separately, they could teach it a "language of touch" that works for almost any sensor. They built a system called DM-VQGAN, which is essentially a super-smart artist that learns to see the most important parts of a touch picture—the big squishes and the tiny textures—while ignoring the boring background noise. Think of it as a sketch artist who only draws the essential lines of a face, ignoring the hair color or the background, so the drawing works no matter who is sitting in the chair.

But here is the real magic: the team figured out that many different sensors are actually "cousins." They grouped these sensors into families, like how a Golden Retriever and a Labrador are both dogs. By using a clever trick called few-shot mixed training, they showed the computer a tiny handful of pictures from a new, unseen sensor (maybe just 50 images) and mixed them with data from its "cousin" sensors. The computer quickly figured out, "Oh, this new sensor is part of the same family!" and instantly learned how to generate touch pictures for it without needing thousands of examples. They then used a discrete diffusion decoder, which is like a creative writer who can take a prompt—like a photo of a rock, a word like "tree," or even a different touch picture—and write a brand new, high-quality touch picture that matches the prompt perfectly.

The results are impressive. When tested, VQ-Touch didn't just guess; it recreated touch images with much higher accuracy than previous methods, even when it had very little data to work with. In fact, it could generate touch pictures for sensors it had never seen before just by looking at a few examples and knowing the sensor's "family." Whether the robot needed to recognize an object by its texture or simulate a touch scene for training, this new framework did it better, faster, and with less data than the state-of-the-art models. It suggests that we might soon be able to teach robots to feel the world as easily as we teach them to see it, without needing to spend years collecting physical data for every single new tool they might use.

Technical Summary: VQ-Touch

Problem Statement

Tactile image generation aims to synthesize high-fidelity tactile data to reduce reliance on expensive, wear-prone vision-based tactile sensors (VBTS) and the laborious real-world data acquisition pipeline. While existing methods have made progress, they face two critical limitations:

Data Inefficiency: They require large-scale, diverse datasets for pretraining specific sensor types, leading to degraded performance in data-scarce or vision-limited environments.
Poor Generalization: Learned features often fail to generalize to image generation tasks involving different sensors, hindering cross-sensor applicability.

The core challenge addressed is how to learn a data-efficient tactile representation that facilitates generation across different sensors and scenarios without requiring full retraining on massive datasets for every new device.

Methodology

The proposed framework, VQ-Touch, operates through a three-stage pipeline designed for cross-sensor and multi-scenario applications:

1. DM-VQGAN: Efficient Tactile Representation Learning

To model the low-variance latent space of tactile images while preserving rich geometric and texture information, the authors propose DM-VQGAN, a tactile-specific encoder-decoder architecture based on Vector Quantized GANs (VQGAN).

Deformable Convolutions: Unlike standard VQGANs that use fixed convolutions, DM-VQGAN integrates deformable convolutions to adaptively learn dynamic spatial offsets. This allows the model to capture the non-rigid, complex, and irregular deformation patterns characteristic of VBTS.
Multi-scale Fusion: A parallel dilated convolution module (with rates 1, 2, and 4) is employed to simultaneously extract macro-level global deformations and micro-level textural features, expanding the receptive field.
Training Objective: The model minimizes reconstruction loss ( $L_1$ ), vector quantization loss (including commitment loss), perceptual loss, and adversarial loss.

2. Few-shot Mixed Training for Cross-Sensor Generalization

To address the scarcity of data for new sensors, VQ-Touch introduces a strategy based on sensor families.

Sensor Family Clustering: The framework identifies that sensors with shared intrinsic properties (design/manufacturing) but different environmental conditions belong to the same "sensor family." An unsupervised clustering strategy using prototype vectors groups sensors based on feature centroids extracted by DM-VQGAN.
Mixed Training: Instead of full retraining, the system constructs a small-sample mixed dataset. This combines a few images from an unseen sensor with existing data from its identified sensor family (e.g., a 1:5 ratio). DM-VQGAN is trained on this mixture, allowing the learned representations to transfer to the new sensor with minimal additional data.

3. Unified Discrete Diffusion Decoder

A discrete diffusion model serves as the generative decoder, conditioned on a unified interface supporting multiple input modalities:

Multimodal Conditioning: The decoder can be conditioned on visual images (via a frozen CLIP encoder), raw tactile images (mapped to discrete indices via the sensor-specific codebook), or semantic labels.
Generation Process: The conditional latent features interact with noisy latent representations through transformer blocks to synthesize high-quality tactile images, enabling tasks such as zero-shot transfer and cross-sensor generation.

Key Contributions

DM-VQGAN: An efficient tactile representation learner that captures generalizable, information-rich features tailored to the specific deformation and texture characteristics of tactile data.
Unified Multimodal Framework: A conditional generation system supporting diverse inputs (visual, tactile, labels) to address varied generation requirements.
Sensor-Family Transfer Strategy: A novel approach using clustering and few-shot mixed training to transfer features from known sensors to new variants within the same family, eliminating the need for full multi-dataset training.
Performance: The framework achieves state-of-the-art results in tactile image reconstruction and generation, particularly excelling in vision-limited and data-scarce scenarios.

Experimental Results

The authors evaluated VQ-Touch on four public datasets (Touch and Go, FabricVST, VisGel, HCT) covering GelSight, GelSight Mini, and DIGIT sensors.

Reconstruction Quality: In reconstruction tasks, DM-VQGAN consistently outperformed baselines (MAE, VQGAN) across varying sample sizes (100 to 10k). Notably, DM-VQGAN trained with only 100 samples achieved better results than MAE trained with 10k samples, preserving fine tactile structures that MAE failed to recover.
Generation Metrics: On tactile generation tasks (HCT and SSVTP datasets), VQ-Touch achieved the best scores in CTTP, LPIPS, SSIM, and PSNR compared to UniTouch, PixArt-α, and TextToucher.
Few-Shot and Zero-Shot: The model demonstrated superior few-shot representation learning and zero-shot generalization to unseen objects. In cross-sensor reconstruction, training with a small number of unseen sensor images mixed with family data allowed for effective transfer, as shown in reconstruction comparisons.
Downstream Utility: Classifiers trained on VQ-Touch generated images achieved higher object recognition accuracy (e.g., 92.48% on FabricVST) compared to those trained on images generated by PixArt-α (88.79%).

Significance

The paper positions VQ-Touch as a solution to the bottleneck of tactile data scarcity. By establishing a new paradigm for few-shot tactile generation, the framework enables cross-sensor compatibility and vision-free multimodal-to-tactile generation. The authors claim that this approach significantly lowers the training cost for downstream robotic tasks (such as manipulation and texture recognition) while maintaining high fidelity. The sensor-family clustering and mixed training strategy are highlighted as key enablers for scalable tactile perception in robotics and human-computer interaction, offering a robust alternative to physical data collection.