💻 computer science

TouchGuide: Inference-Time Steering of Visuomotor Policies via Touch Guidance

TouchGuide is a novel inference-time steering framework that enhances pre-trained visuomotor policies for contact-rich manipulation by integrating a contrastively trained Contact Physical Model to refine coarse visual actions with tactile guidance, supported by the cost-effective TacUMI data collection system.

Original authors: Zhemeng Zhang, Jiahua Ma, Xincheng Yang, Xin Wen, Yuzhi Zhang, Boyan Li, Yiran Qin, Jin Liu, Can Zhao, Li Kang, Haoqin Hong, Zhenfei Yin, Philip Torr, Hao Su, Ruimao Zhang, Daolin Ma

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhemeng Zhang, Jiahua Ma, Xincheng Yang, Xin Wen, Yuzhi Zhang, Boyan Li, Yiran Qin, Jin Liu, Can Zhao, Li Kang, Haoqin Hong, Zhenfei Yin, Philip Torr, Hao Su, Ruimao Zhang, Daolin Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to perform delicate tasks, like tying a shoelace or handing over a fragile potato chip without breaking it. If you only give the robot "eyes" (cameras), it often struggles. It might see the shoe, but it can't feel if the lace is slipping or if it's holding the chip too hard. It's like trying to thread a needle while wearing thick winter gloves; you can see the hole, but you can't feel the thread.

This paper introduces TouchGuide, a new way to teach robots to use their "sense of touch" to fix their mistakes in real-time, without needing to retrain the entire robot brain from scratch.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Blind" Robot

Current robots are great at seeing the big picture (like "pick up the shoe"), but they are bad at the tiny, contact-heavy details (like "is the lace tight enough?"). When they try to do these tasks, they often fail because they rely too much on vision and not enough on the physical feeling of the object.

2. The Solution: The "Touch Guide" (TouchGuide)

The authors created a system called TouchGuide. Think of the robot's main brain (the policy) as a driver who is very good at driving on a straight, empty road (visual tasks). However, when the road gets bumpy and requires delicate steering (contact tasks), the driver gets lost.

TouchGuide acts as a co-pilot who sits next to the driver.

The Driver (Base Policy): First, the driver looks out the window and makes a rough guess about where to steer the car based on what they see. This is the "coarse action."
The Co-pilot (TouchGuide): Before the car actually moves, the co-pilot checks the road conditions using a special sensor (touch). If the driver's guess would cause a crash or a slip, the co-pilot gently nudges the steering wheel to correct the path.
The Result: The car (robot) follows a path that looks good visually and feels physically safe.

Crucially, this co-pilot doesn't need to teach the driver how to drive from scratch. It just steers the driver's existing decisions at the very last moment (during "inference") to ensure the action makes physical sense.

3. The Tool: The "Smart Hand" (TacUMI)

To teach this co-pilot, you need high-quality data. The authors also built a new data collection tool called TacUMI.

Imagine trying to teach a robot by having a human mimic the robot's movements.

Old Way (Teleoperation): The human wears a VR headset and holds a controller. They can see the robot's view, but they can't feel the object. It's like trying to learn how to catch a fragile egg by watching a video of someone else doing it. You might hesitate or squeeze too hard because you can't feel the egg.
The TacUMI Way: The human holds a handheld device that looks exactly like the robot's gripper. It has rigid fingertips that connect directly to the human's fingers. When the human touches the object, they feel it instantly, just like their own hand.
- Analogy: It's the difference between trying to thread a needle while looking at a screen versus doing it with your own fingers. The TacUMI system lets humans collect "perfect" data because they can feel exactly how hard to squeeze and where to hold.

4. How It Works Together

The paper tested this on five tricky tasks:

Shoe Lacing: Threading a string through tiny holes.
Chip Handover: Passing a potato chip without crushing it.
Cucumber Peeling: Peeling a vegetable without cutting the flesh.
Vase Wiping: Wiping a curved vase without knocking it over.
Lock Opening: Putting a key in a lock and turning it.

The Results:

Without TouchGuide, the robots often dropped the chip, broke the lace, or couldn't turn the key.
With TouchGuide, the robots used the "co-pilot" to adjust their grip and angle in real-time.
Success: The robots became significantly better at these tasks. For example, in the "Chip Handover" task, the success rate jumped from roughly 25% to 60%. In "Lock Opening," it went from 20% to 30% (and much higher with the best setup).

5. The "Secret Sauce": The Contact Physical Model (CPM)

How does the co-pilot know when to steer? It uses a small brain called the Contact Physical Model (CPM).

This model is trained on the high-quality data collected by TacUMI.
It learns a simple rule: "Does this action feel physically possible?"
If the robot tries to grab a chip too hard, the CPM says, "No, that will break it," and steers the action toward a gentler grip.
It acts like a reality check, ensuring the robot's plan matches the laws of physics.

Summary

The paper claims that by adding a "touch guide" that corrects a robot's visual plans in real-time, and by using a new handheld tool that lets humans feel exactly what the robot feels, robots can finally master delicate, contact-heavy tasks that were previously too difficult for them. They don't need to be retrained from scratch; they just need a little nudge from a touch-sensing co-pilot.

Technical Summary: TouchGuide

Problem Statement

Fine-grained and contact-rich robotic manipulation (e.g., shoe lacing, handing over fragile chips) remains a significant challenge for robots. While imitation learning has enabled complex tasks, many intuitive skills require the effective integration of visual and tactile information. Visual inputs provide global context, but tactile feedback is essential for localized, contact-specific control. Existing visuo-tactile policies typically rely on feature-level concatenation (which can suffer from modality dominance) or policy-level composition (which often requires training multiple single-modality policies and struggles to capture critical cross-modal correlations). Furthermore, existing data collection systems for tactile learning often fail to balance high precision with low cost, or rely on indirect haptic feedback (e.g., VR visualization) that hinders intuitive control and data quality.

Methodology

The paper introduces TouchGuide, a novel cross-policy visuo-tactile fusion paradigm, and TacUMI, a cost-effective data collection system.

1. TouchGuide: Inference-Time Steering

TouchGuide operates as a two-stage inference-time steering mechanism for pre-trained visuomotor policies (based on diffusion or flow-matching, such as $\pi_0.5$ ). It does not require retraining the base policy.

Stage 1 (Visual Coarse Action): During the early sampling steps of the denoising or flow-matching process, the base policy generates a coarse, visually plausible action using only visual inputs.
Stage 2 (Touch Guidance): In the later sampling steps, a task-specific Contact Physical Model (CPM) provides touch guidance. The CPM is trained via contrastive learning on limited expert demonstrations to output a feasibility score.
- Architecture: The CPM uses pre-trained DINOv2 encoders for both visual and tactile inputs, fusing them via a Transformer encoder to produce a latent observation. It computes the cosine similarity between this latent observation and the latent action embedding to generate the feasibility score.
- Steering Mechanism: The feasibility score acts as a classifier guidance signal. For diffusion policies, the gradient of the score is subtracted from the noise predictor; for flow-matching policies, it is subtracted from the velocity predictor. This steers the action distribution toward the "real distribution" that satisfies physical contact constraints.

2. TacUMI: Data Collection System

To facilitate high-quality data collection for TouchGuide, the authors developed TacUMI, a handheld gripper system.

Design Principles: It is low-cost (base cost ~$720), lightweight (540g), and provides direct tactile feedback via rigid mechanical coupling (rigid fingertips) rather than indirect linkage.
Localization: It uses Vive Trackers with Lighthouse base stations for high-precision localization, avoiding the accuracy limitations of SLAM-based systems in feature-sparse scenes.
Synchronization: It features a unified hardware-software interface (RK3576 CPU) that streams synchronized high-resolution visual and tactile signals (including force fields) at 30Hz.

Key Contributions

TouchGuide Paradigm: A novel cross-policy fusion method that steers pre-trained visuomotor policies at inference time using a task-specific CPM. It fuses modalities within the low-dimensional action space, ensuring actions adhere to contact physics without retraining the base policy.
TacUMI System: A cost-effective, high-precision data collection system that enables direct tactile feedback, addressing the trade-off between affordability and accuracy in tactile data acquisition.
Empirical Validation: Extensive experiments on five challenging tasks (Shoe Lacing, Chip Handover, Cucumber Peeling, Vase Wiping, Lock Opening) demonstrating that TouchGuide significantly outperforms state-of-the-art visuo-tactile policies and generalizes across different robots, policies (Diffusion Policy, $\pi_0.5$ ), and tactile modalities (force vs. tactile images).

Experimental Results

Performance Gains: On the Shoe Lacing task, TouchGuide (using $\pi_0.5$ ) achieved a 58.0% success rate compared to 35.9% for the base $\pi_0.5$ and 24.7% for the Policy Consensus baseline. On Chip Handover, it reached 60.0% (vs. 25.0% for base $\pi_0.5$ ).
Generalization: The method demonstrated robustness under visual domain shifts (unseen objects/scenes) and occlusions. It successfully generalized across different base policies (Diffusion Policy and $\pi_0.5$ ) and different tactile sensors (force and tactile images).
Data Quality: Comparisons showed that policies trained on TacUMI data significantly outperformed those trained on SLAM-based UMI data or VR teleoperation data, particularly in low-data regimes (e.g., 20 demonstrations for Lock Opening).
Efficiency: TouchGuide adds negligible inference time overhead while providing substantial performance improvements.

Significance and Claims

The authors claim that TouchGuide addresses the fundamental limitation of existing visuo-tactile methods by performing fusion in the action space rather than the feature space. By using a CPM to steer the policy distribution toward the real distribution of contact physics, the method produces more robust actions that satisfy physical constraints at the moment of contact, rather than merely reacting to tactile changes after they occur.

The paper emphasizes that TouchGuide allows for the integration of tactile feedback into powerful pre-trained models (like VLAs) without the computational cost of retraining the entire policy. Additionally, the introduction of TacUMI provides a scalable, accessible solution for collecting the high-fidelity tactile data necessary for training such models, bridging the gap between high-precision research and practical, affordable deployment. The work suggests that explicit contact physics steering is a critical component for advancing fine-grained robotic manipulation.