💻 computer science

AT-VLA: Adaptive Tactile Injection for Enhanced Feedback Reaction in Vision-Language-Action Models

The paper proposes AT-VLA, a novel framework featuring an Adaptive Tactile Injection mechanism and a Tactile Reaction Dual-Stream architecture to enhance Vision-Language-Action models' performance in contact-rich manipulation tasks by minimizing interference with pretrained representations while achieving real-time tactile responses within 0.04 seconds.

Original authors: Xiaoqi Li, Muhe Cai, Jiadong Xu, Juan Zhu, Hongwei Fan, Yan Shen, Guangrui Ren, Hao Dong

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaoqi Li, Muhe Cai, Jiadong Xu, Juan Zhu, Hongwei Fan, Yan Shen, Guangrui Ren, Hao Dong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a robot that is incredibly smart at looking at the world and understanding language, but it's a bit clumsy when it actually needs to touch things. It can see a zipper and understand the command "unzip this bag," but when it tries to pull the zipper, it might get stuck, rip the fabric, or just give up because it doesn't "feel" the resistance.

This paper introduces AT-VLA, a new way to teach these robots how to use their "sense of touch" without forgetting everything they already know about seeing and talking.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Clumsy Genius"

Think of a standard robot brain (called a VLA model) as a genius librarian. This librarian has read millions of books (pre-trained on huge datasets) and can tell you exactly where a book is on a shelf (visual grounding) or how to ask for it (language).

However, this librarian has never actually held a book before. If you ask them to gently place a fragile book on a table, they might slam it down because they don't understand the physical sensation of "softness" or "pressure."

Recent attempts to fix this just shoved a "touch sensor" into the librarian's brain. But this was like handing a blindfolded person a map while they are trying to read a book. The new information (touch) confused the librarian, and they started forgetting where the books were (losing their visual skills).

2. The Solution: The "Adaptive Touch Switch"

The authors created a system called AT-VLA that acts like a smart traffic light for the robot's brain.

The "Touch Gate" (The Traffic Light):
The robot has a special sensor that checks: "Am I touching something right now?"
- Green Light (No Touch): If the robot is just looking at a bag or reaching for it, the "Touch Gate" stays OFF. The robot relies entirely on its "Genius Librarian" brain (vision and language). This ensures it doesn't get confused and still knows exactly where to grab the object.
- Red Light (Touching): The moment the robot's fingers make contact with the zipper or the stamp, the Gate flips ON. Suddenly, the touch sensor data is allowed into the brain.
The "Adaptive Injection":
Instead of forcing the touch data into the brain all the time (which causes confusion), the system only injects it when it's actually needed. It's like only turning on the headlights when you drive into a dark tunnel, rather than leaving them on in the bright sun where they might blind you.

3. The Speed Trick: The "Slow Thinker" and "Fast Reactor"

Even with the touch sensor, robots are usually too slow to react to things happening in real-time. If a zipper jams, the robot needs to stop instantly, not wait for a slow thought process.

AT-VLA solves this with a Dual-Stream strategy, like a CEO and a Security Guard:

The Slow Stream (The CEO): This part of the brain is the "Genius Librarian." It looks at the picture and the command ("Unzip the bag"). It thinks deeply and slowly about the overall plan. It updates maybe once every few seconds.
The Fast Stream (The Security Guard): This part is purely focused on the touch sensors. It runs at lightning speed (40 times faster than the CEO). If the Security Guard feels a sudden "jerk" or "pressure" from the zipper, they immediately shout, "Stop! Adjust!" and change the robot's hand movement instantly.

The robot uses the CEO's plan for the big picture but lets the Security Guard make split-second adjustments to keep things safe and smooth.

4. The Results: Better at Touching, Still Good at Seeing

The researchers tested this on real robots doing tricky tasks like:

Unzipping a bag (without ripping it).
Stamping a piece of paper (without crushing the desk).
Wiping a curved vase (without knocking it over).

The findings were impressive:

Better at Touch: The new robot was much better at these "touchy-feely" tasks than previous robots. It didn't get stuck or break things.
Still Good at Seeing: Because the "Touch Gate" kept the touch data away when it wasn't needed, the robot didn't lose its ability to find and grab objects. It remained a "Genius Librarian."
Robustness: Even if the touch sensor broke or was turned off during a test, the robot could still do the job almost as well as before. It had learned how to touch, so it could guess what it should feel based on what it saw.

Summary

AT-VLA is like giving a robot a pair of gloves that are "smart." The gloves only activate their special sensors when the robot actually touches something. This way, the robot gets the benefit of feeling the world without getting confused or forgetting how to see it. It combines the slow, smart planning of a human with the fast, reflexive reactions of a nervous system.

Technical Summary: AT-VLA

Problem Statement

Vision-Language-Action (VLA) models have advanced robotic capabilities in diverse tasks but struggle in contact-rich manipulation scenarios requiring precise physical interactions. Current limitations include:

Disruption of Pretrained Knowledge: Introducing new modalities (like tactile signals) during downstream finetuning often disrupts the model's existing pretrained visual and linguistic representations, leading to degraded performance in basic tasks like object localization and grasping.
Latency in Feedback Loops: The inherently slow inference speed of VLA models hampers real-time responsiveness. This delay prevents effective utilization of high-frequency tactile feedback for rapid action adjustments, which is critical for safe and successful closed-loop manipulation.

Methodology: AT-VLA

The authors propose Adaptive Tactile Vision-Language-Action (AT-VLA), a framework designed to balance the preservation of pretrained capabilities with the integration of tactile feedback. The system is built upon a pretrained VLA (specifically GO-1) and introduces two core mechanisms:

1. Adaptive Tactile Injection

This mechanism dynamically determines the timing and location of tactile signal integration to minimize interference with pretrained representations.

Tactile Gate: A learnable gating network classifies whether the robot is in a "contact" or "non-contact" state based on tactile tokens.
- Non-Contact: The gate remains inactive. The model operates identically to the vanilla VLA, relying solely on vision and language to preserve pretrained visual grounding and localization capabilities.
- Contact: The gate activates, allowing tactile signals to influence action generation.
Adaptive Cross-Attention: To accommodate the gate's state without altering the model architecture or feature dimensions:
- When the gate is inactive, the state token serves as the query in the Action Expert's cross-attention module (matching the vanilla VLA).
- When the gate is active, the query source switches to the tactile token. This allows the model to condition action generation on tactile feedback only when necessary, preserving the pretrained structure during non-contact phases.

2. Tactile Reaction Dual-Stream

To address the latency issue and enable rapid responses to high-frequency tactile feedback, the sensory processing is decoupled into two asynchronous streams:

Slow Stream (Low Frequency): Processes visual and language inputs via the large Vision-Language Model (VLM). It handles high-level task understanding and perceptual reasoning, providing a latent context (Key/Value) for the action expert.
Fast Stream (High Frequency): Continuously processes tactile feedback. It operates at a higher inference rate (e.g., 3:1 ratio relative to the slow stream) to generate immediate action adjustments.
Integration: The Action Expert conditions each action prediction on the latest tactile feedback (Fast Stream query) while utilizing the most recent vision-language reasoning output (Slow Stream Key/Value) within the same action chunk horizon. This achieves a closed-loop response time of 0.04 seconds.

Key Contributions

Adaptive Tactile Injection: The first approach to balance pretrained VLA knowledge with newly introduced tactile representations, preventing the disruption of visual grounding and object localization.
Tactile Reaction Dual-Stream: A novel mechanism that decouples sensory processing frequencies, enabling accurate, real-time action adjustments during contact while maintaining the stability of high-level reasoning.
Modality-Agnostic Robustness: The model demonstrates strong generalization, maintaining performance comparable to vanilla VLAs even when tactile signals are absent during inference, despite being trained with tactile feedback.
Empirical Validation: Extensive real-world experiments show superior performance in contact-rich tasks compared to state-of-the-art (SOTA) VLA and tactile-based policies.

Experimental Results

The authors evaluated AT-VLA on a dual-arm robot (AgiBot Genie1) equipped with tactile sensors across six tasks: four contact-rich (Unzip Bag, Stamp, Wipe Vase, Unscrew Lid) and two non-contact (Pick and Place, Open Drawer).

Contact-Rich Tasks: AT-VLA significantly outperformed baselines (GO-1, $\pi_0.5$ $π_{0} .5$ , VTLA, RDP).
- In the Unzip Bag task, AT-VLA achieved a 33% success rate compared to 20% for GO-1 and 0% for $\pi_0.5$ .
- In the Stamp task, AT-VLA reached 46% overall success, compared to 13% for GO-1 and 20% for $\pi_0.5$ .
- The model successfully avoided common failure modes of baselines, such as zipper jamming or excessive force collisions.
Modality-Agnostic Evaluation: When tested without tactile input during inference, AT-VLA maintained performance comparable to the vanilla VLA on non-contact tasks and even showed slight improvements on the "Stamp" task, suggesting it implicitly learned contact dynamics from training data.
Ablation Studies:
- Direct Incorporation: Simply adding tactile tokens without gating degraded performance by 9% compared to the vanilla VLA, confirming the risk of disrupting pretrained knowledge.
- Dual-Stream: Removing the dual-stream mechanism (keeping synchronous frequencies) resulted in an 11% performance drop, highlighting the necessity of rapid tactile response.
- Tactile Formats: The framework proved robust across different tactile formats (6D force, 2D markers, visual-tactile images), with 6D force yielding the best results.

Significance

The paper claims that AT-VLA represents a significant step toward more responsive and generalizable multimodal robotic systems. By successfully integrating tactile feedback without compromising the robustness of pretrained vision-language capabilities, AT-VLA addresses a critical bottleneck in robotic manipulation: the ability to perform precise, safe, and rapid physical interactions in contact-rich environments. The framework's ability to function effectively even when tactile sensors are unavailable further enhances its practical viability for real-world deployment.