💻 computer science

Tabero: Learning Gentle Manipulation with Closed-Loop Force Feedback from Vision, Touch, and Language

The paper introduces Tabero, a benchmark and model suite that addresses the scarcity of tactile data and the lack of closed-loop force feedback in Vision-Language-Action models by repurposing existing trajectories and employing a decoupled force-position architecture to achieve human-like gentle robotic manipulation with significantly reduced grip forces.

Original authors: Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, Renjing Xu

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiwei Wu, Rui Zhang, Xin Xiang, Tao Li, Weihua Zhang, Junjie Lai, Renjing Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to pick up a ripe strawberry without squishing it, or to hand a fragile teacup to a person without cracking the handle. This is the challenge of "gentle manipulation."

Currently, most advanced robot "brains" (called Vision-Language-Action or VLA models) are like people who have read millions of books and seen billions of photos but have never touched anything. They know what a cup looks like and what the word "cup" means, but they don't understand how hard to squeeze it. If you ask them to "pick up the cup gently," they might grab it with the same crushing force they'd use to pick up a rock, because they lack the sense of touch.

The paper introduces Tabero, a new system designed to give robots this missing sense of touch and teach them how to be gentle. Here is how it works, broken down into simple parts:

1. The Problem: Robots Are "Touch-Blind"

Existing robot models are trained on data that only includes what the robot sees and what it does. They lack tactile data (information about pressure, texture, and force).

The Analogy: Imagine trying to learn how to hold an egg by watching a video of someone else holding it, but you are wearing thick boxing gloves. You can see the egg, but you can't feel if you're squeezing too hard.
The Consequence: Without touch, robots either drop things or crush them.

2. The Solution: Tabero (The "Touch-Training" Gym)

The authors created a new system called Tabero to solve two big problems: a lack of touch data and a lack of a way to measure "gentleness."

Recycling Old Data: Instead of building expensive robots to collect new data from scratch, Tabero takes existing open-source robot movement data (like old training videos) and "replays" them in a super-advanced computer simulation.
Adding the Sense of Touch: In this simulation, they equip the robot with a virtual "skin" (a tactile sensor). As the robot moves in the simulation, it records not just what it sees, but also the pressure it feels on its fingertips.
The Result: They created a massive library of data that includes Vision (sight), Language (instructions), and Touch (pressure), all perfectly synchronized.

3. The New Robot Brain: Tabero-VTLA

Using this new data, they built a new robot model called Tabero-VTLA.

The "Decoupled" Brain: Think of this robot's brain as having two separate jobs. One part decides where to move the hand (position), and the other part decides how hard to squeeze (force).
The Feedback Loop: When the robot moves, its "skin" feels the pressure. If it feels like it's squeezing too hard, it instantly adjusts its grip, just like you would if you felt an egg starting to crack in your hand.
The Outcome: The robot learns to follow instructions like "pick up gently" by actually feeling the force and adjusting in real-time.

4. How They Measure Success: The "Gentleness Score"

Usually, we only ask robots, "Did you finish the task?" (Yes/No). Tabero introduces a new way to grade them:

Task Success: Did the robot pick up the object?
Interaction Quality: Did it break the object?
The Metrics: They measure things like "Average Grip Force" (how hard it held on) and "Maximum Transient Force" (sudden spikes of pressure that might cause damage).
The Analogy: It's the difference between a student who gets an "A" on a test by guessing, versus a student who gets an "A" by understanding the material perfectly. Tabero ensures the robot understands the physics of the task, not just the outcome.

5. The Results

When they tested this system:

The robot could follow instructions to be "gentle" or "firm."
Under "gentle" instructions, the robot reduced its grip force by over 70% compared to standard robots.
Crucially, it didn't just become weak; it still successfully picked up the objects. It learned to be dexterous, not just soft.

Summary

Tabero is a toolkit that teaches robots to "feel" their way through tasks. By simulating touch in a computer and training a new type of AI model, the researchers showed that robots can learn to handle delicate objects with human-like care, reducing the risk of damage while still getting the job done.

Note: The paper focuses entirely on simulation and data generation. It mentions they are working on a real-world system in the future, but the results presented here are from their high-fidelity simulation environment.

Technical Summary: Tabero – Learning Gentle Manipulation with Closed-Loop Force Feedback

1. Problem Statement

Current Vision-Language-Action (VLA) foundation models have demonstrated remarkable progress in robotic manipulation but predominantly rely on visual and linguistic data, lacking the tactile modality essential for human-like, gentle interaction. This limitation prevents VLA models from performing force-sensitive tasks, such as handling fragile objects without damage. The field faces two primary challenges:

Data Scarcity: There is a lack of large-scale, aligned vision-tactile-language datasets. Existing tactile data collection is hindered by the high cost and complexity of custom hardware, while simulation pipelines often fail to generate high-fidelity, synchronized tactile signals.
Evaluation Gaps: Standard evaluation protocols focus solely on task success rates (outcome-oriented), neglecting the quality of physical interaction (process-oriented). They fail to quantify whether objects were damaged or excessive forces were applied during manipulation.

2. Methodology

2.1 The Tabero Benchmark and Data Pipeline

To address data scarcity, the authors introduce Tabero, a benchmark and model suite designed to generate diverse vision-tactile-language data efficiently.

Cross-Platform Data Reutilization: Instead of collecting data from scratch, Tabero repurposes open-source robot manipulation trajectories (originally from MuJoCo or other platforms) by replaying them in a high-fidelity Isaac Sim environment.
Hardware Adaptation: The pipeline adapts trajectories to a robot equipped with a visuo-tactile gripper (integrating a simulated GelSight sensor). To handle differences in end-effector geometry and control dynamics (e.g., switching from Operational Space Control to joint control), the system adjusts the robot's base pose and employs a high-gain PD joint controller to minimize tracking errors.
Multimodal Acquisition: The system captures synchronized streams at 20 Hz, including:
- Vision: RGB-D data from wrist-mounted and third-person cameras.
- Tactile: High-resolution tactile images (320×240) and marker displacement fields generated via the Taxim framework.
- Force: Ground-truth 6D contact forces from simulated fingertips.
Force Diversity Generation: To overcome the binary nature of standard gripper commands, the pipeline modulates low-level impedance controller parameters ( $K_p$ and $K_d$ ). This allows the generation of trajectories with varying force magnitudes (e.g., "gentle" vs. "firm") while maintaining the same high-level task logic. Language instructions are augmented with adverbs (e.g., "gently," "firmly") to align semantics with the measured forces.

2.2 Tabero-VTLA Architecture

The authors propose Tabero-VTLA, a force-aware model suite built upon the Pi0 infrastructure with flow matching.

Tactile Tokenization: The model integrates tactile data through specialized tokenizers:
- Force-Field Tokenizer: Encodes spatiotemporal marker displacement fields using a Temporal Convolutional Network (TCN).
- Tactile-Image Adapter: Processes composite tactile images via the visual encoder.
- Force Tokenizer: Encodes raw 6D force vectors via an MLP.
Decoupled Force-Position Control: The policy outputs coordinated commands for both gripper pose and target forces. These are executed by a fixed hybrid low-level controller that does not learn compliance but tracks the policy's commands.
- Applied Force Control: Uses an admittance-based position correction to adjust the end-effector pose based on the difference between target and measured contact forces.
- Grip Force Control: Implements a feedback loop with feedforward gain to precisely regulate grip force, using a deadzone to handle actuator imprecision.

2.3 Evaluation Protocol

The paper introduces a multidimensional evaluation protocol that moves beyond binary success rates to quantify interaction quality:

Maximum Transient Grip Force (MG): Average of the top 5% grip force values (captures aggressive spikes).
Average Grip Force (AG): Mean grip force during contact.
Maximum Transient Applied Force (MA): Average of the top 5% applied force magnitudes.
Average Applied Force (AA): Mean applied force magnitude.

3. Key Results

3.1 Data Validation and Diversity

Fidelity: The replayed trajectories in Isaac Lab achieved success rates comparable to the original MuJoCo datasets when using identical kinematics. However, introducing the tactile-equipped gripper resulted in a measurable drop in success rates for delicate tasks, highlighting the sensitivity of contact-rich tasks to force regulation.
Force Distribution: The data generation pipeline successfully created distinct multimodal distributions for different force levels (100%, 25%, and 10% of nominal force), with clear variations in tactile images and force fields corresponding to linguistic instructions like "firmly" vs. "gently."

3.2 Model Performance

Necessity of Tactile Feedback: Ablation studies showed that removing tactile input led to complete failure in force modulation. Models without tactile feedback could not adapt grip force to linguistic instructions.
Force Supervision: Adding explicit force supervision significantly improved performance. The best-performing configuration combined rich tactile representations (marker fields) with force supervision, achieving a success rate of 0.86 under "firm" conditions and 0.52 under "gentle" conditions (Dataset B), while reducing average grip force by over 70% compared to firm grasps.
Controller Efficacy: Ablation on the low-level controller demonstrated that both feedforward and admittance terms are essential for accurate force tracking. Removing either term resulted in poor force regulation or object dropping.

3.3 Semantic Generalization

The model demonstrated the ability to generalize to unseen linguistic adverbs (e.g., "lightly," "forcefully"). While success rates decreased for out-of-domain instructions, the model produced intermediate force levels that aligned with the semantic meaning of the adverbs, indicating a degree of zero-shot compositional understanding of force-related language.

4. Significance and Claims

The paper claims that Tabero provides a practical pathway toward safer and more dexterous robotic interaction in contact-rich environments. Its primary contributions are:

Scalable Data Generation: Establishing the first pipeline to generate large-scale, synchronized vision-tactile-language datasets by repurposing existing open-source trajectories, thereby addressing the data scarcity bottleneck.
Standardized Evaluation: Introducing the first standardized protocol that explicitly quantifies interaction "gentleness" alongside task success, shifting the focus from purely outcome-based metrics to process-aware physical interaction quality.
Force-Aware Control: Demonstrating that integrating tactile feedback into VLA architectures via a decoupled force-position interface enables robots to significantly reduce interaction forces (by >70%) while maintaining high task success, a capability previously unattainable with standard VLA models.

5. Limitations

The authors acknowledge that the current framework does not jointly optimize for both task success and minimal interaction force, noting an inherent trade-off between gentleness and reliability in ultra-gentle regimes. They suggest that future work could explore reinforcement learning to balance these objectives and are developing a real-world data collection system for physical deployment.