💻 computer science

XRoboToolKit-T: Teleoperation with High Stability and Precision with Tactile Sensing for Contact-rich Manipulation

This paper presents XRoboToolKit-T, a versatile teleoperation system that integrates tactile-informed force control with haptic assistance modules to achieve high-frequency, stable, and precise data collection for challenging contact-rich manipulation tasks.

Original authors: Xiwen Dengxiong, Xueting Wang, Ke Jing, Rui Li, Yunbo Zhang

Published 2026-09-16

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Xiwen Dengxiong, Xueting Wang, Ke Jing, Rui Li, Yunbo Zhang

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots are becoming increasingly capable of moving through the world, but they still struggle with the subtle, physical interactions that humans perform without a second thought. While a machine can easily navigate a room or pick up a rigid box, it often falters when faced with tasks that require feeling its way through a situation. This is particularly true for "contact-rich" manipulation, where a robot must handle soft, squishy, or fragile objects like a rubber bulb or a medical needle. In these scenarios, sight alone is not enough. A camera can see that a needle is touching a surface, but it cannot feel the tiny amount of pressure that tells a human hand to stop pushing or to shift slightly to avoid bending the metal. To teach robots these delicate skills, scientists need to collect vast amounts of data from human experts performing the tasks. However, current methods for recording these human movements often fail to capture the necessary force and pressure details, leading to clumsy robot attempts that break the objects or stop working entirely.

A team of researchers has developed a new system designed to bridge this gap, allowing humans to guide robots with a level of stability and precision that was previously difficult to achieve. The system, named XRoboToolKit-T, acts as a sophisticated teleoperation tool. Teleoperation is simply the act of controlling a robot from a distance, but this new version adds a crucial layer of "tactile" awareness. Instead of relying only on the operator's hand movements and what the robot's cameras see, the system listens to the robot's sense of touch. It uses special sensors on the robot's fingers to measure the pressure and force applied to an object in real time. When the robot encounters a situation where the force becomes too high or the contact becomes unstable, the system automatically steps in to help. It does this in two ways: first, by gently resisting the operator's hand if they push too hard or move too quickly, preventing the robot from making a mistake; and second, by using an advanced computer model to suggest tiny, corrective adjustments to the robot's position based on the pressure data and the specific task at hand.

The researchers tested this system on some of the most difficult tasks for a robot to perform. In one experiment, an operator had to guide a robot arm to pick up a medical syringe, remove its protective cap, and carefully insert the needle into a soft training pad that mimics human skin. The needle is incredibly thin, and the pad is soft, meaning that even a slight error in force or angle would cause the needle to bend or the robot to trigger an emergency stop. Without the new tactile assistance, the robot struggled, often bending the needle and failing to extract the liquid inside. With the system active, the robot was able to make micro-adjustments, keeping the needle steady and successfully completing the task. In another test, the robot had to grasp a deformable rubber pipette to transfer liquid. Because the rubber bulb squishes easily, applying too much force would crush it, while too little would cause it to slip. The system helped the operator find the perfect balance, allowing the robot to hold the object securely without damaging it.

The results of these tests showed a clear improvement in both the quality of the data collected and the speed of the work. When the researchers compared their new system to older methods that did not use tactile feedback, they found that the new system allowed for more successful attempts in the same amount of time. In a fifteen-minute window, the system with tactile assistance helped the robot complete 138 successful grasps of paper cups, compared to 128 with the older system. The system also proved to be much faster at updating the robot's movements. While older systems might update the robot's position about 20 times a second, the new system could do so nearly 100 times a second when using a controller, making the robot's movements feel much smoother and more responsive. This high speed is vital because it allows the robot to react to changes in pressure almost instantly, rather than lagging behind the operator's intentions.

The core of this success lies in how the system processes the information from the robot's touch sensors. It does not just show the operator a picture of the pressure; it actively uses that data to control the robot. One part of the system acts as a stabilizer, constantly analyzing the pressure distribution on the robot's fingers. If it detects that the force is becoming uneven or that the object is about to slip, it automatically dampens the operator's commands to prevent a sudden, damaging movement. Another part of the system acts as a refiner. It uses a type of artificial intelligence that understands both the visual pressure map and the written description of the task, such as "insert the syringe." Based on this understanding, it predicts the best small movement to make the task more precise and suggests that adjustment to the robot. By combining these two layers of assistance, the system creates a partnership where the human provides the intent and the machine provides the steady, sensitive execution.

This work suggests that the future of robot learning may depend less on perfecting the robot's vision and more on giving it a reliable sense of touch. By making it easier for humans to teach robots through physical interaction, rather than just visual observation, researchers can gather higher-quality data to train robots for the complex, real-world jobs that require a gentle hand. The system was tested on specific tasks like handling medical tools and fragile household items, and while it is not yet a universal solution for every robot in every factory, it demonstrates a significant step forward. It shows that by integrating touch directly into the control loop, we can help robots navigate the messy, unpredictable physical world with a confidence that was previously out of reach.

Technical Summary: XRoboToolKit-T

Problem Statement

Collecting high-quality robot data for contact-rich manipulation tasks—particularly those involving flexible or deformable objects—is essential for enabling robots to acquire real-world skills. However, existing teleoperation solutions face significant limitations. Current mainstream methods often lack the capability to obtain stable, high-frequency tactile feedback. Without this feedback, human operators struggle to control contact forces precisely; aggressive operations or unintentional hand movements can lead to excessive force application, task failure (e.g., breaking fragile objects), or triggering the robot's emergency stop. Furthermore, simply visualizing tactile signals is often insufficient for operators to make rapid adjustments, and existing haptic feedback mechanisms (such as data gloves) may be too slow to prevent manipulation failures or lack the precision required for fine adjustments. The core challenge is designing a teleoperation system that utilizes tactile information not only for operator feedback but also for active, early-stage control of the manipulation to ensure stability and precision.

Methodology

The authors propose XRoboToolKit-T (XRT-T), a versatile teleoperation system built upon PICO's XRoboToolkit, designed to provide tactile-driven assistance for stable and precise force control. The system integrates high-resolution tactile sensing with a dual-module control architecture:

1. System Architecture and Perception

Hardware Integration: The system supports both parallel two-finger grippers (augmented with third-party tactile sensors) and dexterous hands (with integrated tactile arrays). It utilizes a generic sensor reading module to abstract vendor-specific protocols, converting raw measurements into a unified pressure grid format ( $x \times y$ ).
Data Synchronization: Tactile signals are synchronized with robot kinematics and timestamped to maintain temporal consistency across perception channels. The system publishes pressure distributions, force details, and derived contact features to XR devices or PC services.

2. Tactile-Driven Control Modules

The core innovation lies in two complementary modules that process tactile data to assist the operator:

Contact Force Stabilizer (Real-time):
- Function: This module analyzes the temporal variation of the normal force distribution captured by the tactile array. It estimates a pseudo shear force by calculating the displacement of the maximum-pressure location between consecutive frames.
- Mechanism: It computes a stabilizing control factor ( $I_f$ ) inversely related to the estimated shear magnitude. If the shear force exceeds a threshold or the contact direction changes abruptly (indicating instability), $I_f$ approaches zero, effectively suppressing the operator's teleoperation commands to prevent slip or damage. If forces are stable, $I_f$ approaches one, allowing normal control.
- Frequency: Operates at approximately 20 Hz.
VLA-Based Action Refiner (High-level):
- Function: This module utilizes a Vision-Language-Action (VLA) model to perform high-level corrective adjustments.
- Inputs: Instead of standard RGB images, the model takes tactile images (spatial pressure distributions) as visual observations and combines them with task-level language instructions (e.g., "Insert the medical syringe...").
- Output: The model predicts a corrective action vector ( $\delta_{refine}$ ), representing small Cartesian displacements to refine the end-effector's pose.
- Frequency: Operates at a lower frequency of approximately 7 Hz, updating only when new refinement signals are available.

3. Control Fusion

The final robot command ( $U_{final}$ ) is a fusion of the operator's raw input ( $U_{current}$ ), the stabilizer's modulation, and the refiner's correction:
$U_{final} = U_{current} + I_f \cdot \delta_{teleop} + I_s(t) \cdot \delta_{refine}$
Where $I_s(t)$ is a synchronization indicator ensuring the refinement is applied only upon update.

Key Contributions

XRoboToolKit-T System: A high-frequency, tactile-enabled teleoperation system adaptable to both gripper-based and dexterous hand-based manipulation for contact-rich tasks involving deformable objects.
Dual-Module Assistance: The proposal of a contact force stabilizer (for real-time stability via pseudo shear force estimation) and a VLA-based action refiner (for precision via tactile-image reasoning), which collectively assist operators in achieving stable and precise force control.
Experimental Validation: Comprehensive experiments on challenging tasks, including grasping deformable rubber pipettes for liquid transfer, inserting medical syringes into vascular training pads, and grasping water bottles/paper cups, verifying the system's effectiveness and efficiency.

Results

The authors evaluated XRT-T against state-of-the-art baselines (specifically the original XRT and Twist2) across several metrics:

Manipulation Efficiency: In a 15-minute paper cup grasping task, XRT-T with a two-finger gripper achieved 138 successful manipulations, outperforming the non-tactile baseline Twist2 (128) and XRT-T with a dexterous hand (132). This indicates improved data collection efficiency.
Control Frequency: The system demonstrated significant improvements in control update rates. While the original XRT with hand tracking operated at ~20.2 Hz, XRT-T with hand tracking reached 48.6 Hz. When tactile sensing was enabled, the control frequency increased to 96.4 Hz, attributed to the tactile module providing direct contact feedback independent of the vision pipeline.
Task Success (Qualitative): In the medical syringe insertion task, the baseline XRT system failed due to needle bending and emergency stops caused by uncontrolled contact forces. In contrast, XRT-T successfully completed the task by performing micro-adjustments and maintaining stable contact, successfully extracting liquid.
Ablation Study: Combining both the stabilizer and refiner yielded the highest performance (9.6 manipulations per minute), outperforming configurations with only the refiner (9.0) or only the stabilizer (8.8). The combination allowed the system to dynamically switch between cautious contact phases and rapid transfer phases.

Significance

The paper claims that XRoboToolKit-T addresses a critical gap in robotic data collection by enabling stable and precise force control in contact-rich scenarios where visual information alone is insufficient. By integrating tactile sensing directly into the control loop through both real-time stabilization and high-level refinement, the system not only prevents manipulation failures (such as emergency stops or object damage) but also significantly improves the efficiency of data collection. The authors posit that this approach is essential for training robust robot policies for complex, real-world interactions involving deformable objects and delicate physical contact, paving the way for more reliable teleoperation in applications like medical training and assembly.