💻 computer science

ViTac-Tracing: Visual-Tactile Imitation Learning of Deformable Object Tracing

The paper proposes ViTac-Tracing, a unified visual-tactile imitation learning framework that integrates weighted local and global task losses with a low-cost teleoperation system to achieve robust 1D and 2D deformable object tracing across diverse seen and unseen objects.

Original authors: Yongqiang Zhao, Haining Luo, Yupeng Wang, Emmanouil Spyrakos Papastavridis, Yiannis Demiris, Shan Luo

Published 2026-03-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yongqiang Zhao, Haining Luo, Yupeng Wang, Emmanouil Spyrakos Papastavridis, Yiannis Demiris, Shan Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a tangled ball of yarn or a crumpled handkerchief. To fix it, you don't just grab it and hope for the best; you have to gently trace your finger along the edge, smoothing it out until it's flat and straight. This is a very hard job for a robot because these objects (like cables, ropes, or towels) are floppy, unpredictable, and have "infinite" ways they can bend.

This paper introduces a new robot brain called ViTac-Tracing that teaches robots how to do this "tracing" job perfectly, whether it's a thin wire (1D) or a flat towel (2D).

Here is how it works, broken down into simple concepts:

1. The Problem: Robots Are "Blind" and "Clumsy"

When a robot tries to trace a floppy object, it usually fails for two reasons:

The Blind Spot: The robot's camera can't see exactly where its fingers are touching the object because the fingers block the view. It's like trying to thread a needle while wearing thick gloves; you can't see the eye of the needle.
The Slip: If the robot holds the object too close to the edge of its fingers, the object slips off.

2. The Solution: Giving the Robot "Super Senses"

The researchers built a special robot hand equipped with two types of senses:

Eyes (Vision): A camera looking down at the workspace to see the big picture.
Touch (Tactile): A special "skin" on the robot's fingertips that takes high-resolution photos of what it's touching. This is like giving the robot a pair of eyes on its fingers.

To teach the robot, they didn't just program it with math. Instead, they used Imitation Learning. Think of it like a human teacher guiding a student's hand. A human operator controls the robot remotely (teleoperation), but they get multi-modal feedback:

They see what the robot sees.
They see what the robot's fingertips "feel" (via the tactile camera).
They feel vibrations in their controller if the robot arm gets into a weird, awkward position (like a human arm getting "stuck" in a joint). This helps the human teacher avoid bad moves.

3. The "Brain" Tricks: Two Special Rules

The robot learns from the human teacher's demonstrations, but the researchers added two special "rules of the road" to make the learning smarter:

Rule #1: The "Center Stage" Rule (Local Loss)
Imagine you are holding a slippery bar of soap. If you hold it near the edge, it slides off. If you hold it right in the center, it's safe.
The robot's "brain" has a special rule that says: "If the object is touching the center of my fingertip camera, give that action a high score. If it's near the edge, give it a low score." This forces the robot to constantly adjust its grip to keep the object perfectly centered, preventing it from dropping.
Rule #2: The "Finish Line" Rule (Global Loss)
Sometimes, a robot traces a cable too far and pulls it out of a socket, or stops too early.
The researchers added a second rule that acts like a progress bar. The robot learns to predict: "How much of the object have I traced so far?" This helps it know exactly when to stop, just like a runner knowing when they've crossed the finish line.

4. The Results: A Unified Master

The coolest part is that this single robot brain learned to handle both thin wires (1D) and flat towels (2D) using the same set of rules.

On objects it practiced with: It succeeded 80% of the time.
On brand new objects it never saw before: It still succeeded 65% of the time.

The Big Picture Analogy

Think of this like teaching a child to walk a tightrope.

Old methods tried to write a math textbook on physics for the child (too hard, doesn't work in the real world).
This method puts a safety net (tactile sensors) under the child and a coach (the human teacher) who vibrates a stick when the child leans too far.
The child learns to keep their balance in the center (Center Loss) and knows exactly when they have reached the end of the rope (Task Loss).

In short, ViTac-Tracing is a robot that can "feel" its way through messy, floppy objects, keeping them safe in its grip until the job is done, all by learning from a human teacher who can see and feel exactly what the robot feels.

1. Problem Statement

The paper addresses the challenge of deformable object tracing, a fundamental task in robotics where a gripper slides along a deformable object (1D linear objects like cables or 2D planar objects like towels) from a fixed end to a free end, maintaining continuous contact.

Challenges: Deformable objects possess effectively infinite Degrees of Freedom (DoF) and often appear in unstructured, crumpled configurations.
Limitations of Existing Methods:
- Model-based controllers require accurate physics modeling, which is difficult to generalize across different object types.
- Reinforcement Learning (RL) often suffers from the sim-to-real gap and requires complex reward design.
- Current Imitation Learning (IL) systems often lack tactile feedback, making it difficult to maintain stable contact due to visual occlusion by the gripper.
Goal: To develop a unified policy capable of tracing both 1D and 2D deformable objects in the real world without explicit object modeling, leveraging both visual and tactile sensing.

2. Methodology

The authors propose ViTac-Tracing, a visual-tactile imitation learning framework built on the Action Chunking Transformer (ACT) architecture.

A. Visual-Tactile Teleoperation System

To collect high-quality expert demonstrations, the authors built a budget-efficient teleoperation system:

Robot Side: An ABB YuMi dual-arm robot equipped with a top-down stereo camera (ZED 2) and a GelSight-based vision-based tactile sensor mounted on the follower gripper.
Teleoperator Side: The human operator receives multi-modal feedback:
- Real-time streaming of visual and tactile images.
- Haptic feedback: Vibration motors on the leader gripper trigger when the robot approaches kinematic singularities (calculated via the Yoshikawa Manipulability Index), preventing unstable configurations.
Data Collection: The system records robot kinematics, visual images, tactile images, and ground-truth actions (End-Effector poses) at 30 Hz.

B. Policy Learning Architecture

The policy is trained using a Transformer-based network that takes a sequence of observations (kinematics, visual, tactile) and predicts a sequence of future actions (Action Chunking). The training objective combines three specific loss functions:

Reconstruction Loss ( $L_{reconst}$ ): Standard Mean Absolute Error (MAE) to reconstruct the expert action sequence.
Local Center Loss ( $L_{center}$ ):
- Motivation: To prevent object slippage, the contact point must remain near the center of the tactile sensor.
- Mechanism: The system processes the tactile image to locate the contact point. A weight factor ( $w_t$ ) is calculated based on the distance of the contact point from the sensor center. This weight is applied to the reconstruction loss, penalizing actions that move the contact point toward the sensor edges.
Global Task Loss ( $L_{task}$ ):
- Motivation: To ensure the robot knows when to stop (e.g., reaching the end of a cable or a specific corner of a towel).
- Mechanism: An auxiliary branch in the network predicts a completion index ( $I$ ), representing the ratio of the traced length to the total object length. The loss minimizes the Mean Squared Error (MSE) between the predicted and ground-truth completion indices, regulating task progression.

Total Loss: $L = L_{center} + \lambda_{reg}L_{reg} + \lambda_{task}L_{task}$

3. Key Contributions

Unified Policy: The first method to successfully trace both 1D (linear) and 2D (planar) deformable objects using a single, unified imitation learning model.
Novel Loss Functions:
- Introduction of a Local Center Loss to enforce fine-grained contact stability via tactile feedback.
- Introduction of a Global Task Loss to enable the policy to learn task termination and progression.
Enhanced Teleoperation System: A low-cost, multi-modal teleoperation setup that integrates vision-based tactile sensing and haptic feedback (vibration) to improve the quality and consistency of demonstration data.
Proprioceptive Representation: Demonstrated that using End-Effector (EE) poses (Cartesian space) as input yields significantly better performance (80% success) compared to joint angles (70% success) for tracing tasks.

4. Experimental Results

The method was evaluated on four "seen" objects (shoelace, braided cable, face towel, microfiber cloth) and two "unseen" objects (synthetic rope, cotton napkin).

Performance on Seen Objects:
- Achieved an average success rate of 80%.
- Outperformed ablation variants (e.g., removing tactile input dropped success to 60%; removing center loss increased object dropping).
Performance on Unseen Objects:
- Achieved a success rate of 65%, demonstrating strong generalizability despite differences in visual appearance.
- Failure modes were primarily related to task termination (early stopping or over-tracing) rather than object dropping, suggesting the tactile component generalizes well.
Ablation Studies:
- Sensing: Tactile sensing is crucial for maintaining contact (reducing drops), while visual sensing is crucial for task termination (preventing over-tracing).
- Losses: The center loss significantly reduced object dropping; the task loss improved the ability to stop at the correct location.
- Multi-Object Training: Training on a unified dataset of mixed 1D and 2D objects did not degrade performance compared to training on single-object datasets, validating the unified approach.

5. Significance

This work bridges a critical gap in deformable object manipulation by moving away from object-specific modeling toward a generalizable, sensor-fusion approach.

Practicality: The use of a low-cost teleoperation system with tactile feedback makes data collection more accessible and robust for real-world deployment.
Robustness: By explicitly optimizing for contact stability (local) and task completion (global), the system handles the high-DoF dynamics of deformable objects more reliably than previous RL or model-based approaches.
Scalability: The unified model proves that a single policy can handle diverse geometries (1D and 2D), paving the way for more versatile robotic assistants in tasks like cable management, clothing handling, and assistive dressing.