💻 computer science

Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation

This paper introduces TacThru, a see-through-skin sensor capable of simultaneous visual and robust tactile perception, and TacThru-UMI, an imitation learning framework that integrates these multimodal signals via a Transformer-based Diffusion Policy to significantly improve robotic manipulation performance in complex tasks.

Original authors: Yuyang Li, Yinghan Chen, Zihang Zhao, Puhao Li, Tengyu Liu, Siyuan Huang, Yixin Zhu

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuyang Li, Yinghan Chen, Zihang Zhao, Puhao Li, Tengyu Liu, Siyuan Huang, Yixin Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pick up a tiny, slippery grape while wearing thick, heavy oven mitts. You can feel something is there, but you can’t see exactly where it is or how it’s moving. Now, imagine you are trying to pick up a single strand of hair while wearing thin silk gloves. You can see the hair, but your sense of touch is so dull you can’t tell if you’ve actually grabbed it.

In the world of robotics, most "fingers" suffer from this exact problem: they are either good at "seeing" (using cameras) or good at "feeling" (using tactile sensors), but they struggle to do both at the exact same time, especially when the object gets in the way.

This paper introduces TacThru, a new kind of "super-skin" for robot fingers that solves this dilemma.

The Problem: The "Blindfold" Effect

Most robots use a "switch" approach. When the robot is reaching for an object, it uses its eyes. But the moment the finger touches the object, the finger itself often blocks the camera's view—like trying to read a book while pressing your palm against the page. You lose your "sight" just when you need it most to see how the object is shifting.

The Solution: The "Glass-Skin" Finger

The researchers created TacThru, which works like a high-tech contact lens for a robot. Here is how they did it:

The Transparent Skin (The Clear Window): Instead of using a solid, opaque rubber, they used a completely clear, see-through material. This allows the camera inside the finger to "see through" the skin to look at the object, even during contact. It’s like having a finger made of clear jelly.
The Keyline Markers (The Magic Dots): To "feel" pressure, they put tiny, special patterns (called keyline markers) inside the clear jelly. When the robot touches something, these patterns stretch and move. Because the skin is clear, the robot can track these moving dots perfectly, even if the background is messy. It’s like watching how a pattern on a balloon stretches when you squeeze it.
The Brain (TacThru-UMI): They didn't just build a better finger; they built a better brain. They used "Imitation Learning," which is essentially teaching the robot by showing it how a human does it. The robot's brain learns to listen to the "eyes" and the "touch" at the same time, deciding which one to trust more in the moment.

Why This Matters (The "Superpowers")

Because of this dual-sensing, the robot gained three new "superpowers":

The "Ghost" Touch: It can detect incredibly thin and soft things, like a single piece of tissue paper, which usually "confuse" traditional tactile sensors.
The "X-Ray" Vision: It can tell the difference between two bolts that look identical to a normal camera but feel slightly different, or see the color of a tiny screw that a wrist camera is too far away to notice.
The "Adaptive" Reflex: If the robot is trying to put a cap on a bottle and its view gets blocked, it doesn't panic. It automatically switches from "using its eyes" to "using its touch" to finish the job, much like how you might find a keyhole in the dark by feeling for it.

The Bottom Line

TacThru turns a robot's finger from a blunt tool into a sophisticated sensory organ. By allowing robots to see through their own skin, they can finally handle the delicate, messy, and complex tasks that humans do every day without breaking things or dropping them.

Technical Summary: TacThru: Simultaneous Tactile-Visual Perception for Learning Multimodal Robot Manipulation

1. Problem Statement

Robotic manipulation requires a seamless transition from pre-contact (approach) to post-contact (interaction) phases. Current sensing modalities face a "perception gap":

Vision: Provides global context but suffers from severe occlusions during the actual manipulation phase.
Vision-based Tactile Sensors (VBTS): Offer high-resolution contact data but are "blind" during the approach phase and provide only local, sparse signals.
Existing See-Through-Skin (STS) Sensors: While they attempt to combine both, they typically rely on mode-switching (e.g., toggling illumination or moving mechanical parts), which introduces control complexity, latency, and prevents truly simultaneous multimodal perception. Furthermore, tracking tactile markers in STS sensors is often unreliable due to noisy, unpredictable backgrounds.

2. Methodology

The authors propose two integrated components: a novel hardware sensor (TacThru) and a learning framework (TacThru-UMI).

A. TacThru Sensor Design:
To achieve simultaneous perception without mode-switching, the sensor employs three core design principles:

Transparent Elastomer & Persistent Illumination: Unlike traditional opaque VBTS, TacThru uses a fully transparent elastomer and constant LED illumination. This allows the camera to see through the "skin" to perceive object proximity and appearance while simultaneously detecting contact via light reflection changes and marker deformation.
Keyline Markers: To solve the problem of markers becoming invisible against transparent backgrounds or being confused with environmental noise, the authors introduced "keyline markers"—concentric circles (black inner, white outer). This ensures the edge remains detectable regardless of the background.
Robust Marker Tracking: A Kalman filter-based tracking algorithm is implemented to maintain marker state ( $x_t$ ) despite environmental noise. It uses a data association step to reject "false positive" blobs from the environment, achieving an efficient processing time of ~6.08 ms per frame.

B. TacThru-UMI Learning Framework:
The authors extend the Universal Manipulation Interface (UMI) and Diffusion Policy to handle multimodal streams:

Multimodal Tokenization: The system encodes wrist-camera images (via DINOv2 ViT-B), TacThru sensor images (via DINOv2 ViT-S), marker deviations (via MLP), and proprioception (via MLP) into unified tokens.
Transformer-based Diffusion Policy: A Transformer architecture learns to attend to these diverse signals dynamically, allowing the robot to switch between visual-heavy and tactile-heavy strategies based on the task context.

3. Key Contributions

Hardware Innovation: A novel STS sensor that enables truly simultaneous tactile and visual perception without the need for mechanical or illumination-based mode switching.
Robust Perception: The introduction of keyline markers and a specialized Kalman filtering strategy that enables stable, high-frequency (up to 120Hz) tactile tracking in complex environments.
Learning Integration: The TacThru-UMI framework, which demonstrates how to effectively fuse high-dimensional visual tokens with low-dimensional tactile/proprioceptive signals using modern imitation learning.

4. Experimental Results

The system was evaluated on five challenging real-world tasks (Pick-and-Place, Pulling Tissue, Bolt Sorting, Hanging Scissors, and Cap Insertion).

Superior Success Rates: TacThru-UMI achieved an average success rate of 85.5%, significantly outperforming the tactile-only baseline (66.3%) and the vision-only baseline (55.4%).
Handling Thin/Soft Objects: In the "Pull Tissue" task, where traditional tactile sensors failed to detect contact due to low pressure, TacThru succeeded by using visual feedback to monitor the tissue's position.
Visual Discrimination: In "Sort Bolt," the sensor successfully distinguished bolts of different colors and shapes that were too small for the wrist camera to resolve and too geometrically similar for pure tactile sensing.
Adaptive Behavior: In "Insert Cap," the policy demonstrated emergent intelligence: it used visual servoing when the target was visible and automatically "fell back" to tactile-based alignment when the view became occluded.

5. Significance

This work bridges the gap between high-level visual guidance and low-level tactile precision. By proving that standard, pre-trained visual encoders (like DINOv2) can be effectively used with novel tactile hardware, the authors provide a scalable pathway for integrating complex multimodal sensing into existing robot learning pipelines. This enables robots to handle a wider variety of objects—particularly those that are thin, soft, or require millimeter-level precision—making them more capable in unstructured, real-world environments.