💻 computer science

Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation

This paper introduces MiTaS, a multi-resolution tactile imitation learning framework that fuses data from RGB cameras, a GelSight Mini, and a high-frequency Evetac sensor via a transformer-based architecture to achieve significantly higher success rates in contact-rich robotic manipulation tasks compared to vision-only or standard visual-tactile baselines.

Original authors: Rickmer Krohn, Erik Helmut, Niklas Funk, Jan Peters, Vignesh Prasad, Georgia Chalvatzaki

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rickmer Krohn, Erik Helmut, Niklas Funk, Jan Peters, Vignesh Prasad, Georgia Chalvatzaki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to put a key into a very tight, rusty lock. If you only use your eyes, you might see the keyhole, but you won't feel the tiny bumps or the moment the key starts to slip. You'd likely jam the key and give up. Now, imagine if you had a super-powerful sense of touch that could feel not just where the key is, but also the tiny, rapid vibrations happening a thousand times a second as you wiggle it into place.

This paper introduces a robot system called MiTaS (Multi-Resolution Tactile Sensing) that does exactly this. It teaches robots to "feel" their way through tricky tasks by combining three different types of "senses," much like how humans use both their eyes and their sensitive fingertips.

The Three "Senses" of the Robot

The robot doesn't just have one camera on its hand; it has a special trio of sensors working together:

The Wide-Angle Eye (RGB Camera): This is like a standard security camera. It sees the big picture, like where the table is and where the object is sitting. It's good at seeing the scene but bad at seeing tiny details or fast movements.
The High-Definition Fingertip (GelSight Sensor): Imagine a tiny, soft, squishy camera attached to the robot's finger. When the finger touches something, this camera takes a super-clear photo of the exact shape of the object it's pressing against. It's like having a fingerprint scanner that can see the texture of a key or a gear. However, it takes photos at a normal speed, so it might miss very fast events.
The Super-Fast "Strobe" Eye (Evetac Sensor): This is the secret sauce. It's an event-based sensor that doesn't take normal photos. Instead, it acts like a high-speed strobe light that only flashes when something changes. If the key slips even a tiny bit or vibrates, this sensor screams, "Hey! Something moved!" It captures thousands of these tiny changes per second, things the other two sensors would completely miss.

How They Work Together: The "Conductor"

The problem with having three different sensors is that they speak different languages and move at different speeds. The camera is slow, the GelSight is medium, and the Evetac is lightning fast.

MiTaS acts like a conductor in an orchestra. It has a special brain (a neural network) that listens to all three instruments at once.

It takes the "slow" visual info and the "medium" texture info.
It mixes in the "fast" vibration alerts from the Evetac sensor.
It fuses them all into a single, super-smart understanding of what is happening.

Once the robot understands the situation, it uses a "flow-matching" policy. Think of this as a GPS that doesn't just tell the robot where to go, but calculates the perfect, smooth path to get there, adjusting in real-time based on what the sensors are feeling.

The Big Test: Five Tricky Tasks

The researchers tested this system on five difficult jobs that require delicate touch, such as:

Assembling gears: Fitting teeth together without them getting stuck.
Wiping a board: Pressing just hard enough to clean a line without tilting the sponge.
Screwing in a lightbulb: Aligning the threads perfectly.
Inserting a key: The classic "tight lock" challenge.
Connecting a bulb: Aligning small pins into slots.

The Results:

Vision-Only Robots: When the robot only used its eyes, it failed almost everything (only 31% success). It couldn't see through the "occlusion" (when the hand blocks the view) or feel the tiny misalignments.
Standard Touch Robots: Robots with just the "fingertip" camera (GelSight) did better (54%), but they still struggled with fast, tricky moments like the key getting stuck.
MiTaS (The Winner): By combining all three senses, the robot achieved an 80% success rate. It could feel the vibrations of a slipping key and adjust instantly, something the other robots couldn't do.

The "Cheat Code" Training Method

Here is a clever trick the researchers used. They trained the robot using all three sensors, but then they told the robot: "Okay, now go do the job, but you can't use the super-fast Evetac sensor anymore."

Surprisingly, the robot still performed better than if it had never seen the Evetac sensor at all. It was as if the robot had learned a "secret language" from the fast sensor during training, and even without it, it could "hallucinate" or guess the right moves based on what it learned. This is called co-training, and it boosted performance by over 10% in some tasks.

The Bottom Line

This paper shows that for robots to handle delicate, contact-heavy tasks (like fixing a watch or assembling electronics), they need more than just eyes. They need a mix of high-resolution touch (to see the shape) and high-speed touch (to feel the vibration and slip). By teaching robots to listen to both, MiTaS allows them to solve complex problems that were previously impossible for machines to handle reliably.

Technical Summary: Multi-Resolution Tactile Imitation Learning for Contact-Rich Robotic Manipulation (MiTaS)

Problem Statement

Robotic manipulation in contact-rich environments (e.g., insertion, assembly, wiping) often fails when relying solely on vision due to occlusions and the inability to perceive subtle contact dynamics. While vision-based tactile sensors (e.g., GelSight) provide high-resolution spatial information about contact geometry, they operate at standard frame rates (e.g., 25 Hz) and struggle to capture rapid events like impacts, incipient slip, or high-frequency vibrations. Conversely, event-based tactile sensors (e.g., Evetac) offer high temporal resolution to capture these fast dynamics but lack the dense spatial geometry provided by frame-based sensors. Current imitation learning frameworks typically rely on a single tactile modality, failing to leverage the complementary strengths of heterogeneous sensors operating at different temporal resolutions.

Methodology: MiTaS Framework

The authors propose Multi-Resolution Tactile Sensing (MiTaS), a representation framework designed to fuse heterogeneous tactile sensors (RGB vision, GelSight, and Evetac) to condition a flow-matching imitation learning policy.

1. Sensor Modalities and Input

The system integrates three distinct sensing streams mounted on a gripper:

Egocentric RGB: Captured by a wrist-mounted camera (25 Hz), providing scene context and hand-object configuration.
GelSight Mini: A vision-based tactile sensor (25 Hz) providing high-resolution spatial data on local contact geometry and deformation.
Evetac: An event-based tactile sensor (200 Hz) capturing rapid contact dynamics. To align with the policy's timestep, 16 event frames are stacked per observation step.

2. Architecture

The MiTaS architecture employs a modality-specific CNN-stem followed by transformer-based fusion:

Sensor Stems: Each modality is processed by a dedicated Convolutional Neural Network (CNN) stem to extract token representations.
- RGB: Strided 2D convolutions on stacked frames.
- GelSight: Mirrored 2D convolutions adapted for rectangular resolution.
- Evetac: A 4-layer 3D convolutional stem that jointly compresses temporal and spatial dimensions, reflecting the higher temporal density of event data.
Positional and Modality Encoding: Tokens are augmented with learned positional vectors (encoding location within the sensor grid) and modality-specific embeddings (identifying the source sensor).
Sensor Fusion: The augmented token sequences are processed through a Transformer encoder using multi-head self-attention and cross-attention mechanisms. This allows for rich multisensory feature integration without creating a perception bottleneck.

3. Policy: Flow-Matching with Diffusion-Transformer (DiT)

The fused sensory tokens condition a Flow-Matching generative model for action generation:

Action Space: The policy predicts a sequence of delta commands (translation $\Delta x, \Delta y, \Delta z$ and yaw $\Delta \psi$ ) in the robot's local frame.
Training: The model learns to regress a target velocity field along a straight-line probability path from random noise to the demonstrated action sequence.
Inference: Actions are generated by integrating the learned velocity field via explicit Euler integration.
Constraint: The policy is state-agnostic; it does not receive absolute robot state or proprioception, forcing it to infer relative motion solely from high-dimensional multimodal sensory inputs.

4. Multi-Tactile Co-Training

To enable deployment with only a subset of sensors (specifically Vision + GelSight) while leveraging the high-frequency Evetac data during training, the authors introduce an asymmetric co-training scheme:

During training, batches are constructed stochastically: 50% of sequences include Evetac data, while 50% exclude it.
This forces the visual-tactile representation to implicitly learn and compensate for the missing high-frequency features, regularizing the model to perform robustly even when the Evetac encoder is omitted during inference.

Experimental Evaluation

The framework was evaluated on a 7-DoF Franka Panda robot across five challenging contact-rich tasks:

Gear Assembly: Inserting a gear between two others.
Board Wiping: Maintaining contact pressure while wiping a line.
Lamp Installation: Aligning and screwing a bulb into a socket.
Key in Lock: Precise insertion of a key into a lock.
Lightbulb Connection: Aligning and rotating a GU10 bulb into a socket.

Key Results

Overall Performance: MiTaS achieved an average success rate of 80% across all tasks.
Baselines:
- Sparsh-X (a multimodal transformer baseline using the same sensors): Achieved 54% average success. Notably, it failed to complete the "Key in Lock" task, getting stuck in the keyway despite multiple attempts.
- Vision-Only Models (ViT/ViT-CNN): Achieved only 31% and 26% respectively, failing completely on tasks requiring tight insertions (Key, Lightbulb) due to occlusion and lack of contact feedback.
Sensor Ablation:
- Removing Evetac (Vision + GelSight) dropped performance to 51% average, confirming the value of high-frequency data.
- Removing GelSight (Vision + Evetac) resulted in 37% average, showing that spatial geometry is critical for initial alignment.
Co-Training Impact: Co-training the Vision+GelSight policy with Evetac data improved performance by >10% in specific tasks (e.g., Board Wiping +15%, Lightbulb +25%), even though Evetac was not used during evaluation.

Attention Analysis

Visualization of cross-attention scores revealed that the policy dynamically shifts reliance on sensors based on the task phase:

Reaching: Primarily attends to RGB vision.
Screwing/Insertion: Shifts focus to GelSight for geometric alignment.
Critical Events: Attends to Evetac during jamming or ambiguous states (e.g., distinguishing between hovering and fully inserted in the Lightbulb task) to resolve uncertainty.

Significance and Contributions

The paper claims the following contributions:

Novel Architecture: The first work to combine a classic frame-based tactile sensor (GelSight) with an event-based tactile sensor (Evetac) using a modality-specific CNN-stem and transformer fusion to handle different temporal resolutions.
Co-Training Scheme: A method to leverage high-frequency sensor data during training to boost the performance of standard visual-tactile policies at inference time, without requiring the high-frequency sensor during deployment.
Dataset and Benchmark: A new multi-resolution tactile dataset covering five complex manipulation tasks, demonstrating that multi-resolution fusion enables robust control in scenarios where vision-only or single-modality tactile approaches fail.

The authors emphasize that while the framework shows strong performance, it is currently limited to 4-DoF actions and relies on teleoperated data collection. Future work aims to scale to 6-DoF control and explore broader sensor integration.