Technical Summary: VIBEACT

Problem Statement

Dexterous robotic manipulation relies heavily on rapid, local contact events (such as initial touch, sticking, and sliding) that are often brief, subtle, or visually occluded. While piezoelectric microphones offer a compact, high-bandwidth, and inexpensive modality for sensing these interactions via structure-borne vibrations, they present a significant challenge for sim-to-real policy learning. The raw vibro-acoustic signals depend on complex, difficult-to-model factors including finger material, adhesive properties, mounting locations, object textures, and background motor noise. Consequently, accurately simulating these raw audio signals for end-to-end reinforcement learning (RL) is currently impractical. Conversely, learning directly from real-world vibrotactile data is hindered by the prohibitive cost and safety risks of online RL, as well as the difficulty of collecting large-scale, contact-rich teleoperated demonstrations for multi-fingered hands.

Methodology: VIBEACT Framework

VIBEACT proposes a framework that bridges real vibrotactile sensing and simulation-based control by introducing a shared, low-dimensional physical representation of contact and slip. This approach decouples the sensing problem (mapping audio to physics) from the control problem (learning policies in simulation).

1. Tactile Representation

Instead of using raw audio, VIBEACT defines a compact tactile observation vector $z_t$ for each finger $i$, consisting of three components:

• Contact Onset ($e_t^i$): A sparse, binary pulse indicating the moment a new contact is established.
• Slip Presence ($b_t^i$): A binary state indicating if the finger is currently sliding (thresholded at 5 mm/s).
• Slip Magnitude ($m_t^i$): A continuous scalar representing the severity of the slip.

This representation intentionally excludes privileged simulator information (e.g., contact location, surface normals, force vectors) to ensure it remains predictable from physical microphone signals and transferable across objects.

2. Data Collection and Digital-Clone Labeling

The system utilizes a dexterous hand (LEAP) mounted on a robot arm (xArm7) with piezoelectric microphones embedded in each fingertip.

• Real-World Collection: Teleoperated trajectories are recorded, capturing synchronized microphone audio and robot states.
• Digital Clone Replay: These recorded trajectories are replayed in a calibrated MuJoCo simulation environment. The simulator's contact solver generates ground-truth labels for contact onset, slip presence, and slip magnitude for each finger.
• Automation: This process creates a dataset of real audio paired with simulated physical labels without manual annotation.

3. Tactile Estimator

A deep learning model is trained to map real-world microphone waveforms to the physical tactile representation.

• Architecture: The estimator consists of four independent subnetworks (one per finger). Inputs are log-mel spectrograms from synchronized microphones. The network employs a learnable microphone-gating layer, convolutional encoders with frequency-only pooling, temporal convolutions, and attention pooling.
• Training: It uses a multi-task objective with class-weighted binary cross-entropy for contact onset and slip presence, and Huber loss for slip magnitude. A sequential training strategy (pretraining on fixed-object data, then fine-tuning on moving-object data) is employed to handle the domain gap between static and dynamic slip.

4. Policy Learning

Reinforcement learning policies (PPO) are trained entirely in simulation.

• Observations: The policy receives proprioception, point clouds, and the tactile representation $z_t$ (computed directly from the simulator's contact dynamics).
• Deployment: At deployment, the trained tactile estimator replaces the simulator's ground-truth channel, feeding the estimated tactile representation derived from real microphone signals into the simulation-trained policy.

Key Contributions

1. Sim-to-Real Framework: A novel framework that bridges real vibrotactile sensing and simulation-based policy learning via a shared contact-and-slip representation, avoiding the need to simulate raw audio.
2. Digital-Clone Labeling Pipeline: An annotation-free method that automatically generates per-finger contact and slip supervision by replaying real-world teleoperation data in a calibrated digital clone.
3. Empirical Validation: A study demonstrating that this physically grounded representation serves as an effective observation channel for RL, significantly improving performance on contact-rich tasks compared to proprioception and vision-only baselines.

Experimental Results

The framework was evaluated on five contact-rich tasks: Box Climb, Can Climb, Peg in Hole, Cube Rotation, and Nut Rotation.

• Tactile Estimator Performance: The full VIBEACT estimator (using sequential training with independent encoders and heads) achieved the highest accuracy, with a Contact Onset F1 of 0.597, Slip Presence F1 of 0.913, and Slip Magnitude MAE of 4.736 mm/s. Ablation studies showed that independent per-finger networks outperformed shared architectures, highlighting the distinct vibration characteristics of each fingertip.
• Simulation Performance: VIBEACT consistently outperformed a baseline using only proprioception and point clouds (Prop+PC).
  • Gains: The most significant improvements were seen in tasks requiring sustained reactive control. For example, success rates increased by +51% for Cube Rotation and +24% for Peg in Hole.
  • Channel Analysis: The slip magnitude channel was identified as the primary driver of performance gains. Adding slip magnitude alone provided the largest jump in success rates, whereas contact onset alone provided inconsistent or even negative gains in some tasks.
• Real-World Deployment: Policies trained in simulation were deployed directly on the physical robot, replacing the simulator channel with the estimator. VIBEACT improved success rates on all deployed tasks compared to the Prop+PC baseline (e.g., Can Climb improved from 11/20 to 19/20; Nut Rotation from 1/20 to 8/20).

Significance and Claims

The paper claims that VIBEACT provides a practical and effective route for incorporating high-bandwidth tactile sensing into sim-to-real policy learning without the need for accurate audio simulation. By using a physically grounded intermediate representation, the framework allows policies to exploit rapid tactile feedback (specifically continuous slip magnitude) while remaining robust to the domain gap between real sensors and simulators. The authors emphasize that this approach enables the use of reinforcement learning for dexterous manipulation in scenarios where raw audio simulation is intractable and real-world data collection is too expensive or unsafe.

Limitations

The authors acknowledge that the compact representation discards information present in raw signals, such as contact location and surface texture. The estimator is tied to a specific hardware configuration and may require recalibration if microphone placement or finger materials change. Additionally, the digital-clone labeling relies on accurate object pose tracking, which may limit applicability in highly unstructured environments. Finally, the current policy treats the tactile representation as a flat vector, leaving potential for future architectures that model the spatial and temporal structure of per-finger events.