A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning

Technical Summary: A Visuo-Tactile Data Collection System with Haptic Feedback for Coarse-to-Fine Imitation Learning

Problem Statement
Imitation learning (IL) for contact-rich manipulation requires high-quality human demonstrations that capture both physical interaction forces and the temporal structure of tasks (e.g., distinguishing between coarse approach motions and fine adjustments). Conventional data collection pipelines suffer from two primary limitations:

1. Decoupled Haptics: Kinematic interfaces (e.g., pistol grips) physically separate the operator from the gripper's end-effector. This prevents the operator from feeling contact forces, making it difficult to demonstrate the subtle force modulation necessary for fine-grained control.
2. Lack of Structural Annotation: Existing systems lack built-in mechanisms for operators to mark task-critical regions or transitions in real time. Consequently, datasets often lack the structural labels required by coarse-to-fine learning algorithms, which rely on phase or region annotations to model tasks as sequential phases.

Methodology
The authors propose a novel visuo-tactile data collection system designed to address these limitations through direct human-robot interaction and real-time annotation.

• Hardware Design:

  • Direct-Drive Gripper: The system utilizes a custom 3D-printed two-jaw gripper where the operator actuates the jaws directly with their fingers via a compact mechanical linkage. This preserves the natural mapping between finger motion and gripper width, allowing the operator to perceive contact-induced resistance and modulate force naturally.
  • Visuo-Tactile Sensing:
    • Visual: A GoPro camera with a fisheye lens is mounted above the gripper to provide an ego-centric view with a wide field of view, ensuring context is preserved even under partial self-occlusion.
    • Tactile: Custom 3D-ViTac sensors are mounted on the inner surfaces of both jaws. Each sensor consists of a piezoresistive sheet between two orthogonal sets of conductive threads, forming a dense $16 \times 16$ array. This configuration captures contact presence, normal force, and allows for the reconstruction of contact geometry, including surface shapes occluded from the camera's view.
  • Annotation Interface: A push button mounted on the handle allows the operator to label task-critical regions in real time. An Arduino microcontroller reads the button state alongside tactile and visual data, recording time-stamped signals for precise temporal alignment.

• Data Collection and Processing Pipeline:

  • Synchronization: During a demonstration, the system simultaneously records video, tactile arrays, and button states with precise timestamps.
  • Post-Processing: After data collection, the 6-DoF pose of the end-effector is estimated using the ORB-SLAM3 algorithm applied to the recorded video. Gripper width is tracked separately using ArUco markers attached to the jaws.
  • Dataset Construction: All modalities (RGB images, 6-DoF pose, gripper width, $16 \times 16$ force arrays, and binary annotation signals) are synchronized using their timestamps to create a coherent, time-indexed dataset.

• Data Representation:
  The resulting dataset consists of time-aligned sequences including visual observations, proprioceptive data, tactile readings, and critical-region labels. The system demonstrates this capability on a toy insertion task (grasping a hex wrench and inserting it into a tilted stand), where the operator labels the "coarse" phase (approach and grasp) and the "fine" phase (precision alignment and insertion).

Key Contributions

1. Haptic-Preserving Interface: A direct-drive gripper design that enables operators to perform contact-rich manipulation while retaining natural haptic feedback, facilitating the demonstration of subtle force modulation.
2. In-Situ Temporal Annotation: A real-time labeling mechanism that allows operators to explicitly mark task transitions (e.g., coarse-to-fine switches), generating datasets compatible with coarse-to-fine learning algorithms.
3. Multimodal Synchronization: A pipeline that fuses in-hand force perception (via 3D-ViTac) with in-situ temporal annotations, producing structured datasets that capture both contact geometry and task phase information.

Results and Demonstrations
The paper presents a single-demonstration summary of a toy insertion task to illustrate the system's output. The collected data includes:

• Trajectory Visualization: 3D position trajectories estimated via SLAM, highlighting critical intervals (e.g., tight clearance during insertion).
• Tactile Feedback: Visualizations of tactile imprints showing the wrench shape during grasp and approach, providing direct signals for correct insertion angles that are difficult to infer from images alone.
• Temporal Structure: Clear segmentation of the task into coarse and fine phases based on the operator's button inputs, visualized alongside gripper width and event markers (grasp, approach, insertion).

Significance and Claims
The authors claim that this system provides a valuable tool for advancing robotic manipulation research by addressing the gap in data quality for contact-rich tasks. Specifically, the system enables the development of high-quality manipulation policies by:

• Facilitating coarse-to-fine learning algorithms that exploit structural task knowledge (phase/region annotations).
• Supporting research in inverse constraint learning, where constraints are recovered from demonstrations.
• Enabling large language model-assisted imitation learning by providing rich, structured multimodal data.

The paper emphasizes that by preserving natural haptic perception and explicitly capturing temporal structure, the system overcomes the limitations of conventional decoupled interfaces, allowing for the collection of demonstrations that are both physically accurate and structurally informative.