Hoi! - A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation

1. Problem Statement

Current research in embodied AI and robotics faces a fundamental disconnect between human-centric video datasets (which capture long-horizon activities but lack physical force data) and robotics datasets (which often focus on short-horizon primitives, static scans, or simulated environments).

Key gaps identified include:

• Lack of Force Grounding: Most video datasets do not capture the physical forces (torque, pressure, interaction forces) required to manipulate objects, making it difficult to train models for physical reasoning.
• Single Viewpoint/Embodiment: Existing datasets rarely provide synchronized data from multiple viewpoints (egocentric, exocentric, wrist-mounted) or across different embodiments (human hand vs. robotic gripper) performing the same task.
• Static vs. Dynamic: Many articulation datasets rely on static 3D scans without paired motion data, failing to capture the dynamic geometry changes during interaction.

This gap hinders the ability to study how visual appearance correlates with physical forces and how human skills can be effectively transferred to robotic agents.

2. Methodology & Dataset Construction

The authors introduce Hoi!, a multimodal dataset designed to bridge these gaps by capturing human interactions with articulated furniture (drawers, doors, fridges, dishwashers) in "in-the-wild" indoor environments.

A. Data Collection Pipeline

• Scale: 3,048 interaction sequences across 381 articulated objects in 38 diverse environments (kitchens, offices, living rooms).
• Four Embodiments: Each object is manipulated in four distinct modes to enable cross-embodiment transfer research:
  1. Human Hand: Bare hand interaction.
  2. Human Hand + Wrist Cam: Hand with a wrist-mounted camera (Project Aria).
  3. UMI Gripper: A handheld Universal Manipulation Interface gripper.
  4. Hoi! Gripper: A custom-designed, open-source robotic gripper (see below).
• Multi-View Recording:
  • Egocentric: Project Aria glasses (RGB, SLAM, eye gaze, hand pose).
  • Exocentric: Two static iPhone 13 Pro cameras (RGB + LiDAR depth).
  • Wrist-mounted: ZED Mini stereo camera on the gripper.
• Ground Truth Geometry: High-resolution 3D point clouds of every environment are captured using a Leica RTC360 laser scanner before and after manipulation to provide dense 3D ground truth and scene context.

B. The Hoi! Gripper

A novel, open-source hardware component designed specifically for this dataset:

• Design: A 2-finger parallel gripper with an antipodal mechanism (inspired by ALOHA).
• Sensing:
  • Tactile: Two GelSight Digit sensors for high-resolution tactile imaging.
  • Force/Torque: A 6-DoF force-torque sensor at the wrist.
  • Actuation: Powered by a Dynamixel motor with a load cell in the handle for human-operated force modulation.
• Mobility: Runs on a backpack-mounted NVIDIA Jetson Orin Nano, allowing fully mobile data collection.

C. Data Alignment & Processing

• Temporal Alignment: A dynamic QR code displaying the Unix timestamp is flashed at 25 Hz into all camera streams. This allows post-processing alignment of all modalities (vision, force, haptics) with an accuracy of ~10–25 ms.
• Spatial Alignment: All sensor trajectories are aligned to a common world frame using visual localization against the high-resolution 3D laser scans.
• Annotations: Includes articulation types (prismatic/revolute), joint parameters, 3D object masks (generated via SAMv2), and language descriptions.

3. Key Contributions

1. First Force-Grounded Cross-View Dataset: The first large-scale dataset coupling vision, force, and tactile sensing across human and robotic embodiments for articulated manipulation.
2. Novel Hardware: The open-source Hoi! Gripper, enabling humans to demonstrate interactions with robot-grade physical sensing (force/torque/tactile).
3. Comprehensive Ground Truth: Dense 3D scene scans and precise time-aligned multi-modal streams (RGB, Depth, Force, Torque, Tactile, Pose).
4. Benchmarks: Established baselines for three critical tasks:
   • In-the-wild Articulation Estimation.
   • Tactile Force Estimation.
   • Visual Force Estimation.

4. Results & Evaluations

The authors evaluated state-of-the-art methods on the Hoi! dataset to highlight current limitations and the dataset's value.

• Articulation Estimation:

  • Methods like ArtiPoint and ArtGS (Gaussian Splatting) performed significantly worse on Hoi! compared to controlled datasets (e.g., Arti4D).
  • Reason: These methods struggle with clutter, hands occluding objects, and the lack of dense, accurate depth in monocular settings.
  • Observation: Large Vision-Language Models (GPT-5) showed surprising robustness in predicting articulation types from single images, though they lack precise kinematic parameter estimation.

• Tactile Force Estimation:

  • Evaluated the Sparsh model (self-supervised tactile representation) on predicting forces from GelSight images.
  • Result: Performance degraded significantly (RMSE ~3–4 N) compared to the model's original benchmark (millinewton accuracy).
  • Cause: The model failed to generalize to "out-of-distribution" contact geometries (complex furniture handles vs. simple indenters) and load regimes found in real-world human operation.

• Visual Force Estimation:

  • Evaluated ForceSight (text-guided force prediction).
  • Result: The model struggled with stiff or high-force mechanisms (e.g., magnetic drawers, ovens), showing high RMSE (up to 3.5 N) compared to its training data.
  • Insight: Current models lack exposure to the wide variance of physical forces required in real-world manipulation.

• Multimodal Policy Learning (Spot Robot):

  • A force predictor trained on Hoi! data was integrated into a Boston Dynamics Spot robot's controller.
  • Result: Adding the learned force prior to a standard impedance controller increased the success rate on stiff objects (e.g., from 0% to 60% on dishwashers), demonstrating the dataset's utility for training robust policies.

5. Significance & Future Impact

The Hoi! dataset represents a paradigm shift in embodied AI research by:

• Bridging the Sim-to-Real Gap: By providing real-world force data, it allows researchers to move beyond simulation and static scans to understand the physics of interaction.
• Enabling Cross-Embodiment Transfer: It is the first dataset to explicitly align human and robotic data for the same task, facilitating research into how human skills (demonstrated with force) can be retargeted to robots.
• Advancing Multimodal Perception: It highlights that vision alone is insufficient for manipulation; integrating tactile and force sensing is crucial for robust interaction with the physical world.

The authors conclude that while current methods show promise, they are heavily dependent on ideal conditions. Hoi! provides the necessary "stress test" data to develop agents that can truly understand and act upon the physical constraints of the real world.