💻 computer science

HRDexDB: A Paired Human-Robot Dataset for Cross-Embodiment Dexterous Grasping

The paper introduces HRDexDB, a comprehensive dataset featuring 2,100 synchronized human and robotic dexterous grasping trials across 100 diverse objects, providing high-fidelity spatiotemporal ground truth to serve as a foundational benchmark for cross-embodiment manipulation research.

Original authors: Jongbin Lim, Taeyun Ha, Mingi Choi, Jisoo Kim, Byungjun Kim, Subin Jeon, Hanbyul Joo

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jongbin Lim, Taeyun Ha, Mingi Choi, Jisoo Kim, Byungjun Kim, Subin Jeon, Hanbyul Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to pick up a delicate teacup. You could show it a video of a human doing it, but there's a problem: human hands and robot hands are built differently. A human hand has five flexible fingers that bend in unique ways, while a robot hand might have four stiff fingers or a different shape entirely. If the robot just copies the human's movements exactly, it might drop the cup or crush it because its "fingers" can't bend the same way.

This paper introduces HRDexDB, a massive new "training library" designed to solve this exact problem. Think of it as a bilingual dictionary for hands.

Here is how it works, broken down simply:

1. The "Side-by-Side" Recording Studio

Most previous datasets were like watching a movie of a human picking up a cup, or a separate movie of a robot doing the same thing, but never together. HRDexDB is different. It records both a human and a robot picking up the same object at the same time.

The Setup: Imagine a room surrounded by 23 high-definition cameras (like a security system on steroids) plus cameras worn on the robot's "head" and the human's "head."
The Action: A human picks up one of 100 different objects (from a coffee mug to a screwdriver). Immediately after, a robot teleoperated by a human (using a special glove suit) picks up that same object, trying to match the human's intent but using its own mechanical hands.
The Result: The system creates a perfect, synchronized 3D movie of both actions happening in the same space, capturing every finger movement, the object's rotation, and even the "squeeze" force the robot feels.

2. Why This is a Big Deal (The "Translator" Problem)

The paper argues that robots need more than just a video of a human; they need a translation guide.

The Analogy: Imagine a human trying to write a letter with a giant, clumsy glove. They can't write the same way they do with bare hands. They have to adapt their grip.
The Dataset's Role: HRDexDB provides the data needed to teach robots how to adapt. It shows the robot: "When the human touches the cup here with their thumb, you should touch it there with your specific finger to get the same result."

3. What They Tested (The "Exam")

The authors didn't just collect the data; they used it to test if robots could actually learn from it. They ran two main experiments:

Experiment A: The Contact Map Transfer
They took the "touch map" of where a human's fingers touched an object and tried to translate it into a "touch map" for a robot.
- The Result: When the robot used the translated map (learned from HRDexDB) instead of just copying the human's map directly, it was much more successful at picking up the object without dropping it. It learned to "speak robot" while keeping the human's "intent."
Experiment B: The "Find the Match" Game
They asked the system: "Here is a human picking up a weird-shaped tool. Can you find a robot in our database that knows how to pick up that same tool?"
- The Result: The system successfully found robot grasps that matched the human's style, even for objects the robot hadn't seen before during training.

4. The "Hard Mode" Challenge

The paper also used this dataset to test how good current computer vision software is at seeing hands and objects when they are tangled together.

The Challenge: When a hand grabs an object, it blocks the view. This is "occlusion."
The Finding: The authors tested top-tier AI programs on their dataset and found that these programs struggled much more with HRDexDB than with simpler datasets. This proves that HRDexDB is a "hard mode" benchmark that pushes AI to get better at seeing through the mess of fingers and objects.

Summary

HRDexDB is a massive, high-quality library of paired videos showing humans and robots doing the same dexterous tasks. It acts as a bridge, allowing robots to learn from human dexterity not by blindly copying, but by understanding how to translate human strategies into their own mechanical reality. The paper claims this is the first dataset of its kind to offer this level of detail (3D motion, tactile sensors, and multiple robot hands) for 100 different objects.

Technical Summary: HRDexDB

Problem Statement

Enabling robots to achieve human-level dexterity is a central goal in robotics. While human manipulation provides a natural source of demonstrations, transferring these skills to robots is hindered by the "embodiment gap." Human and robotic hands differ significantly in morphology, kinematics, and actuation, leading to distinct feasible contact patterns and grasp strategies. This gap is further complicated when transferring skills across different robotic hand embodiments, as each imposes unique physical constraints.

Existing datasets fail to adequately address this challenge. Most human-object interaction (HOI) datasets focus solely on human hands, often lacking 3D information or object tracking. Conversely, robotic datasets typically focus on robot execution, often leaving object motion partially tracked or lacking paired human demonstrations. While a few recent works attempt to collect paired human-robot data, they generally lack multi-embodiment coverage, markerless RGB observations, or tactile signals. Consequently, there is a lack of comprehensive, paired datasets that capture comparable grasping behaviors across human hands and multiple diverse robotic embodiments over shared objects.

Methodology: HRDexDB Construction

To address these limitations, the authors introduce HRDexDB, a paired cross-embodiment dexterous manipulation dataset.

Data Collection System

The dataset was captured using a synchronized multi-modal system comprising:

23 Cameras: A rig of 21 synchronized exocentric RGB cameras surrounding the workspace and 2 egocentric stereo cameras (one for robotic trials, one custom helmet for human demonstrations).
Robotic Embodiments: Four dexterous robotic hands were used: Allegro Hand V4, Allegro Hand V5 Plus, Inspire Hand RH56DFTP, and Inspire Hand RH56F1.
Human Subjects: Human demonstrations were recorded alongside robotic executions.
Teleoperation: Robotic grasps were performed via teleoperation using an Xsens inertial motion-capture suit and MANUS gloves, mapping human wrist and finger motions to the robot.
Protocol: A two-stage protocol was employed. A human subject first performed a natural grasp on a target object. A teleoperator then observed this demonstration and performed a semantically corresponding grasp with the robotic embodiment, preserving intent while allowing for embodiment-specific differences in morphology and timing.

Reconstruction Pipeline

The raw multi-view recordings were processed to generate high-fidelity 3D ground truth:

Human Hand Reconstruction: Utilizing the MANO parametric model, 2D keypoints were detected via HaMeR, triangulated, and optimized for pose parameters. Subject-specific shape was calibrated using silhouette alignment with SAM3-generated masks.
Object 6D Tracking: A model-based pipeline estimated object poses ($SE(3)$). Dense depth maps were estimated using FoundationStereo, object masks via SAM3, and 6D poses via FoundationPose. Cross-view geometric consistency was enforced by rendering object meshes into all camera views to minimize silhouette misalignment and reduce drift.
Tactile Signals: Contact force signals were recorded from tactile-enabled robotic fingertips.

Dataset Statistics

Objects: 100 diverse objects (with high-quality meshes), expanding toward 1,000.
Sequences: 2.1K grasping trials.
Frames: 24 million frames.
Modalities: Synchronized RGB (23 views), 3D hand motion (MANO parameters), robot states, object 6D poses, tactile signals, and success/failure annotations.

Key Contributions

First Paired Human-Robot Dexterous Dataset: HRDexDB is the first dataset to provide markerless, paired human and multi-robot dexterous manipulation captures over shared objects, featuring synchronized multi-view RGB, 3D annotations, and tactile signals.
Novel Capture and Reconstruction System: The authors present a dedicated hardware and software solution to overcome the challenges of synchronized 3D hand-robot-object tracking under severe occlusions, including a 21-camera rig and a unified reconstruction pipeline.
Establishment of Downstream Benchmarks: The paper establishes four specific benchmarks to validate the dataset's utility:
- Human-to-robot contact map transfer.
- Cross-embodiment grasp retrieval.
- 3D hand pose estimation under dexterous grasping.
- Object 6D pose estimation under human and robot grasping.

Experimental Results

1. Human-to-Robot Contact Map Transfer

The authors trained a model to predict robot-specific contact maps from human contact patterns.

Result: Using the transferred contact maps ("Transferred-Contact") significantly improved grasp success rates compared to directly optimizing against human contact maps ("Human-Contact").
Metrics: In real-world experiments, success rates increased from 66.7% to 73.3% for the Inspire hand and from 63.3% to 80.0% for the Allegro hand. This demonstrates that learning robot-specific contact strategies from paired data is more effective than direct imitation.

2. Cross-Embodiment Grasp Retrieval

A CLIP-style retrieval model was trained to learn a shared latent space for human and robot grasps.

Result: The model successfully retrieved paired robot grasps from human queries significantly above random chance.
Metrics: For Human $\to$ Inspire retrieval, Recall@1 was 36.36% and Recall@5 was 100%. For Human $\to$ Allegro, Recall@1 was 24.24% and Recall@5 was 72.73%.
Downstream Utility: Initializing grasp optimization (BODex) with retrieved grasps outperformed both vanilla initialization and kinematic retargeting baselines, particularly in episode-level success rates (up to 93.94% for Allegro).

3. 3D Hand Pose Estimation Benchmark

State-of-the-art hand reconstruction models (e.g., HaMeR, WiLoR) were evaluated on HRDexDB.

Result: All models exhibited higher errors on HRDexDB compared to the FreiHAND dataset, confirming that HRDexDB presents a more challenging benchmark due to severe hand-object occlusions.
Utility: Fine-tuning models with HRDexDB data improved their performance on the FreiHAND benchmark, indicating that HRDexDB provides complementary, non-redundant training signals.

4. Object 6D Pose Estimation Benchmark

Object 6D pose estimation methods were evaluated under both human and robot grasping conditions.

Result: All methods performed worse under robot grasping than human grasping, suggesting that rigid robotic links introduce additional visual ambiguities for object localization.
Adaptation: Fine-tuning the MegaPose refiner with HRDexDB robot-grasp data improved mean ADD-S by 10.2% on a held-out robot-grasp environment, demonstrating the dataset's value for adapting perception to interaction-centric settings.

Significance and Claims

The paper claims that HRDexDB serves as a foundational benchmark for cross-embodiment dexterous manipulation. By providing closely aligned captures of human dexterity and robotic execution on the same target objects, it enables the study of how grasp strategies transfer across different morphologies.

The authors emphasize that HRDexDB is not merely a collection of trajectories but a resource that supports:

Data-driven transfer: Moving beyond predefined correspondences to learn robot-specific contact strategies.
Perception evaluation: Providing a rigorous testbed for 3D hand and object pose estimation in highly occluded, interaction-rich scenarios.
Future expansion: The dataset is currently expanding toward 1,000 objects and more complex functional tasks, aiming to facilitate future research in dexterous manipulation and robot learning.

The authors modestly acknowledge limitations, specifically the heterogeneity of tactile sensors across platforms and the ongoing challenge of defining functionally equivalent trajectory correspondences across different hand morphologies.