DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos

Technical Summary: DexImit: Learning Bimanual Dexterous Manipulation from Monocular Human Videos

DexImit is a novel automated framework designed to bridge the gap between human manipulation videos and robotic dexterous manipulation. It addresses the critical challenge of data scarcity in bimanual dexterous robotics by converting monocular human videos—which are abundant and scalable—into physically plausible, high-fidelity robot training data.

---

1. The Problem: The Embodiment and Data Gap

Learning bimanual dexterous manipulation is hindered by two primary bottlenecks:

• Data Scarcity: Collecting real-world data for highly articulated robotic hands is expensive, labor-intensive, and difficult to scale compared to simple grippers.
• The Embodiment Gap: While human videos provide a wealth of manipulation knowledge, there is a massive discrepancy between human hand morphology/action spaces and robotic dexterous hands.
• Limitations of Existing Methods: Previous attempts to use human videos either suffer from visual/action discrepancies (treating the human hand as a heterogeneous embodiment) or rely on absolute depth information and strict reconstruction accuracy, which limits their ability to scale to "in-the-wild" or generated videos.

2. Methodology: The Four-Stage Pipeline

DexImit proposes a pipeline that operates without requiring additional depth or camera information, transforming arbitrary monocular videos into structured robot trajectories through four stages:

Stage I: 4D Hand-Object Reconstruction

• Depth-Free Metric Scaling: To overcome the lack of depth, the framework uses the human hand as a size prior to estimate a metric scale factor, allowing the reconstruction of trajectories in a near-metric world coordinate system.
• Reconstruction Process: It utilizes Vision-Language Models (VLMs) for task understanding, segmentation (Grounded SAM2) for isolating hands and objects, and image-to-3D generation (SAM3D) to create object meshes.
• Coordinate Alignment: It maps arbitrary camera viewpoints into a unified world frame by estimating the table plane (z-axis) and the relative hand positions (x-axis).

Stage II: Subtask Decomposition and Bimanual Scheduling

• Action-Centric Scheduling: To handle long-horizon tasks, the framework uses an "Action-Centric Scheduling Algorithm." It decomposes videos into subtasks (pregrasp, grasp, motion, release) and manages the coordination between two arms (bimanual concurrency or asynchrony) using a priority queue system.

Stage III: Structured Action Generation

• Force-Closure Grasp Synthesis: Instead of simple imitation, DexImit synthesizes grasps by solving an optimization problem that ensures physical stability (force-closure) and minimizes collisions.
• Motion Planning: It treats the hand and object as a single rigid body post-grasp, using key-frame object poses to plan smooth, physically consistent trajectories.

Stage IV: Comprehensive Data Augmentation

• To ensure zero-shot real-world deployment, the framework applies heavy augmentation: randomizing object pose/scale, camera pose, and simulating sensor noise (removing 30% of point cloud points and adding normal perturbations) to bridge the sim-to-real gap.

3. Key Contributions

• Automated Pipeline: A complete system that converts raw, monocular, or even AI-generated videos into high-quality bimanual robot data.
• Action-Centric Scheduling: A robust method for managing complex, long-horizon bimanual coordination.
• Robustness via Augmentation: A specialized augmentation strategy that enables policies trained in simulation to work on real robots without further training.
• Scalability: The ability to leverage massive datasets from the internet or video generation models (like Sora or Veo).

4. Results and Evaluation

• Data Usability: The framework demonstrates high usability across various video qualities, from "in-the-wild" internet videos to high-quality custom captures.
• Task Complexity: DexImit outperformed baselines (like RigVid and DexMan) in complex scenarios. It achieved near-perfect success in short-horizon tasks and maintained high success rates in challenging long-horizon (e.g., making a beverage), tool-using (e.g., cutting an apple), and fine-grained (e.g., stacking cups) tasks.
• Zero-Shot Deployment: Experiments on real UR5e robotic arms with XHands showed that policies trained on DexImit data successfully transferred to the real world, proving the effectiveness of the scale and visual augmentation.

5. Significance

DexImit represents a significant step toward scaling robot learning. By decoupling the learning process from the need for expensive teleoperation and instead leveraging the vast "unlabeled" knowledge contained in human videos, it provides a scalable pathway to training robots for complex, multi-step, and dexterous real-world interactions.