DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitation

1. Problem Statement

The paper addresses the challenge of training physics-based humanoid agents to perform dexterous Human-Object Interactions (HOI) (e.g., grasping, manipulating, wearing objects) without relying on expensive, high-quality 3D motion capture (MoCap) data.

• Limitations of Current Methods: Existing approaches for physics-based HOI imitation typically require precise 3D demonstrations of both the human and the object. Acquiring such data is costly, limited in scale, and difficult to generalize to unseen objects or complex scenarios.
• Limitations of Video Generative Models: While recent video diffusion models can synthesize realistic 2D HOI videos for diverse objects and actions, these videos lack physical fidelity and are purely 2D. Directly using 2D video as a training target for 3D physics simulation is an ill-posed problem due to depth ambiguity and the difficulty of aligning 2D object trajectories with 3D physics constraints.
• Core Challenge: How to leverage the rich, diverse interaction knowledge in synthetic 2D videos to train a physically plausible 3D control policy for dexterous manipulation, specifically handling the misalignment between 2D object tracking and 3D human kinematics.

2. Methodology: DeVI Framework

DeVI (Dexterous Video Imitation) proposes a framework that uses text-conditioned synthetic videos as an HOI-aware motion planner. Instead of generating 3D data directly, it synthesizes 2D videos and extracts hybrid imitation targets to train a reinforcement learning (RL) policy.

A. Synthetic Video Generation

1. Scene Initialization: A scene is initialized with a 3D human (SMPL-X model) and a target object on a table.
2. Realistic Rendering: The SMPL-X mesh is replaced with a realistic textured human mesh (from THuman2.0) to improve the performance of downstream video generation models.
3. Video Synthesis: An image-to-video diffusion model (e.g., Wan) generates a 2D HOI video based on the rendered initial frame and a text prompt (e.g., "a person holding a cup with the left hand").

B. Hybrid Imitation Target Extraction

Since full 3D reconstruction of the object from video is unreliable, DeVI extracts a hybrid target consisting of:

1. 3D Human Reference:
   • Uses monocular 3D human mesh recovery (GVHMR) and hand pose estimators (HaMeR) to get coarse 3D human motion.
   • Visual HOI Alignment: A novel optimization step refines the human pose by minimizing projection losses (aligning joints to 2D video) and an HOI loss (enforcing contact between specific body parts and the object vertices at contact frames). This ensures the 3D human motion is physically consistent with the 2D video and the 3D object.
2. 2D Object Reference:
   • Instead of estimating 6D object poses, the method tracks 2D object vertices using a video tracker (CoTracker3).
   • These 2D trajectories serve as the target for the object's movement in the simulation.

C. Policy Learning via Hybrid Tracking Reward

The humanoid control policy ($\pi_\theta$) is trained using Proximal Policy Optimization (PPO) in a physics simulator (Isaac Gym). The reward function is a product of three components:
$$R(st, at, gt) = R_h \cdot R_o \cdot R_{contact}$$

1. Human Tracking Reward ($R_h$): Encourages the simulated human to match the refined 3D human reference (joint positions, velocities, rotations).
2. Object Tracking Reward ($R_o$): Encourages the 2D projection of the simulated object's vertices to match the tracked 2D object trajectory from the video. This implicitly guides the 3D object's position and rotation without requiring explicit 6D pose estimation.
3. Contact Reward ($R_{contact}$): Encourages the agent to establish physical contact with the object. The timing of contact is inferred from the video (when object vertices start moving) and used as a pseudo-label to modulate force and distance rewards.

3. Key Contributions

1. DeVI Framework: A novel method enabling zero-shot dexterous HOI control using only text-conditioned synthetic videos, eliminating the need for 3D MoCap data.
2. Hybrid Imitation Target: A strategy that combines 3D human references (optimized via visual alignment) with 2D object trajectories. This bypasses the difficulty of accurate 3D object reconstruction while maintaining physical plausibility.
3. Visual HOI Alignment: An optimization module that aligns the reconstructed 3D human motion with the 2D video and the 3D object, specifically resolving hand-object misalignment issues common in naive 3D lifting.
4. Hybrid Tracking Reward: A reward formulation that uses 2D object projections to guide object dynamics, proving more effective and easier to obtain than traditional 6D pose rewards for synthetic data.

4. Experimental Results

The authors evaluated DeVI on the GRAB dataset (20 diverse objects) and compared it against state-of-the-art 3D imitation baselines (PhysHOI, SkillMimic, InterMimic).

• Quantitative Performance:
  • DeVI significantly outperformed baselines in MPJPE (Mean Per Joint Position Error) for both body and hands.
  • It achieved lower Object Translation (Tobj) and Orientation (Oobj) errors compared to methods relying on 6D poses.
  • Success Rate: DeVI achieved a 50% success rate (meeting all criteria: MPJPE < 0.2m, Tobj < 0.2m, Oobj < 1.0), whereas baselines ranged from 0% to 25%.
• Ablation Studies:
  • Removing Visual HOI Alignment resulted in severe misalignment between the hand and object, leading to failure in contact.
  • Removing the 2D Object Reward caused the policy to fail in manipulating the object correctly, confirming the necessity of the 2D tracking signal.
• Generalization:
  • DeVI successfully generalized to multi-object scenes and diverse interaction types (drinking, wearing hats, picking up trash) driven solely by text prompts.
  • It demonstrated robustness in non-tabletop scenarios (e.g., pushing, pick-and-place) when trained on appropriate datasets.

5. Significance and Impact

• Scalability: By leveraging video generative models, DeVI unlocks the ability to train robots on a virtually infinite variety of objects and interactions without the bottleneck of data collection.
• Bridging 2D and 3D: The paper provides a robust solution to the "2D-to-3D" gap in robotics, showing that 2D cues (when combined with smart optimization and hybrid rewards) can effectively drive complex 3D physics-based behaviors.
• Dexterous Manipulation: Unlike previous methods that often simplify grasping or ignore hand articulation, DeVI successfully models complex, dexterous hand-object interactions (e.g., precise finger placement, tool use).
• Zero-Shot Capability: The system can interact with unseen objects by simply generating a video of the interaction, making it highly adaptable to real-world deployment where object diversity is high.

In summary, DeVI represents a paradigm shift from requiring precise 3D demonstrations to using synthetic 2D videos as motion planners, effectively solving the data scarcity problem for dexterous robotic manipulation.