💻 computer science

Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos

This paper introduces Direct Dynamic Retargeting (DDR), a novel single-stage framework that bypasses the geometric biases of traditional pipelines to directly generate high-fidelity, dynamically feasible humanoid trajectories from monocular videos, thereby improving imitation accuracy and accelerating reinforcement learning convergence.

Original authors: Constant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Constant Roux, Ludovic De Matteïs, Armand Jordana, Valentin Guillet, Nicolas Mansard, Olivier Stasse, Philippe Souères

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to dance or perform martial arts. The easiest way is to show it a video of a human doing the move. But here's the problem: humans and robots are built very differently. A human has a flexible spine and soft joints; a robot is made of rigid metal bars and motors. If you just tell the robot, "Copy the human's arm position exactly," the robot might try to twist its metal body in a way that breaks it, falls over, or simply can't do because its motors aren't strong enough.

This paper introduces a new method called Direct Dynamic Retargeting (DDR) to solve this "translation" problem. Here is how it works, broken down into simple concepts:

The Problem: The "Bad Translator"

Currently, most robots use a two-step process to learn from videos, which the authors compare to a translator who gets stuck in the middle of a sentence.

Step 1 (The Geometry Step): First, the system looks at the human video and asks, "What is the closest pose the robot can physically make to match the human's arm and leg positions?" It ignores whether the robot can actually balance in that pose. It's like asking a human to stand on one leg while holding a heavy box, but only checking if their legs are in the right shape, not if they will fall over.
Step 2 (The Physics Step): Then, a second system tries to fix the first step. It takes that "almost right" pose and tries to make it physically possible within a computer simulation.

The Flaw: The authors argue that the first step creates a "bias." Because the robot was forced into a specific shape in Step 1, Step 2 is trapped. It can't find the best way to move because it's stuck trying to fix a shape that was already wrong. It's like trying to paint a perfect circle, but you started with a square outline; no matter how hard you try to smooth the edges, it will never be a true circle.

The Solution: The "Direct Architect"

The authors' new method, DDR, skips the middleman entirely.

Instead of asking, "What is the closest robot pose to the human?" and then fixing it, DDR asks a different question: "What is the best way for the robot to move that looks like the human, but also obeys the laws of physics?"

The Analogy: Imagine you are an architect trying to build a bridge that looks like a famous suspension bridge but uses different materials (steel instead of stone).
- Old Method: You copy the stone bridge's shape exactly, then try to reinforce it with steel. It might be wobbly or require impossible supports.
- DDR Method: You look at the function of the stone bridge (how it spans the gap) and design a steel bridge from scratch that achieves the same result but is perfectly stable for steel. You don't force the steel to look like stone; you let the physics of steel guide the design while keeping the overall look.

How It Works in Practice

The system uses a "smart guesser" (a sampling-based solver) inside a physics simulator. It doesn't just calculate one path; it tries thousands of different ways the robot could move. It keeps the ones that:

Look like the human in the video.
Don't fall over.
Don't break the robot's motors.
Keep the robot's feet planted firmly on the ground (no sliding).

The Results

The team tested this on a real robot (the Unitree H1-2) and compared it to the old methods.

Fewer Falls: The old methods often generated instructions that would make the robot fall or slide its feet. DDR generated instructions that were physically safe 99% of the time.
Better Accuracy: Because DDR wasn't stuck trying to fix a "bad" starting shape, the robot could track the human's movements more closely.
Faster Learning: When they used these new instructions to train the robot using Artificial Intelligence (Reinforcement Learning), the robot learned much faster and performed better than when using the old instructions.
Real-World Success: They successfully deployed the robot to perform complex moves like a "Pistol Squat" (standing on one leg while squatting) and a "Kung Fu" pose in the real world without needing to manually tweak the code for each specific move.

Summary

In short, this paper says: Stop trying to force a robot to copy a human's shape exactly, then fix the physics later. Instead, let the robot figure out how to move dynamically from the start, using the human video only as a guide for the general look. This results in robots that are safer, more accurate, and learn faster.

Technical Summary: Direct Dynamic Retargeting for Humanoid Imitation Learning from Videos

Problem Statement

Imitation Learning (IL) offers a scalable pathway for teaching complex skills to humanoid robots by leveraging expert demonstrations. While recent advances allow for the extraction of human poses from monocular videos, translating these human motions to humanoid robots presents a fundamental challenge: morphological mismatch. Humans and robots differ significantly in kinematics (limb proportions, joint limits) and dynamics (mass distribution, inertias, actuation limits).

Current state-of-the-art approaches typically rely on a two-stage pipeline:

Geometric Retargeting (GR): Uses Inverse Kinematics (IK) to map human keypoints to robot configurations, ignoring dynamic constraints.
Indirect Dynamic Retargeting (IDR): Refines the GR output within a physics simulator to ensure dynamic feasibility.

The authors identify a critical flaw in this paradigm: the intermediate IK step in GR introduces a geometric bias. By restricting the search space to kinematically valid but dynamically infeasible trajectories, the subsequent IDR optimization is biased away from the true dynamic optimum. This results in suboptimal dynamic behaviors, tracking errors, and reduced learning efficiency for downstream reinforcement learning (RL) agents.

Methodology: Direct Dynamic Retargeting (DDR)

To overcome the limitations of intermediate geometric projections, the authors propose Direct Dynamic Retargeting (DDR), a single-stage framework that generates dynamically feasible trajectories directly from noisy video data.

Core Framework

Formulation: The problem is formulated in the task space (keypoint trajectories) rather than the configuration space. The goal is to find a control sequence $U$ that minimizes the distance between the resulting robot keypoint trajectory and the expert reference $x_{ref}$ , subject to the constraint that the trajectory must lie within the Feasibility Set $\mathcal{F}_{q_0}$ (trajectories physically attainable by the robot).
Optimization Solver: The authors employ the Cross-Entropy Method (CEM), a sampling-based Model Predictive Control (MPC) solver.
- Unlike gradient-based solvers (e.g., Crocoddyl) that require strictly predefined contact sequences, CEM's sampling-based nature allows it to intrinsically handle ambiguities in contact identification derived from noisy video data.
- The optimization minimizes an augmented distance metric comprising a spatial tracking term and a relative shape-matching term (using a Laplacian matrix) to preserve the local structural shape of the demonstration while penalizing global translation/rotation.
Input Data: The pipeline utilizes the VideoMimic framework to extract human keypoints from monocular videos via the SMPL model. The reference signal includes trajectories for the torso, shoulders, hands, and feet.

Imitation Learning Integration

Once high-fidelity, dynamically feasible reference trajectories are generated offline, they are distilled into closed-loop RL policies.

Policy Training: A Constrained Markov Decision Process (CMDP) is solved using an actor-critic architecture (PPO via the skrl library in IsaacLab).
Reward Structure: The reward function follows the DeepMimic approach, penalizing deviations in joint positions, velocities, root pose, and end-effector positions, while enforcing safety constraints (joint limits, torques, foot slip).
Goal: The resulting policies are designed to robustly track the DDR references, recovering from initialization variations and external disturbances to facilitate sim-to-real transfer.

Key Contributions

Novel Framework: Introduction of DDR, a CEM-based MPC framework that bypasses the intermediate geometric bias of GR/IDR pipelines to generate dynamically feasible humanoid trajectories directly from video demonstrations.
Quantitative Superiority: Demonstration that eliminating the intermediate geometric bias significantly improves retargeting accuracy. DDR outperforms GR and IDR baselines across diverse agile motions in terms of physical consistency and demonstration tracking.
Downstream Learning Efficiency: Experimental evidence showing that providing physically viable references to RL agents accelerates training convergence and enhances the final execution of agile and balancing behaviors.
Real-World Validation: Successful deployment on the Unitree H1-2 humanoid robot, achieving zero-shot sim-to-real transfer for complex motions (squat, kung fu, pistol squat, balancing) without task-specific manual tuning.

Experimental Results

The authors evaluated DDR against GR and IDR baselines using a dataset of five agile motions (Squat, Kung Fu, One-foot Balance, Pistol Squat, Balancing Stick).

Physical Feasibility: GR produced a high percentage of physically infeasible segments (e.g., 21.74% for Squat, 28.38% for One-foot Balance) due to ignored contact dynamics. DDR and IDR achieved near-perfect feasibility (0.00% for most tasks).
Contact Sequence Accuracy: DDR consistently yielded the lowest contact sequence error rates compared to ground truth. IDR performed worse than DDR, confirming that GR initialization introduces a kinematic bias that hinders IDR's ability to find optimal contact phases.
Success Rate: In stochastic trials, DDR achieved significantly higher success rates than IDR. For example, in the "Balancing Stick" task, IDR had a 0% success rate, while DDR achieved 66.67%. This indicates that GR-induced drifts often pull IDR solvers into unstable regions.
Tracking Accuracy: While GR showed low raw tracking error (by ignoring feasibility), DDR achieved comparable or better tracking accuracy while maintaining physical validity. IDR consistently underperformed relative to DDR, further validating the negative impact of geometric bias.
RL Convergence: Policies trained on DDR references converged faster and achieved higher final cumulative rewards compared to those trained on GR or IDR references. For instance, in the "Pistol Squat" task, DDR reached 90% of the final reward in 79.6k steps, compared to 128.4k for GR and 93.5k for IDR.

Significance and Limitations

The paper claims that DDR represents a significant step forward by addressing the "geometric bias" inherent in current retargeting pipelines. By optimizing directly in the task space with a physics-aware solver, DDR provides higher-fidelity references that translate directly into more robust and efficient imitation learning.

Modest Claims and Limitations:
The authors acknowledge that DDR is not without limitations:

The objective is defined purely in task space, which can lead to ill-conditioned solutions or slower convergence in certain scenarios.
Performance remains sensitive to task-space weighting and initialization, which currently require manual tuning.
The framework is currently limited to specific motions and requires further scaling to handle large, diverse sets of motion variables from in-the-wild web videos.

In conclusion, the work establishes that bypassing intermediate kinematic projections allows for the generation of physically viable references that are superior for both tracking accuracy and downstream RL training, validated through rigorous simulation and real-world deployment on a full-size humanoid.