ADP: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion

Technical Summary: Adversarial Dynamics Priors for Physically Grounded Humanoid Locomotion

Problem Statement
Humanoid locomotion control remains significantly more challenging than quadrupedal control due to high degrees of freedom, underactuated floating-base dynamics, and the necessity of maintaining balance through intermittent contacts. While learning-based control has succeeded in generating natural motions using motion datasets, existing approaches face limitations in perturbation resilience.

• Explicit Tracking: Methods like DeepMimic rely on tracking reference poses and phases. While effective for natural motion generation, they lack flexibility when external perturbations drive the robot away from the reference trajectory.
• Kinematic Priors: Adversarial Motion Priors (AMP) improve flexibility by matching kinematic motion-style distributions (joint poses, velocities, end-effector states) rather than explicit tracking. However, AMP does not directly regularize dynamics-level features (e.g., Center of Mass motion, centroidal momentum, contact forces, and contact states) required for effective balance recovery after a push. Consequently, AMP may recover visible posture while leaving under-damped rotational dynamics or failing to modulate contact forces correctly.

Methodology: Adversarial Dynamics Priors (ADP)
The authors propose ADP, a framework that replaces kinematic motion-style features with selected dynamics features as the target for adversarial regularization. The method operates without explicit motion tracking of reference poses or phases.

1. Reference Dataset Construction ($D_{dyn}$):

   • Instead of using full-body motion capture data, the authors generate a reference dataset using Single-Rigid-Body-Dynamics (SRBD) based Trajectory Optimization (TO).
   • The TO problem minimizes the deviation from commanded velocities while penalizing excessive centroidal angular momentum and contact forces, subject to SRBD consistency, contact activation, and friction cone constraints.
   • This process yields physically grounded locomotion trajectories (walking, running, turning, lateral stepping) that satisfy SRBD-level dynamics. From these, dynamics quantities (CoM motion, centroidal momentum, contact forces, contact indicators) are extracted to form the reference distribution.

2. Dynamics Feature Representation:

   • ADP defines a timestep-level dynamics feature vector $z_t$ containing: heading-frame CoM velocity, centroidal angular momentum, planar angular momentum rate, normalized foot contact forces, and binary contact indicators.
   • To handle varying commands, the discriminator input is conditioned on the velocity command and the instantaneous tracking error, aligning the reference distribution with the current task.
   • Crucially, the method uses a temporal window ($\Xi_t$) of these features rather than single timesteps. This allows the discriminator to capture temporal patterns essential for recovery, such as contact switching, force modulation, and momentum recovery.

3. Adversarial Training:

   • A discriminator is trained to distinguish between windows sampled from the SRBD-derived reference dataset ($D_{dyn}$) and windows generated by the policy rollout ($\Xi^\pi_t$).
   • The policy is optimized using Proximal Policy Optimization (PPO) with a total reward comprising:
     • $r^G_t$: Task reward (locomotion).
     • $r^D_t$: Adversarial dynamics prior reward, derived from the discriminator's output. This reward encourages the policy's induced dynamics features to remain close to the reference distribution.
     • $r^R_t$: Regularization terms.
   • The policy learns to recover from perturbations by keeping its dynamics features within the support of the reference distribution, without being forced to track specific joint poses.

Key Contributions

1. Adversarial Dynamics Priors (ADP): The introduction of a method that uses dynamics features (CoM, momentum, contact forces) for adversarial regularization instead of kinematic features, eliminating the need for reference pose or phase tracking.
2. Perturbation-Sensitive Dynamics Representation: A formulation of temporal dynamics features that exposes post-push transients (in momentum and contact forces) more directly and rapidly than joint-level kinematic features.
3. Perturbation Recovery Evaluation: Comprehensive simulation and hardware validation demonstrating that ADP improves humanoid perturbation recovery compared to Vanilla RL, AMP, and direct dynamics-feature matching baselines.

Experimental Results
The authors evaluated ADP on a 29-DoF humanoid robot (Unitree G1) in simulation and provided qualitative hardware demonstrations. The evaluation compared ADP against Vanilla RL, AMP, and a "Dynamics Reward" baseline (which uses point-wise feature matching without an adversarial discriminator).

• Perturbation Recovery: Under external velocity impulses (up to 4.5 m/s), ADP achieved an 80%-success impulse threshold ($J_{80}$) of 115.5 N·s, a 16.7% improvement over the strongest baseline (AMP, 99.0 N·s).
• Recovery Metrics: Compared to AMP, ADP reduced the direction-averaged recovery time by 47.9% (from 4.76s to 2.48s) and the velocity tracking error by 35.4% (from 1.30 m/s to 0.84 m/s).
• Representation Sensitivity: Analysis showed that the proposed dynamics representation detected perturbation-induced transients 20 ms after impact (reaching 6.0x baseline deviation), whereas the kinematic representation took 160 ms (reaching 4.3x deviation).
• Ablation Studies:
  • Removing centroidal angular momentum reduced success rate to 45.3%.
  • Removing contact indicators (binary contact state) caused the most significant degradation, dropping success rate to 30.5% and increasing recovery time to 8.82s, highlighting the critical role of contact timing in alignment.
  • A temporal window length of $K=8$ was found to be optimal, balancing the need for temporal context (contact switching) against training stability.

Significance and Claims
The paper claims that ADP successfully addresses the limitation of kinematic priors in perturbation recovery by regularizing the policy at the dynamics level. By leveraging SRBD-derived trajectories as a reference distribution for dynamics features, the method encourages the policy to learn recovery behaviors that are physically grounded (consistent with momentum and contact constraints) without the rigidity of explicit motion tracking.

The authors note that while ADP improves robustness, it does not explicitly enforce joint-level motion naturalness. They suggest future work could combine dynamics and kinematic priors to achieve both robustness and naturalness. Additionally, the paper acknowledges that the comparison with AMP used the same TO-derived reference source for fairness, rather than a high-quality motion-capture-based AMP baseline, leaving the characterization of the gap between TO-derived and mocap-based priors for future investigation.