Technical Summary: Perceptive Humanoid Parkour (PHP)

Problem Statement

While recent advances have enabled humanoid robots to walk stably on diverse terrain, emulating the agility, adaptability, and expressiveness of highly dynamic human movements remains a significant challenge. Specifically, executing parkour in complex environments requires:

1. Low-level Robustness: Controlling a high-dimensional action space for contact-rich maneuvers (e.g., wall scaling, obstacle jumping) within fractions of a second.
2. Perception-Driven Decision Making: Tight coupling of exteroception (vision) and control to adapt to environmental variations and unexpected disturbances.
3. Long-Horizon Skill Composition: Consolidating diverse, highly dynamic skills into a single visuomotor policy that enables seamless transitions across complex obstacle courses.

A primary bottleneck is the scarcity of highly dynamic human motion data. Capturing fast, contact-rich maneuvers typically yields sparse datasets (often only one or two demonstrations per skill), which are insufficient for learning tasks over long horizons that require rich variation in approach distances, timing, and smooth transitions between heterogeneous skills.

Methodology

The authors introduce Perceptive Humanoid Parkour (PHP), a modular framework integrating human motion priors, long-horizon skill composition, and perception-driven control. The pipeline consists of three main phases:

1. Adaptive Skill Composition via Motion Matching

To address data scarcity and the need for diverse transitions, the authors employ Motion Matching as an offline synthesis module.

• Atomic Skills: Human motion data is retargeted to the robot (Unitree G1) using OmniRetarget to create a library of atomic parkour skills (e.g., traversing, climbing).
• Composition: Instead of manually designing transitions between every skill pair, the system composes kinematic trajectories over long horizons by chaining locomotion segments with atomic skills in a Locomotion → Skill → Locomotion structure.
• Mechanism: Motion Matching performs a nearest-neighbor search in a feature space (including future trajectory, foot positions, and root velocity) to link motion fragments. This "densifies" the sparse motion library and generates diverse transitions across varying approach distances, directions, and timing.
• Randomization: Obstacle geometry and pose are randomized around the synthesized trajectories to expose the policy to variations without invalidating the underlying reference motion.

2. Teacher-Student Training Pipeline

The framework utilizes a two-stage training process to transfer skills from simulation to reality:

• Teacher Training (Privileged RL): Expert policies for individual skills are trained in simulation using Reinforcement Learning (RL) with motion-tracking rewards. These experts receive privileged observations (e.g., global pelvis position/velocity) to learn recovery behaviors and correct drift, which is critical for contact-rich tasks where small errors accumulate rapidly.
• Student Distillation (DAgger + RL): A single, deep-based student policy is distilled from multiple teachers. The authors argue that pure imitation learning (DAgger) is insufficient for highly dynamic skills as it does not explicitly favor the high torque impulses required for success (e.g., the "pull-up" phase during climbing).
  • Hybrid Objective Function: The student is trained with a combined loss function: $L = \lambda_{PPO} L_{PPO} + \lambda_{D} L_{D}$.
  • Role of RL: The PPO component provides a success-oriented signal that encourages the exploitation of high-torque actions necessary to overcome obstacles, while DAgger serves as regularization to prevent unnatural behavior.
  • Curriculum: A warm-up curriculum gradually shifts the weight from DAgger to PPO to stabilize training.

3. Zero-Shot Sim-to-Real Transfer

The student policy is trained using onboard depth sensing (simulated via Nvidia WARP) and proprioception. It receives only a discrete 2D velocity command and depth images, enabling the autonomous selection and execution of skills (step, climb, jump, roll) without explicit skill labels during inference.

Main Contributions

1. Efficient Kinematic Skill Composition: A pipeline that chains retargeted human motions via Motion Matching into diverse long-horizon trajectories, thereby overcoming the scarcity of dynamic motion data.
2. Scalable Training Framework: A teacher-student architecture that distills multiple privileged experts into a single deep-based policy, utilizing a hybrid DAgger and RL objective to handle the high torque demands of dynamic skills.
3. Successful Zero-Shot Sim-to-Real Transfer: Demonstration of highly dynamic parkour on a physical Unitree G1 robot, including scaling walls up to 1.25 m (96% of robot height) and traversing complex multi-obstacle courses with closed-loop adaptation.

Experimental Results

The framework was validated on a humanoid Unitree G1 (1.3 m tall, 29 degrees of freedom) in both simulation and the real world.

• Human Agility: The robot successfully climbed a 1.25 m high wall in 3.63 seconds, a pace comparable to a human parkour artist performing the same maneuver. It also executed a cat leap over a 0.4 m high obstacle at speeds up to 3.41 m/s and landed jumps from a height of 1.25 m.
• Long-Horizon Traversal: The robot autonomously traversed a continuous 60-second parkour course with multiple obstacles and skill transitions (step, climb, jump) without explicit multi-obstacle supervision during training.
• Adaptability: The system demonstrated closed-loop adaptation to real-time obstacle disturbances (random shifts of ~0.5 m), adjusting approach timing and maneuver selection to complete the traversal.
• Ablation Studies:
  • Motion Matching Density: High-density coverage of approach distances was critical; sparse data (e.g., only extreme distances) led to failure in intermediate scenarios.
  • RL in Distillation: Removing the RL component (DAgger only) resulted in significant performance degradation, particularly at high obstacles where the robot could not generate the necessary torque impulses for the pull-up phase.
  • Benchmarks: The proposed method significantly outperformed benchmarks using pure reward shaping (velocity tracking), uncomposed motion data, and end-to-end deep training without distillation.

Significance and Claims

The authors claim that PHP represents a state-of-the-art approach for agile, adaptive whole-body parkour in the real world. Its significance lies in:

• Bridging the Gap: The successful transfer of highly dynamic, contact-rich human skills to humanoids, an area where prior work was largely limited to less dynamic tasks like stair climbing or walking on sparse terrain.
• Data Efficiency: Demonstrating that Motion Matching can effectively synthesize long-horizon references from sparse human data, enabling the learning of complex skill chains without massive datasets.
• Robustness: Proving that a single visuomotor policy can autonomously select and switch between skills based on onboard perception, handling both static and dynamic environmental changes.

The authors note limitations, including the current lack of semantic scene understanding (reliance solely on geometric depth) and hardware constraints (limited hand strength for extreme climbing or hanging maneuvers). They suggest that future work could incorporate richer conditioning signals (e.g., language) and improved sensing to reduce perceptual ambiguity at high speeds.