🤖 AI

Multi-Gait Learning for Humanoid Robots Using Reinforcement Learning with Selective Adversarial Motion Prior

This paper presents a selective Adversarial Motion Prior (AMP) strategy within a unified reinforcement learning framework that enables a humanoid robot to master five distinct gaits by applying AMP to stability-critical movements while omitting it for highly dynamic actions, thereby achieving superior convergence, tracking accuracy, and sim-to-real transfer performance compared to uniform AMP approaches.

Original authors: Yuanye Wu, Keyi Wang, Linqi Ye, Boyang Xing

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuanye Wu, Keyi Wang, Linqi Ye, Boyang Xing

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to be the ultimate human imitator. You want it to do five very different things: walk normally, march like a soldier (the "goose-step"), run fast, climb stairs, and jump high.

The problem is that these actions feel very different to a robot's brain. Walking is about steady balance; running is about explosive power; jumping is about defying gravity. If you try to teach them all at once using the same strict rules, the robot gets confused. It might try to walk like it's running, or jump like it's marching.

This paper presents a clever solution: A "Selective Coach" for the robot.

Here is the breakdown of how they did it, using simple analogies:

1. The Unified "Gym" (The Framework)

Instead of building five different robots or five different brains for five different tasks, the researchers built one single robot brain and put it in a virtual gym (a simulation).

The Analogy: Think of this like a single actor who has to play five different roles in a play. They don't change their costume or their script structure; they just change their mannerisms and the directions they get from the director.
How it works: The robot uses the same sensors and the same "muscles" (joints) for all five tasks. The only thing that changes is the "target motion" it is trying to copy.

2. The "Selective Coach" (The Core Innovation)

This is the most important part of the paper. They used a technique called Adversarial Motion Prior (AMP).

What is AMP? Imagine a strict dance instructor who watches a video of a perfect human dancer. The instructor watches the robot and says, "Hey, your arm is moving weirdly! Look at the human video; do it exactly like that." This helps the robot learn smooth, natural movements quickly.
The Problem: Sometimes, this strict instructor is a bad idea. If you are teaching a robot to run or jump, it needs to stretch its legs further and move faster than a normal human might in a slow-motion video. If the strict instructor keeps saying, "No, that's too extreme! Go back to the human video!" the robot will never learn to run fast or jump high. It will stay stuck in a slow, safe shuffle.

The Solution: The researchers created a Selective Coach.

For Walking, Marching, and Stair Climbing: The coach is ON. These are steady, rhythmic tasks. The robot needs to be stable and look natural. The coach helps it avoid wobbling and falling.
For Running and Jumping: The coach is OFF. The robot is told, "Forget the video for a second. Just go as fast and as high as you can!" This gives the robot the freedom to explore extreme movements that a normal human video might not show.

3. The "Zero-Shot" Magic (Sim-to-Real)

Usually, when you train a robot in a computer simulation, it falls over the moment you put it in the real world because the floor feels different, or the motors are slightly weaker.

The Analogy: It's like practicing basketball in a video game and then trying to play in the NBA. The physics are different.
The Fix: During training, the researchers "randomized" everything. They made the robot heavier, lighter, the floor slippery, or the motors weak. They even added fake delays to the robot's thinking time.
The Result: Because the robot practiced in a "chaotic" virtual world, it became super tough. When they put it on the real robot, it didn't need any extra tuning. It just worked immediately. This is called Zero-Shot Transfer.

4. The Results

The team tested this on a real robot (a 12-jointed humanoid lower body) and it worked for all five gaits:

Walking & Marching: The robot was smooth, stable, and didn't wobble (thanks to the "Coach" being ON).
Running & Jumping: The robot was explosive and agile, reaching speeds and heights it couldn't have achieved if the "Coach" had been nagging it to stay within normal limits (thanks to the "Coach" being OFF).
Stair Climbing: It climbed up and down stairs confidently.

The Big Takeaway

The paper teaches us that one size does not fit all in robot learning.

For steady, safe tasks, you want a strict teacher to keep things organized.
For wild, dynamic tasks, you want to let the student run free and experiment.

By knowing when to be strict and when to be free, they taught a single robot to master five completely different ways of moving, all without needing to reprogram it for each new task.

1. Problem Statement

Learning diverse locomotion skills for humanoid robots within a single, unified Reinforcement Learning (RL) framework is challenging due to the conflicting requirements of different gaits.

Conflicting Dynamics: Stable, periodic gaits (e.g., walking, stair climbing) require energy efficiency and steady balance, while highly dynamic gaits (e.g., running, jumping) demand rapid force generation, large joint excursions, and explosive dynamics.
Limitations of Current Approaches:
- Separate Policies: Traditional methods often train separate policies for each task, leading to fragmented architectures and redundant engineering.
- Uniform Adversarial Motion Prior (AMP): Existing RL methods using AMP (which regularizes policies toward human motion data) apply it uniformly. However, this creates a trade-off: while AMP stabilizes periodic motions, it can over-constrain highly dynamic gaits, preventing the robot from exploring the extreme motion amplitudes necessary for running and jumping.

2. Methodology

The authors propose a unified framework that learns five distinct gaits (walking, goose-stepping, running, stair climbing, and jumping) using a consistent policy structure, observation space, and reward formulation, differentiated only by reference parameters and a Selective AMP Strategy.

A. Unified RL Formulation

MDP Structure: All gaits share the same Markov Decision Process defined by state space ( $S$ ), action space ( $A$ ), and reward function ( $r$ ).
Observation Space: A 50-dimensional vector per frame (stacked over 21 frames for temporal context) including:
- Base linear/angular velocity and commands.
- Phase encoding (sinusoidal representation of the gait cycle).
- Projected gravity, joint position errors, velocities, and previous actions.
Action Space: 12-dimensional target joint position offsets for the 12 actuated DOFs (hip, knee, ankle).
Reward Function: A unified weighted sum of four categories:
1. Joint Tracking: Penalizes deviation from sinusoidal reference trajectories (or piecewise for jumping).
2. Balance & Posture: Rewards upright orientation, base height, and velocity tracking.
3. Smoothness: Penalizes jerky actions, high joint velocities, and excessive torque.
4. Safety & Gait-Specific: Includes fall termination, knee collision penalties, and specific terms (e.g., jump height targeting, leg straightness for goose-stepping).

B. Selective Adversarial Motion Prior (AMP) Strategy

This is the core innovation. Instead of applying AMP to all tasks, the authors apply it selectively based on gait characteristics:

AMP Enabled (Stability-Critical Gaits): Applied to Walking, Goose-stepping, and Stair Climbing.
- Rationale: These gaits are periodic with moderate joint amplitudes. AMP provides a stabilizing inductive bias, accelerates convergence, suppresses erratic behavior, and ensures foot placement consistency.
- Configuration: High weight on AMP reward ( $\alpha$ ) and interpolation ( $\beta$ ).
AMP Disabled (Dynamic Gaits): Omitted for Running and Jumping.
- Rationale: These gaits require large joint excursions and explosive dynamics that often exceed the range of available human reference data. Applying AMP here would act as an undesired constraint, preventing the policy from discovering high-speed or high-jump strategies.
- Configuration: $\alpha = 0.0$ , $\beta = 1.0$ (pure task reward).

C. Training and Sim-to-Real Transfer

Algorithm: Proximal Policy Optimization (PPO) in Isaac Gym with 4096 parallel environments.
Domain Randomization: Extensive randomization is applied to bridge the sim-to-real gap, including:
- Friction, mass, inertia, center of mass, motor strength, PD gains, and joint damping.
- Action latency (5–10 timesteps) and external disturbances.
Deployment: Policies are deployed on a physical 12-DOF humanoid robot (Lite 11) via zero-shot transfer (no fine-tuning on real hardware). The onboard computer runs the policy at 50 Hz, feeding targets to a high-frequency (200–500 Hz) PD controller.

3. Key Contributions

Unified Multi-Gait Framework: A single policy architecture successfully learns five diverse gaits (walking, goose-stepping, running, jumping, stair climbing) using a consistent observation space and reward structure.
Selective AMP Strategy: The paper identifies that AMP is beneficial for periodic/stable gaits but detrimental for dynamic ones. It introduces a gait-dependent selection mechanism that optimizes convergence for stability-focused tasks while preserving agility for dynamic tasks.
Real-Robot Validation: The study provides a reproducible baseline for multi-gait learning, demonstrating successful zero-shot transfer of all five gaits to a physical humanoid robot without fine-tuning.

4. Results and Analysis

Quantitative Performance:
- Convergence: Selective AMP significantly accelerated convergence for walking-type gaits (e.g., Goose-stepping converged in ~2,500 steps with AMP vs. slower without).
- Tracking Error: AMP reduced joint tracking errors for periodic gaits (e.g., Walking error: 0.40; Goose-stepping: 0.94).
- Success Rates: High success rates were achieved across all gaits (Walking: 95%, Running: 96%, Goose-stepping: 97%, Stair Climbing: 85%, Jumping: 83%).
Qualitative Analysis:
- AMP vs. No-AMP: For Goose-stepping, AMP enforced the strict "straight-leg" kinematic style and rapid reward growth. For Jumping, removing AMP allowed the policy to utilize full actuator capacity for explosive takeoff and aerial stabilization, which would have been penalized by a human-motion prior.
- Robustness: The robot demonstrated robustness to external disturbances, start-stop transitions, and long-horizon execution on physical hardware.

5. Significance

This work addresses a critical bottleneck in humanoid robotics: the ability to switch between diverse locomotion modes within a single learning framework.

Paradigm Shift: It challenges the "one-size-fits-all" application of motion priors in RL, demonstrating that context-aware regularization (applying priors only where they help) yields superior performance.
Practical Viability: By achieving zero-shot sim-to-real transfer for complex, dynamic gaits like running and jumping, the paper moves humanoid control closer to practical deployment in unstructured human environments.
Scalability: The unified framework suggests that future humanoid systems can learn a broad repertoire of behaviors without needing to re-architect the control system for every new task.