Technical Summary: Terrain Consistent Reference-Guided RL for Humanoid Navigation

Problem Statement

Contemporary humanoid locomotion methods have achieved high agility and robustness, often leveraging human data or trajectory optimization to guide Reinforcement Learning (RL). However, a significant bottleneck remains in deploying these behaviors within larger navigation stacks for complex, unstructured environments. Existing reference-guided RL approaches face two primary limitations:

1. Lack of Planning Interface: Many methods require full-degree-of-freedom reference trajectories or live motion capture as inputs, making them incompatible with standard navigation planners that issue simple velocity commands (e.g., SE(2) velocities).
2. Absence of Environmental Conditioning: Standard reference generation pipelines do not account for terrain geometry (e.g., valid footholds, stairs, slopes). Consequently, policies trained on static references struggle to track trajectories on rough terrain, leading to instability or failure when deployed in real-world human-centric environments.

While some works address perceptive locomotion without references, they often lack the robustness of reference-guided methods. Conversely, reference-guided methods often fail to integrate with autonomy stacks or adapt to terrain. This paper addresses the intersection of perceptive, reference-guided RL and closed-loop navigation autonomy.

Methodology

The proposed method, Terrain Consistent Reference-Guided RL, synthesizes environmentally consistent reference trajectories directly inside the RL training loop. The system is designed to output a policy that accepts standard SE(2) velocity commands while utilizing depth camera data for terrain adaptation.

1. Environmentally Consistent Reference Synthesis

The core innovation is the rapid generation of reference trajectories using a Linear Inverted Pendulum (LIP) model that is modulated to match the terrain geometry.

• Base Generation: A desired SE(2) velocity command ($v_d$) is converted into desired step lengths and a Center of Mass (CoM) trajectory using the LIP model dynamics. This generates a nominal reference for the CoM, swing foot, and arms.
• Terrain Projection: Before the RL agent receives the reference, the system projects the desired footstep location onto the nearest valid foothold derived from the terrain geometry (represented as a grid of polygons).
• Trajectory Modification:
  • Footsteps: The desired step is projected to the valid foothold, and the terrain's roll and pitch at that point are incorporated.
  • Swing Foot: The swing foot trajectory is modified to connect the initial pose to the projected pose, including the terrain's orientation. The $z$-height is adjusted by adding the height of the upper convex hull of the terrain points along the swing path to ensure clearance.
  • CoM: The CoM height trajectory is adjusted based on the line joining the current stance foot to the projected step location.
• Efficiency: This synthesis runs rapidly within the training loop, allowing the policy to learn from references that are physically consistent with the immediate environment.

2. Locomotion Policy and Training

• Architecture: The policy uses a single-stage, Multi-Layer Perceptron (MLP) based architecture. It maps proprioception and a single depth camera image (26x30 resolution) directly to actions.
• Perception: A ZED Mini depth camera mounted on the robot's torso provides input. In simulation, this is modeled via raycasting with aggressive downsampling to mimic real-world noise and artifacts.
• Reward Design: The training utilizes Control Lyapunov Functions (CLF-RL). Quadratic Lyapunov functions are defined for tracking objectives (pelvis, swing foot, joints). The reward function consists of two terms:
  1. A term encouraging close tracking of the reference signal.
  2. A term incentivizing the satisfaction of the Lyapunov decrescent condition (ensuring the system moves toward the reference).
     This approach densifies the reward signal, providing feedback even when tracking is imperfect but improving.
• Asymmetric Actor-Critic: The actor receives noisy proprioception and depth data, while the critic receives noise-free observations and privileged information (ground truth references, contact states, etc.).

3. Navigation Autonomy Stack

The trained locomotion policy is integrated into a full autonomy stack via a clean SE(2) velocity interface.

• Planner: A Model Predictive Control (MPC) planner operates on a single integrator with heading model. It takes a goal pose and an environment map (generated via LiDAR and FAST-LIO2) to plan collision-free paths.
• Control Barrier Functions (CBF): The MPC incorporates discrete-time CBF constraints to ensure obstacle avoidance and robustness against tracking errors.
• Execution: The MPC outputs a trajectory of velocity commands, which are tracked by a feedback controller and passed to the RL locomotion policy.

Key Contributions

1. Environment-Conditioned Reference Synthesis: The paper introduces the first reference-guided RL pipeline for humanoids where references are modulated online to be consistent with terrain geometry (stairs, slopes, discrete footholds) using a reduced-order LIP model within the training loop.
2. Simplified Architecture: The resulting policy relies solely on an MLP with no history buffers, eliminating the need for complex distillation or recurrent networks, yet achieving performance comparable to attention-based models.
3. Closed-Loop Autonomy Integration: The method successfully integrates a perceptive, reference-guided RL policy into a standard navigation stack. It demonstrates long-horizon (>70m) autonomous navigation on the Unitree G1 through outdoor environments containing rough terrain and consecutive flights of stairs, with all sensing and computation onboard.

Results

Simulation

• Tracking Performance: Conditioning references on the environment significantly improved tracking performance compared to using unmodified (environment-agnostic) references. On stair terrain, the tracking error was reduced by a factor of two.
• Model Complexity: The simple MLP-based policy ("Ours") achieved statistically similar tracking errors to a more complex architecture with cross-attention ("Ours (Attn)") across flat, sloped, and stair terrains. However, the MLP version required significantly less VRAM (14GB vs. 60GB) and is easier to implement.

Hardware Deployment

• Platform: Unitree G1 humanoid robot equipped with a ZED Mini depth camera and a Livox MID360 LiDAR.
• Performance: The system successfully performed closed-loop autonomous navigation in outdoor human-centric environments.
  • Stairs: The robot autonomously navigated 15 consecutive stairs.
  • Long-Horizon: The robot navigated over 70 meters, traversing tree roots, rough terrain, curbs, and two sets of stairs.
  • Indoor/Outdoor Transition: The robot climbed outdoor stairs, entered a building, and climbed indoor stairs.
• Comparison: Unlike prior work that relies on blind, model-based controllers for navigation or perceptive RL methods not integrated into autonomy stacks, this work demonstrates a perception-conditioned, reference-guided policy serving as the locomotion layer for a complete autonomy stack.

Significance and Claims

The paper claims that this work bridges the gap between high-performance reference-guided locomotion and practical navigation autonomy. By synthesizing references that are consistent with the terrain during training, the method allows the RL policy to learn robust behaviors for unstructured environments without requiring complex, history-based architectures.

The significance lies in the SE(2) velocity interface, which enables the locomotion policy to be a drop-in component for standard navigation planners. This allows for the deployment of agile, perceptive humanoid locomotion in real-world scenarios (stairs, rough terrain) that were previously inaccessible to reference-guided methods due to their lack of environmental conditioning and planning compatibility. The authors position this as a step toward general humanoid navigation in human-centric environments, moving beyond flat ground or lab-built custom terrains.