Technical Summary: Mind Your Steps: A General Learning Framework for Accurate Humanoid Foothold Tracking

Problem Statement

Enabling humanoid robots to navigate complex, dynamic environments remains a critical challenge, primarily limited by the ability to place feet safely and accurately. While reinforcement learning (RL) with velocity-commanded policies has achieved robustness in locomotion, it lacks explicit control over foothold placement. This limitation often leads to unsafe behaviors (e.g., stepping on obstacles or humans) and imprecise navigation, which hinders subsequent manipulation tasks.

Conversely, explicit foothold-tracking policies offer a promising alternative by commanding target foot poses directly. However, existing approaches suffer from significant limitations:

• Unrealistic Assumptions: Many rely on precise base state estimation (localization) or binary ground contact flags, which are difficult to estimate accurately in the real world.
• Lack of Generality: Most are confined to simulation, restricted to simple forward walking on flat terrain, or embedded as rigid components within specific, monolithic pipelines, preventing seamless integration with diverse downstream tasks.
• Fragility: Methods using root-frame targets are highly sensitive to localization errors, while those relying on complex perception pipelines often fail to generalize.

Methodology

The authors propose a novel, lightweight, model-free framework for training general-purpose 3D foothold-tracking policies. The core innovation lies in a training and deployment architecture that decouples the low-level controller from high-level planners and specific terrain assumptions.

1. Goal and Observation Formulation

The policy is designed to be agnostic to the high-level planner. Key design choices include:

• Stance-Foot Frame Representation: Unlike prior works that express targets in the robot's base (root) frame, this framework expresses foothold targets (position and yaw orientation) relative to the current stance foot.
  • Benefit: The target remains constant throughout the swing phase, eliminating the need for precise, real-time base state estimation to update targets.
• Input Observations: The policy relies solely on proprioception (base orientation, angular velocity, joint positions/velocities), the previous action, the gait phase, and the foothold goal.
• Exclusion of Contact Flags: The framework avoids using binary ground contact flags, which are unreliable on hardware, instead relying on gait phase variables to manage swing/stance transitions.

2. Training Architecture

• Algorithm: The authors employ Asymmetric Proximal Policy Optimization (PPO).
• Agent Structure: An actor-critic architecture where the actor receives proprioceptive observations, and the critic has access to privileged simulation states (e.g., linear velocity of the base).
• Goal Sampler (GS): A modular component used during training to generate feasible 3D foothold targets.
  • It samples movement directions, step lengths, and vertical offsets relative to the current stance foot.
  • It utilizes a "hold-still" mode to enable standing capabilities.
  • Crucially, the GS operates only in simulation; during deployment, it is replaced by task-specific planners (e.g., vision-based estimators or teleoperation interfaces).
• Curriculum Learning: Training begins on flat terrain to learn a robust planar tracker, followed by fine-tuning on full 3D terrains with moving pillars that adapt to sampled heights.

3. Reward Design

The reward function balances tracking accuracy with natural locomotion style:

• Foothold-Tracking Terms: Explicitly penalize errors in 3D position (planar and height) and yaw orientation for the swinging foot. The stance reward retains the accuracy achieved at the moment of touchdown.
• Feet Swing Term: Encourages foot clearance during the specific swing phase window defined by the gait clock.
• Swing Knee Height Term: Encourages lifting the knee of the swing leg to ensure sufficient clearance for the target height, preventing collisions with steps or obstacles.
• Locomotion Style: Penalization terms (adapted from LocoMuJoCo) enforce natural movement without relying on motion capture data.

Key Contributions

1. General-Purpose Framework: A model-free framework for training 3D foothold-tracking policies that are agnostic to high-level planners and ready for immediate deployment.
2. Robust Target Representation: The introduction of a stance-foot frame formulation that removes the dependency on precise base localization, significantly enhancing robustness against state estimation noise.
3. Modularity: The policy acts as a standalone low-level controller that can be seamlessly paired with various high-level generators (e.g., vision-based planners, teleoperation).
4. Zero-Shot Sim-to-Real Transfer: Successful deployment on the Booster T1 humanoid hardware using a simple vision-based foothold generator, demonstrating the framework's viability without external localization systems.

Experimental Results

The framework was evaluated extensively in simulation and on real-world hardware (Booster T1).

Simulation Experiments

• Goal Reaching: The foothold-tracking (FT) policy reached targets significantly faster and more accurately than a state-of-the-art velocity-tracking (VT) baseline. As the success threshold tightened, the VT policy's performance degraded sharply (18.9% increase in time), whereas the FT policy remained stable (5.3% increase).
• Frame Ablation: When noise was injected into base state estimates, the root-frame policy (FT-R) performance degraded sharply, while the stance-foot frame policy (FT) remained unaffected, validating the robustness of the proposed formulation.
• Narrow Bridge Traversal: In scenarios with bridge widths as narrow as 0.15m (narrower than the foot), the VT controller's success rate dropped drastically, while the FT controller maintained near-perfect success rates by explicitly centering steps.
• Complex Environments:
  • Cluttered Navigation: Successfully navigated environments with up to 20 obstacles using an A* planner.
  • Staircases: Demonstrated robust traversal of straight and spiral stairs with varying step heights and lengths. The policy adapted to different geometries, with failure modes shifting from riser contacts (short steps) to balance issues (long steps).
  • Ramps: Successfully generalized to continuous ramps despite being trained only on discrete steps (via moving pillars), confirming strong generalization capabilities.

Real-World Deployment

• Setup: The policy was deployed on the Booster T1 humanoid using an onboard RealSense D455 camera to detect blue circular markers placed on the floor.
• Performance: The system achieved a 93.08% success rate in stepping on designated targets while maintaining stable locomotion.
• Challenges: The primary sources of error were motion-induced image blur and depth sensing artifacts. A simple delay compensation mechanism was implemented to handle communication and inference latencies.
• Outcome: The experiment verified that the policy can be integrated with imperfect visual perception to support downstream tasks without relying on external localization.

Significance and Claims

The paper claims that this work addresses the fundamental gap between high-level planning and low-level execution in humanoid locomotion. By providing explicit, robust control over foot placement, the framework enables:

• Safety: The ability to constrain foot placements to valid, safe contact regions, preventing unsafe behaviors like stepping on obstacles.
• Precision: Enhanced accuracy in reaching targets and traversing constrained environments (narrow bridges, stairs) compared to velocity-based approaches.
• Versatility: A foundational block for complex loco-manipulation tasks in safety-critical scenarios, capable of operating in real-world settings without the need for precise localization or complex, task-specific pipelines.

The authors emphasize that while the framework demonstrates significant improvements in agility and accuracy, it is designed as a general-purpose tool intended to be paired with diverse upstream planners, rather than a monolithic solution for all navigation problems.