Technical Summary: Efficiently Learning Robust Torque-based Locomotion Through Reinforcement with Model-Based Supervision

Problem Statement

Learning effective torque-based locomotion policies for bipedal robots remains a significant challenge. While torque control enables responsive and compliant behaviors, it requires precise handling of complex, nonlinear dynamics. Direct policy learning at the torque level is particularly difficult in the presence of real-world uncertainties, such as contact uncertainty, sensor noise, actuation delays, and unmodeled dynamics (e.g., joint friction, structural flexibilities).

Traditional model-based control methods (e.g., DCM trajectory planning with Whole-Body Control) rely on accurate system models. However, their performance degrades when discrepancies between the model and the physical system become substantial. Conversely, purely data-driven Deep Reinforcement Learning (DRL) approaches, while robust to modeling errors via domain randomization, often suffer from data inefficiency, sensitivity to reward design, and a lack of interpretability or safety assurances. Existing hybrid strategies often rely solely on reward signals, requiring extensive trial-and-error iterations and offering limited guidance during training.

Methodology

The authors propose a control framework that integrates model-based bipedal locomotion with residual reinforcement learning (RL), enhanced by a novel model-based supervision mechanism. The architecture consists of three primary components:

1. Base Policy ($\pi_b$): A model-based controller comprising a Divergent Component of Motion (DCM) trajectory planner and an inverse dynamics-based Whole-Body Controller (WBC). This policy operates under an inaccurate dynamics model and noisy state observations, serving as a reliable but imperfect foundation.
2. Oracle Policy ($\pi^*$): A model-based controller with "privileged" access to ground-truth system information, including exact robot dynamics, true state (noise-free), and motor parameters (e.g., actuation delays, efficiency). This policy generates ideal corrective actions under randomized conditions but is used only during training as a supervisor.
3. Residual RL Policy ($\pi_r$): A learnable policy trained via Proximal Policy Optimization (PPO) to output residual torque commands that compensate for the inaccuracies of the Base Policy.

Key Technical Components:

• Domain Randomization: The training environment randomizes sensor noise, robot dynamics (mass, friction, damping, motor efficiency), and environmental factors (floor friction, unevenness) to expose the policy to real-world variability.
• Supervised Reinforcement Learning: Unlike standard RL that relies solely on reward shaping, this framework introduces a supervised loss term ($L_{sup}$). The residual policy is explicitly guided to mimic the corrective actions of the Oracle policy.
  • The total loss is a weighted sum: $L_{total} = \omega_{rl} L_{rl} + \omega_{sup} L_{sup}$.
  • The supervised loss encourages the policy to assign high likelihood to the Oracle's residual action ($\tau^*_t - \tau^b_t$).
• Observation Space: The policy utilizes a recurrent neural network (LSTM) to process a history of noisy observations (joint states, base policy torques, tracking errors). The critic network additionally utilizes privileged observations (true states, Oracle torques) for more accurate value estimation during training.

Key Contributions

The paper outlines four primary contributions:

1. Integrated Control Framework: A novel architecture combining the structured reliability of model-based control with the adaptability of residual RL to mitigate model inaccuracies.
2. Model-Based Oracle Policy: The introduction of an Oracle policy that leverages privileged information to generate near-optimal control signals, serving as a supervisor for the learning process.
3. Supervised RL Objective: A training framework that incorporates Oracle actions into a supervised loss, complementing the standard RL objective to improve learning efficiency without relying exclusively on reward shaping.
4. Cross-Platform Validation: Comprehensive evaluations on three distinct bipedal robots (Kangaroo, Unitree H1-2, and Bruce), demonstrating robustness and generalization across different hardware platforms without specific parameter tuning.

Results

The method was evaluated on Kangaroo, Unitree H1-2, and Bruce robots under extensive domain randomization ($\beta=1$).

• Performance vs. Baselines:
  • Oracle (O): Achieved 100% success rate across all platforms with minimal tracking errors, validating its role as an effective supervisor.
  • Base Policy (B): Failed completely (0% success) under randomization, highlighting the brittleness of standard model-based control without adaptation.
  • Standard Residual RL (ResRL): Showed poor performance (0-10% success), struggling to learn torque-based locomotion without explicit guidance.
  • Imitation Learning (IL): Performed well on Kangaroo (70-100% success) but struggled on Bruce (80% success), suggesting sensitivity to the Oracle's reliability in specific dynamic regimes.
  • Proposed Method (BOR - Base + Oracle Residual): Achieved 100% success on Kangaroo and H1-2, and 80% on Bruce. It significantly outperformed standard ResRL and pure IL, approaching Oracle-level performance in DCM and foot tracking.
• Robustness to Uncertainty: When tested with varying levels of randomization ($\beta \in [0.1, 0.7]$), the proposed method maintained a 100% success rate, whereas the standard Model-Based Controller (MBC) failed significantly as uncertainty increased.
• Training Efficiency: The supervised approach converged to over 80% success rate in just 1,500 episodes (approx. 4.16 hours of simulation), whereas standard residual RL showed little progress under the same conditions.
• Generalization: The framework successfully transferred to uneven terrain and different robot morphologies without re-tuning, demonstrating the ability to learn adaptive behaviors that go beyond simple imitation of the Oracle.

Significance and Claims

The paper claims that this approach effectively bridges the gap between model-based control and data-driven RL. By integrating model-based priors with data-driven adaptation, the method offers a scalable solution for sim-to-real transfer in bipedal locomotion.

The authors emphasize that the framework improves learning efficiency and robustness by using the Oracle to guide the policy toward corrective behaviors without extensive reward engineering. They note that while the Oracle provides a strong supervisory signal, the RL component is crucial for exploring actions that deviate from the Oracle when necessary (e.g., in highly stochastic conditions or on uneven terrain where the Oracle's model-based planning may be suboptimal). The work demonstrates that supervised reinforcement learning provides an efficient, robust, and transferable solution for torque-based locomotion control in humanoid robots.