Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

1. Problem Statement

Humanoid locomotion faces significant challenges due to intermittent contacts, high degrees of freedom, and strict safety constraints. While Model Predictive Control (MPC) is a dominant paradigm for handling these constraints via online optimization, it relies on simplified models (e.g., centroidal dynamics) to meet real-time computational requirements. This simplification leads to model mismatch with high-fidelity physical execution, often resulting in suboptimal performance that requires extensive manual tuning of cost weights and terminal penalties.

Reinforcement Learning (RL) offers a data-driven alternative but suffers from sample inefficiency and difficulty in enforcing hard safety constraints. Existing hybrid approaches (MPC-RL) that treat MPC as a learnable function approximator face a critical bottleneck: standard gradient-based training requires repeatedly solving the MPC optimization problem within the learning loop. For complex, time-critical humanoid systems, this "solve-in-the-loop" approach is computationally prohibitive.

2. Methodology: Cost-Matching MPC (CM-MPC)

The authors propose Cost-Matching MPC (CM-MPC), a framework that learns MPC parameters by minimizing the discrepancy between the MPC's predicted cost-to-go and the actual long-horizon returns observed from system execution, without solving the MPC optimization during training.

Core Concept

Instead of differentiating through the MPC solver, the method treats the MPC as a parameterized function approximator for the action-value function ($Q$).

1. Parameterization: The MPC predictive model (centroidal dynamics) and the cost function (stage and terminal costs) are parameterized by a vector $\theta$.
2. Surrogate Value ($Q^{MPC}_{\theta}$): For a given state and a recorded sequence of actions, the MPC value is estimated by rolling out the parameterized model forward. This involves accumulating stage costs and adding differentiable penalties for constraint violations (soft constraints) along the trajectory.
3. Measured Return ($Q^{meas}$): This is the actual discounted cumulative cost observed from the high-fidelity system (simulator or robot) over a long horizon.
4. Learning Objective: The goal is to minimize the Mean Squared Error (MSE) between the surrogate value and the measured return:
   $$\min_{\theta} \mathbb{E} \left[ (Q^{MPC}_{\theta} - Q^{meas})^2 \right]$$
5. Training Loop:
   • Collect trajectories using the current MPC policy.
   • Compute $Q^{meas}$ from the recorded data.
   • Compute $Q^{MPC}_{\theta}$ by rolling out the parameterized model on the same action segments.
   • Update $\theta$ via gradient descent.
   • Crucial Advantage: This avoids solving the constrained optimization problem (5) during training, reducing computational complexity from exponential/iterative to linear with respect to the horizon length.

System Instantiation

• Dynamics: Uses a complete centroidal dynamics model (OCS2 formulation) capturing linear/angular momentum and joint velocities.
• Constraints: Includes joint limits, foot collision avoidance (SDF), friction cones, and Center of Pressure (CoP) constraints.
• Cost Function: Decomposed into tracking, base stability, CoM balance, swing-foot pose, and torque regularization.
• Learnable Parameters:
  • Gains in the momentum propagation equations (to correct model mismatch).
  • Diagonal scaling vectors for quadratic cost weights ($Q, R, Q_f$).

3. Key Contributions

1. Efficient Learning Framework: Introduces a cost-matching approach that eliminates the need for repeated MPC solves during training, making MPC-RL feasible for complex, real-time humanoid systems.
2. Theoretical Convergence: Proves that the stochastic cost-matching update converges to a first-order stationary point under standard smoothness and bounded-variance assumptions.
3. Robustness via Data-Driven Adaptation: Demonstrates that learning the terminal costs and model gains effectively compensates for the "myopia" of short-horizon MPC and systematic model mismatches (e.g., unmodeled compliance, tracking stack abstractions).
4. Open-Source Implementation: Provides a full implementation for a commercial humanoid platform (Unitree G1).

4. Results and Analysis

The method was validated in high-fidelity simulation on a Unitree G1 humanoid robot, comparing the learned CM-MPC against a manually tuned baseline.

• Value Consistency: The training successfully minimized the discrepancy between the MPC surrogate cost-to-go and the measured long-horizon returns. The terminal cost weights ($Q_f$) increased significantly during training, effectively acting as an implicit "tail-cost" approximation to correct for short-horizon bias.
• Disturbance Rejection: Under external disturbances (lateral force pulses and yaw torque):
  • Faster Recovery: CM-MPC reduced settling time by 31.25% compared to the baseline.
  • Smoother Control: It exhibited lower control effort and reduced "jitter" (input differences), indicating more stable corrective actions.
  • Trade-offs: While strict momentum tracking errors remained comparable, the controller prioritized whole-body stability and disturbance rejection over exact reference tracking, a behavior implicitly learned through the cost-matching process.
• Statistical Improvements:
  • Post-Push Peak Error: Reduced by 4.80%.
  • Integral Absolute Error (IAE): Reduced by 6.82%.
  • Control Effort: Reduced high-percentile control effort by 5.26%.

5. Significance

This work bridges the gap between the safety guarantees of constrained MPC and the adaptability of Reinforcement Learning.

• Computational Efficiency: By decoupling the learning process from the MPC solver, it enables the training of complex locomotion policies that were previously too computationally expensive to learn via standard MPC-RL.
• Robustness: It provides a systematic way to tune MPC parameters using real-world data, automatically compensating for model inaccuracies and simplifying the manual tuning burden for humanoid robots.
• Practical Deployment: The resulting controller retains the standard constrained MPC structure for deployment, ensuring safety and feasibility while benefiting from data-driven performance improvements.

In summary, Cost-Matching MPC offers a scalable, efficient, and robust solution for learning optimal locomotion behaviors in humanoid robots, successfully addressing the trade-off between computational cost and performance in model-based reinforcement learning.