⚡ electrical engineering

Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

This paper introduces MPC-RL, a framework that accelerates and scales humanoid locomotion and manipulation training by combining a centroidal-dynamics MPC reward formulation with $\pi^n$ MPC, a novel parallel-in-horizon GPU solver that eliminates construction overhead and enables efficient, constraint-aware reinforcement learning.

Original authors: Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang, Ján Drgoňa, Aaron D. Ames

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Junheng Li, Liang Wu, Sergio A. Esteban, Lizhi Yang, Ján Drgoňa, Aaron D. Ames

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to walk and push heavy objects. You have two main ways to do this:

The "Gym Coach" (Reinforcement Learning): You let the robot try millions of random movements in a video game simulator. It learns by trial and error, eventually figuring out how to walk without falling. It's great at being tough and adaptable, but it can take a long time to learn the basics, and sometimes it learns "bad habits" that look weird or inefficient.
The "Physics Professor" (Model Predictive Control): You give the robot a strict set of physics equations. It calculates exactly how to move its center of mass and where to put its feet to stay balanced. It's very precise and safe, but it's slow to compute and can be rigid, struggling if the real world gets messy or unpredictable.

This paper introduces a new method called MPC-RL that combines the best of both worlds. Think of it as hiring the "Physics Professor" to be a training coach for the "Gym Coach" student, but only during practice, not during the actual game.

The Core Idea: A Smart Training Guide

Instead of letting the robot figure out walking from scratch, the system uses the "Physics Professor" (MPC) to generate a perfect roadmap of how the robot should move. It doesn't control the robot directly; instead, it gives the robot a "reward" (like a gold star) whenever it follows this roadmap.

Over time, the robot (using Reinforcement Learning) learns to mimic this perfect roadmap so well that it eventually becomes an expert on its own. Once the training is done, the robot forgets the "Physics Professor" and runs on its own, ready to handle real-world bumps and pushes.

The Two Big Problems They Solved

1. The "Traffic Jam" Problem (Speed)
Usually, asking a physics engine to calculate a perfect roadmap for a robot is slow. If you try to do this for thousands of robots at once (which is needed for fast training), the computer chokes. It's like trying to solve a million math problems one by one; it takes forever.

Their Solution ( $\pi$ nMPC): They built a special tool called $\pi$ nMPC. Imagine a massive factory assembly line where, instead of solving one math problem at a time, the computer solves thousands of them simultaneously on a graphics card (GPU). They also removed the need to "build" the math problem from scratch every time (construction-free), allowing the system to run incredibly fast without running out of memory. This makes training fast enough to be practical.

2. The "Too Much Information" Problem (Confidence)
The "Physics Professor" is great at predicting the next step, but its predictions get fuzzier the further out it looks (like trying to predict the weather a month from now). If you tell the robot to follow the roadmap too strictly for the whole future, it might get confused by the "fuzzy" parts.

Their Solution (Confidence Weighting): They taught the robot to trust the roadmap differently depending on how far away it is.
- Near-term (Next step): "Follow this exactly! I'm 100% sure."
- Far-term (Future steps): "This is a good general direction, but don't stress if you drift a little."
  This "tapered" trust helps the robot learn the big picture without getting stuck on minor details.

What Did They Achieve?

The researchers tested this on a humanoid robot (a robot that looks and moves like a human) in two main areas:

Walking: The robot learned to walk faster and more stably than robots trained with standard methods. It could recover from being pushed hard (like someone shoving it) and keep walking without falling. It also handled walking while wearing a heavy vest or carrying a load.
Pushing Heavy Things (Loco-Manipulation): This is the big flex. They taught the robot to push a cart.
- The Result: The robot successfully pushed a cart weighing 290 kg (639 lbs).
- The Scale: That cart weighs 829% of the robot's own body weight. To put that in perspective, if you weigh 150 lbs, this robot pushed a cart that weighed as much as 1,243 lbs (roughly the weight of a small car or a grand piano).

The Bottom Line

The paper shows that by using a fast, parallel computer solver to give a "physics-based" roadmap during training, you can teach robots to walk and push heavy objects much better and faster than before. The robot learns the "physics" of movement quickly, then becomes a robust, independent worker that doesn't need the computer to do the math in real-time.

In short: They built a super-fast training coach that helps robots learn to walk and push heavy loads by showing them the "perfect path," resulting in a robot that can push a cart nearly 10 times its own weight.

Technical Summary: Accelerating and Scaling MPC-Guided Reinforcement Learning for Humanoid Locomotion and Manipulation

1. Problem Statement

Humanoid motion control faces a dichotomy between Model Predictive Control (MPC) and Reinforcement Learning (RL). MPC offers physically grounded prediction and explicit constraint handling but often struggles with real-time computation and robustness to model mismatch. Conversely, RL enables robust, agile whole-body skills through large-scale simulation but often lacks the physical structure and long-horizon planning capabilities inherent to MPC.

While hybrid approaches exist, integrating MPC inside the RL training loop presents significant challenges:

Computational Overhead: Repeatedly solving MPC problems during training is time-consuming.
Scalability Bottlenecks: Existing GPU-based MPC solvers often parallelize across environments but fail to parallelize effectively across the prediction horizon, leaving sequential recursions as a bottleneck.
Memory and Compilation Costs: Traditional sparse Quadratic Programming (QP) formulations for MPC require time-consuming problem construction and symbolic compilation, leading to high VRAM usage that scales poorly with problem size.
Limited Scope: Prior hybrid frameworks focus primarily on locomotion, leaving loco-manipulation (simultaneous locomotion and object interaction) under-explored despite the high value of physical models in these tasks.

2. Methodology

The proposed framework, MPC-RL, decouples the roles of MPC and RL: MPC acts as a training-time guide to generate structured rewards, while the learned policy operates without MPC at deployment time.

A. Centroidal-Dynamics MPC Guidance (CD-MPC)

Instead of using MPC for direct control, the system uses a Centroidal Dynamics (CD) model to generate predictive references for the RL agent.

Model: The CD-MPC tracks the Center of Mass (CoM), centroidal linear momentum ( $l_G$ ), and centroidal angular momentum ( $k_G$ ). It handles time-varying dynamics by incorporating active contact sets (feet for locomotion; feet and hands for loco-manipulation).
Landmark Extraction: Rather than forcing the policy to track the immediate next MPC step (which discards long-horizon structure), the system extracts a set of predictive landmarks from the full MPC trajectory at specific time intervals ( $\delta_\ell$ ).
Prediction-Confidence Weighting: The reward function compares the realized CoM against these landmarks. A horizon-monotone weighting schedule ( $\alpha_0 > \alpha_1 > \dots$ ) is applied, heavily weighting near-horizon landmarks (which are more reliable) and down-weighting far-horizon landmarks (which are subject to model mismatch and integration error). This creates a bounded reward signal that fuses long-horizon guidance with sharp near-term tracking.

B. $\pi$ nMPC: Parallel-in-Horizon and Construction-Free Solver

To make CD-MPC feasible for massively parallel RL training, the authors introduce $\pi$ nMPC, a batched GPU solver.

Architecture: It extends the parallel-in-horizon ADMM structure of previous work ( $\pi$ MPC) to handle time-varying dynamics (essential for legged robots with changing contact sequences).
Key Innovations:
1. Velocity-Form MPC: The solver optimizes over input increments rather than absolute inputs, lifting the state to include previous inputs.
2. Variable Splitting: A novel splitting scheme introduces auxiliary variables to decouple the dynamics from the constraints. This allows the ADMM updates to be executed in parallel across the entire prediction horizon on the GPU, rather than sequentially.
3. Construction-Free: The solver operates directly on time-varying state matrices ( $A_k, B_k, e_k$ ) without assembling a sparse QP or requiring symbolic code generation/compilation at every step. This significantly reduces VRAM usage and eliminates pre-compilation delays.
Implementation: Realized in both PyTorch and JAX, supporting massive batch sizes (up to 4096 environments) and long horizons (up to $N=400$ ).

C. Training and Deployment

Algorithm: Proximal Policy Optimization (PPO) is used with an asymmetric actor-critic architecture.
Observations: The actor (deployed on hardware) receives only proprioception and commands. The critic (training only) is privileged with the MPC's predicted CoM and angular momentum references.
Rewards: The total reward combines standard regularization (joint limits, foot clearance) with the MPC landmark guidance terms (CoM, velocity, angular momentum, foot placement, and ground reaction forces).

3. Key Contributions

Structured MPC Reward Formulation: A novel reward mechanism using prediction-confidence weighting to extract long-horizon motion and dynamics guidance from centroidal MPC trajectories, validated through ablation studies on horizon length and update rates.
$\pi$ nMPC Solver: A construction-free, memory-efficient, batched GPU MPC solver that achieves parallel-in-horizon execution for time-varying dynamics, overcoming the scalability bottlenecks of existing QP-based solvers.
Extension to Loco-Manipulation: The first demonstration of MPC-guided RL for humanoid loco-manipulation, successfully transferring to hardware for pushing heavy payloads (up to 290 kg).

4. Results

A. Solver Scalability

Performance: $\pi$ nMPC scales favorably on GPUs compared to state-of-the-art solvers (Consensus ADMM, qpth, qpax, CusADi).
Efficiency: Unlike CusADi, which requires hours of symbolic compilation and high RAM, $\pi$ nMPC operates without compilation. It remains feasible for long horizons ( $N=400$ ) and large batch sizes where other solvers run out of VRAM or become prohibitively slow.
Implementation: The JAX variant is computationally faster, while the PyTorch variant offers lower memory overhead during RL training.

B. Humanoid Locomotion

Velocity Tracking: MPC-RL consistently outperforms Pure RL in translational velocity tracking, including out-of-distribution commands.
Push Recovery: The framework demonstrates superior robustness, recovering from pushes up to 450 N (depending on configuration), significantly higher than the Pure RL baseline (300 N).
Hyperparameters: A horizon of $N=10$ (approx. 0.7s, one gait cycle) with a 10 Hz update rate yielded the best balance of performance and training efficiency. Longer horizons did not yield monotonic improvements.
Reward Design: The proposed "tapered" landmark weighting schedule outperformed "next-step" and "averaged" reward designs in both tracking accuracy and push recovery.

C. Loco-Manipulation

Hardware Validation: The policy was deployed on the Westwood Robotics Themis V2 robot.
Capability: The robot successfully pushed a cart with a 290 kg (639 lbs) payload, representing 829% of the robot's body mass.
Comparison: In simulation, the MPC-guided policy achieved better velocity tracking and pushing capability compared to Pure RL, which often stalled. The structured MPC rewards provided necessary physical guidance for forceful interaction that pure RL struggled to discover through exploration alone.

5. Significance and Claims

The paper claims that predictive optimization can serve as a practical training tool and a source of physical structure for learning humanoid behaviors with competitive training efficiency.

Practicality: The framework avoids the need for real-time MPC at deployment, relying instead on a learned policy that is robust to disturbances (e.g., unknown weights, pushes).
Efficiency: By introducing $\pi$ nMPC, the authors demonstrate that long-horizon MPC guidance is computationally feasible within the massive parallelism of modern RL training, removing the "time-consuming problem construction" barrier.
Generalizability: The success in loco-manipulation suggests that MPC guidance is not limited to locomotion but is critical for tasks requiring complex force interactions and dynamic consistency.

Limitations and Future Work:
The authors acknowledge that the current approach relies on reduced-order centroidal dynamics with prescribed contact schedules. Future work aims to extend the framework to richer contact modes using nonlinear MPC and to investigate neural-network-augmented batch solvers for even more efficient training-time guidance.