RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion

This paper proposes a contact-explicit hierarchical architecture that combines Reinforcement Learning for high-level gait and navigation planning with low-level Model Predictive Control, successfully achieving robust zero-shot sim-to-sim and sim-to-real transfer across diverse legged and hybrid robotic platforms without domain randomization.

The Gist

Imagine you are teaching a robot dog to run, jump, and roll around a room. The biggest challenge isn't just making it move; it's figuring out when to put a foot down, when to lift it up, and when to switch from walking on wheels to walking on legs.

Traditionally, engineers tried to solve this by writing a massive, complex rulebook for the robot. They'd say, "If you are here, lift your left leg. If you turn, switch to wheels." But real life is messy. If the robot slips or the ground changes, the rulebook often breaks.

This paper introduces a smarter way to teach robots: A "Brain" and a "Reflex" working together.

The Two-Part Team

Think of the robot's control system as a team of two people:

1. The High-Level Brain (Reinforcement Learning): This is the strategic commander. It doesn't worry about the tiny details of muscle movement. Instead, it looks at the big picture: "I need to go to that corner," or "I need to turn around." It learns through trial and error, just like a puppy learning to walk. It figures out the rhythm of movement: When should I step? When should I jump? When should I roll?
2. The Low-Level Reflex (Model Predictive Control - MPC): This is the expert mechanic. It knows exactly how the robot's body works physically. It takes the Brain's vague command ("Go forward, maybe jump a bit") and instantly calculates the exact force needed for every motor to make that happen without falling over. It handles the physics, the friction, and the balance.

The Magic Trick: Learning the "Gait"

In the past, robots needed a pre-programmed "gait" (like a specific trot or gallop). If the robot needed to do something weird, like a backflip or a sudden stop, the pre-programmed gait failed.

In this new system, the Brain learns the gait on the fly.

• The Analogy: Imagine learning to dance. Instead of memorizing a specific dance routine, you just listen to the music and let your body figure out the steps. Sometimes you do a slow shuffle, sometimes a quick hop, sometimes you spin. You don't need a script; you just react to the music.
• The Result: The robot discovers non-periodic gaits. This means it doesn't just repeat the same "left-right-left-right" pattern. It might do "left-right-jump-roll-left" depending on what the task requires. It adapts instantly.

Why This is a Big Deal

1. No "Cheat Codes" Needed (Zero-Shot Transfer)
Usually, to teach a robot to walk in the real world, engineers have to simulate thousands of different scenarios in a computer first (random wind, slippery floors, broken sensors) so the robot learns to handle anything. This is called "domain randomization."

• The Paper's Breakthrough: They trained the robot in a simple simulation and then sent it straight to the real world (a 120kg robot named Centauro) without any of those cheat codes. It worked immediately.
• The Analogy: It's like learning to ride a bike in a quiet parking lot and then immediately riding it down a busy, bumpy city street without falling. The "Reflex" (MPC) is so good at physics that it bridges the gap between the computer and reality.

2. Efficiency
The robot learns to be energy-efficient. When it's just rolling on wheels, it does that. When it needs to climb a step, it switches to legs. It figures out the most energy-saving way to move, just like a human who walks instead of running when they aren't in a hurry.

3. The "Software Factory"
To make this work, the team built a special software factory that can run thousands of these robot simulations at the same time on a single computer. This allowed the "Brain" to learn millions of steps in a few days, something that would take years if done one by one.

The Real-World Test

They tested this on three different robots:

• A small 50kg dog-like robot.
• A medium 80kg wheeled robot.
• A large 120kg humanoid robot with wheels and legs (Centauro).

The Result: The robot successfully walked, rolled, and climbed stairs. When the path was flat, it rolled. When it hit a pyramid of steps, it switched to legs and climbed up, adjusting its steps dynamically. It didn't stumble because it had a pre-set plan; it succeeded because it learned to adapt in real-time.

Summary

This paper is about giving robots a flexible mind and a strong body.

• The Mind (RL) learns what to do by experimenting.
• The Body (MPC) knows how to do it by understanding physics.

Together, they create a robot that doesn't just follow a script, but can figure out how to move through a messy, unpredictable world on its own. It's the difference between a robot that is programmed to dance a specific waltz, and a robot that can dance to any song, in any style, without ever missing a beat.

Technical Summary

Here is a detailed technical summary of the paper "RL-Augmented MPC for Non-Gaited Legged and Hybrid Locomotion."

1. Problem Statement

Legged and hybrid (wheeled-legged) robots face a fundamental challenge in locomotion control: contact scheduling.

• Model-Based Limitations: Traditional Model Predictive Control (MPC) is highly effective for motion planning and constraint handling but struggles with the combinatorial complexity of determining when and where to place contacts (gait timing). Solving for optimal contact sequences online often leads to Mixed-Integer Nonlinear Programming (MINLP) problems that are too computationally expensive for real-time execution.
• Model-Free Limitations: Pure Reinforcement Learning (RL) approaches can learn contact-implicit policies but often suffer from sample inefficiency, require extensive domain randomization for sim-to-real transfer, and rely heavily on reward shaping. They also lack the explicit constraint handling and interpretability of MPC.
• The Gap: There is a need for an architecture that leverages the robustness and constraint handling of MPC while allowing an RL agent to learn complex, adaptive, and non-periodic gait patterns without predefined demonstrations or heavy tuning.

2. Methodology

The authors propose a hierarchical, contact-explicit architecture that couples a high-level RL agent with a low-level MPC controller.

A. Hierarchical Architecture

• High-Level (RL Agent): Uses Soft Actor-Critic (SAC) to learn a policy that outputs:
  1. Navigation Commands: Base twist references ($\xi_{MPC}$).
  2. Contact Scheduling: Injection actions ($\chi_{MPC}$) that trigger flight phases (lifting a foot) or modify contact timing.
  • Key Innovation: The RL agent learns acyclic (non-periodic) contact patterns directly through trial and error, rather than relying on predefined gaits (e.g., trot, walk) or demonstrations.
• Low-Level (MPC Controller): A full rigid-body dynamics MPC that executes the motion.
  • It assumes a predefined contact schedule over the optimization horizon but allows the RL agent to inject flight phases dynamically at runtime.
  • It solves a parametric Nonlinear Programming (NLP) problem using an Iterative Linear Quadratic Regulator (ILQR) solver.
  • It handles constraints such as friction cones, unilaterality, and joint limits.

B. Key Technical Components

• Contact Injection Logic: The MPC horizon is divided into phases. The RL agent can trigger a "flight phase" (lifting a foot) at specific injection nodes. This allows the system to adapt gait timing based on task demands (e.g., turning, stopping) without re-solving a combinatorial contact problem.
• State Observation: The RL agent observes a minimal set of MPC-derived states (predicted linear velocity, contact forces, and a "health" index indicating solver convergence) rather than the full MPC state, making the approach more robust to partial observability in real-world scenarios.
• Reward Function: Designed to be minimal and task-agnostic, focusing on:
  1. Tracking: Following the commanded base twist.
  2. Action Regularization: Penalizing rapid changes in control inputs.
  3. Cost of Transport (CoT): Promoting energy efficiency.
• Software Framework: A scalable, modular system capable of running thousands of parallel MPC instances on CPU while interfacing with GPU-based simulation (IsaacSim/MuJoCo). This enables high-throughput training without the need for domain randomization.

3. Key Contributions

1. Acyclic Gait Emergence: The system successfully learns non-periodic, task-adaptive contact patterns (e.g., alternating single/double flight phases) purely through RL, eliminating the need for predefined gait libraries.
2. Zero-Shot Transfer: The architecture achieves zero-shot sim-to-sim and zero-shot sim-to-real transfer across diverse platforms (50kg to 120kg) without domain randomization. This is a significant departure from standard deep RL practices.
3. Hybrid Locomotion: The method unifies control for both standard legged robots and hybrid wheeled-legged robots (like the Centauro), allowing seamless transitions between wheel-based rolling and stepping behaviors.
4. Scalable Training Framework: The authors developed a custom parallelization library that runs thousands of MPCs on CPU, enabling efficient training of complex rigid-body models that are typically too slow for standard vectorized RL.

4. Experimental Results

The approach was validated on three distinct platforms:

• Simplified 50kg Quadruped (Legged and Wheeled configurations).
• Unitree B2-W (80kg Wheeled Quadruped).
• Centauro (120kg Wheeled-Legged Humanoid).

Key Findings:

• Performance: Policies trained in simulation transferred directly to real hardware (Centauro) with no fine-tuning. The robot successfully performed legged and hybrid locomotion on flat terrain and non-flat terrains (stepped pyramids).
• Gait Adaptation: The robot demonstrated adaptive behaviors, such as slowing down gait frequency near targets and switching between symmetric and asymmetric trotting patterns based on direction changes.
• Energy Efficiency: Hybrid locomotion (using wheels where possible and stepping when necessary) showed a significantly lower Cost of Transport (CoT ≈ 0.12) compared to pure legged locomotion (CoT ≈ 0.35).
• Sample Efficiency: The approach required fewer environment steps to converge compared to blind end-to-end RL, despite the computational cost of running MPCs.
• Robustness: The system maintained stability and tracking performance even when the MPC was running in open-loop mode during real-world deployment (due to lack of onboard odometry).

5. Significance

This work represents a significant step forward in legged robotics control by bridging the gap between the robustness of model-based control and the adaptability of learning-based methods.

• Practicality: By removing the need for domain randomization and predefined gaits, the method offers a more practical path to deploying complex locomotion controllers on real-world robots.
• Generalization: The ability to handle different morphologies (quadrupeds vs. humanoids) and locomotion modes (wheeled vs. legged) with a single framework suggests high scalability.
• Future Potential: The architecture's ability to incorporate raw heightmap data and control flight phase parameters opens the door to navigating unstructured, non-flat environments, moving beyond the limitations of flat-terrain research.

In summary, the paper demonstrates that coupling RL with MPC allows robots to learn how to move (gait timing) while the MPC ensures safe and feasible execution, resulting in a robust, efficient, and highly adaptable locomotion system.