💻 computer science

Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion

This paper proposes an Imitating and Finetuning Model Predictive Control (IFM) framework that combines a conventional MPC expert policy with imitation learning and deep reinforcement learning to achieve robust, symmetric, and energy-efficient quadrupedal locomotion on challenging terrains.

Original authors: Donghoon Youm, Hyunyoung Jung, Hyeongjun Kim, Jemin Hwangbo, Hae-Won Park, Sehoon Ha

Published 2026-05-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Donghoon Youm, Hyunyoung Jung, Hyeongjun Kim, Jemin Hwangbo, Hae-Won Park, Sehoon Ha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a four-legged robot dog how to run, jump over obstacles, and walk on slippery floors without falling over. This is a incredibly hard job because the robot has to balance itself perfectly while its legs move in complex patterns.

This paper introduces a new teaching method called IFM (Imitating and Finetuning Model Predictive Control). Think of IFM as a three-step apprenticeship program that combines the best of two worlds: a strict, math-heavy teacher and a creative, trial-and-error student.

Here is how the process works, using simple analogies:

Step 1: The "Strict Math Teacher" (The Expert)

First, the researchers create a "perfect" controller using advanced math (Model Predictive Control or MPC).

The Analogy: Imagine a brilliant but rigid chess grandmaster who calculates every possible move perfectly. This grandmaster knows exactly how the robot should move based on physics equations.
The Problem: This grandmaster is very slow. To calculate the next step, the computer has to do heavy math that takes too long for a real robot to react in real-time. Also, if the robot steps on something unexpected (like a banana peel or a moving conveyor belt), the grandmaster's strict rules might fail because they weren't programmed for that specific weird situation.

Step 2: The "Shadow Student" (Imitation Learning)

Next, they train a neural network (a type of AI brain) to copy the grandmaster.

The Analogy: They put a student in the room with the grandmaster. The student watches the grandmaster make moves and tries to mimic them exactly. This is called "Imitation Learning."
The Result: The student learns the grandmaster's perfect, symmetric, and efficient walking style. But unlike the grandmaster, the student is a simple AI that can make decisions instantly (very fast). However, the student is still just a copy; if the situation changes drastically, the student might still get stuck.

Step 3: The "Adventure Camp" (Reinforcement Learning)

Finally, they send this student out into a "training camp" with difficult terrains (rough rocks, slippery ice, moving belts) to practice on its own.

The Analogy: Instead of starting from scratch, the student already knows how to walk well. Now, they are given a challenge: "Walk across this moving treadmill without falling." The student tries different things. If they slip, they learn. If they succeed, they get a "good job" reward.
The Magic: Because the student started with a good foundation (from Step 2), they don't need thousands of hours of trial and error to learn the basics. They just need to "finetune" their skills to handle the messy, real world.

Why is this better than the old ways?

Speed vs. Brains: The original "Math Teacher" was smart but slow. The new "Student" is almost as smart but can think 15 times faster. This means the robot can react instantly to bumps and slips.
Better Walking Style: If you just let a robot learn from scratch (called "Vanilla RL"), it often learns to walk in a weird, jerky, or asymmetrical way (like a drunk person stumbling). It might work, but it's inefficient and hard to transfer to real hardware. The IFM method forces the robot to keep the "elegant" walking style it learned from the Math Teacher, so it stays symmetric and energy-efficient.
Less "Reward Shaping": Usually, teaching a robot requires the human programmer to write dozens of specific rules (rewards) to tell the robot, "Don't lift your leg too high," or "Keep your back straight." This is tedious and hard to get right. Because IFM starts with a good teacher, the robot already knows the basics of good posture. The researchers only needed a few simple rules to guide the robot through the difficult terrains.

The Results

The team tested this on a real robot called the Mini Cheetah.

Obstacles: The robot could successfully jump over a 7.5cm high step (about the height of a thick book), which other methods failed to do.
Slippery Floors: It could walk across a surface with very low friction (like a banana peel) without falling, whereas the old math-based controller would slip and crash.
Moving Ground: It could walk on a moving conveyor belt, adjusting its footsteps perfectly to stay balanced.

In summary: The paper proposes a method where you first teach a robot the "perfect" way to walk using math, then teach an AI to copy that perfect style, and finally let that AI practice on rough terrain to become a robust, fast, and graceful runner. It's like taking a master dancer, teaching a student their routine, and then sending that student to dance on a trampoline.

Technical Summary: Imitating and Finetuning Model Predictive Control for Robust and Symmetric Quadrupedal Locomotion

Problem Statement

Designing robust locomotion controllers for legged robots remains a significant challenge due to the under-actuated, discrete, and high-dimensional dynamics of these systems. Traditional model-based control (e.g., Model Predictive Control or MPC) has demonstrated impressive agility and robustness but often relies on extensive mathematical modeling and predefined heuristics (like Raibert's). These models struggle in complex, unconventional situations such as rough, slippery, or conveyor terrains where foot slippage or unexpected obstacles occur. Conversely, learning-based approaches (e.g., Deep Reinforcement Learning or Deep RL) can automatically solve unintuitive problems using massive data but often require extensive manual reward shaping to produce high-quality, symmetric, and periodic gaits. Without careful tuning, learned policies may exhibit jerky, asymmetric motions that fail to transfer to real hardware. Furthermore, existing hybrid approaches often suffer from high computational costs due to the need to execute MPC at high frequencies or fail to generalize beyond single reference trajectories.

Methodology: The IFM Framework

The authors propose a novel Imitating and Finetuning Model Predictive Control (IFM) framework that integrates the strengths of model-based control and learning-based algorithms through a three-stage process:

Stage 0: Expert Model Predictive Control (MPC)

The framework begins by developing an expert controller ( $\pi_E$ ) using Differential Dynamic Programming (DDP) and Raibert heuristics.

Design: The MPC predicts future states and generates optimal control inputs based on a convex cost function. It utilizes friction pyramid constraints for stance legs and task-space impedance control for swing legs.
Limitation: While robust, this MPC is computationally expensive (approx. 12.4ms per optimization) and relies on a fixed model, making real-time execution on hardware difficult and limiting adaptability to unseen terrains.
Action Conversion: To bridge the gap with learning algorithms, the MPC's torque outputs are converted into target joint angles using a proportional-derivative (PD) inverse model, making the data compatible with neural network policies.

Stage 1: Imitation Learning (IL)

The expert MPC is used to generate a dataset for pre-training a neural network policy ( $\pi_I$ ).

Algorithm: The authors employ Dataset Aggregation (DAgger). Instead of simple behavior cloning, the policy is rolled out, and the expert MPC corrects the actions for unseen states, aggregating this data to mitigate distribution mismatch.
Architecture: A Multi-Layer Perceptron (MLP) with three hidden layers (512, 256, 64 neurons) takes observations including body orientation, joint states, history, and user commands.
Goal: This stage creates a "clone" of the MPC that is computationally efficient and learnable, inheriting the expert's symmetric and periodic gaits without the heavy optimization cost.

Stage 2: Reinforcement Learning (RL) Finetuning

The imitated policy is further finetuned on challenging terrains using Proximal Policy Optimization (PPO).

Curriculum Learning: The policy is trained on a curriculum of terrains (rough, discrete rough, step, cliff) with increasing difficulty (terrain factor) to prevent the loss of pre-trained motion styles.
Reward Formulation: The authors utilize a Simple Reward (SR) setting (tracking velocity, minimizing unnecessary movement, torque regularization) and a Contact Reward (CR) setting (adding contact biasing). Crucially, the pre-training significantly reduces the burden of reward engineering compared to Vanilla RL.
Constrained Exploration: To maintain the beneficial motion styles learned during imitation, the RL finetuning employs constrained exploration (small initial standard deviation, low learning rates, and small PPO clip parameters).
Sim-to-Real: Domain randomization is applied to observation noise, motor friction, PD gains, ground friction, and command latency to ensure zero-shot transfer to hardware.

Key Contributions

Novel Framework: The proposal of the IFM framework, which systematically combines MPC and RL to leverage the robustness of model-based control and the adaptability of learning-based methods.
Performance Improvement: Demonstration that IFM significantly improves the robustness of the expert MPC on rough, slippery, and conveyor terrains while drastically reducing computation time (from ~15ms to ~1ms).
Efficiency and Quality: Evidence that IFM produces more symmetric, periodic, and energy-efficient gaits compared to Vanilla RL, achieving this with minimal reward shaping and faster convergence.
Hardware Validation: Successful deployment on the Mini Cheetah robot, showing superior success rates in stepping over obstacles and tracking velocities compared to both the original MPC and pure RL baselines.

Experimental Results

The authors conducted extensive simulations and hardware experiments on the Mini Cheetah:

Robustness: In simulation, IFM outperformed the expert MPC and DAgger-only policies on rough, slippery, and unseen conveyor terrains. While MPC failed on conveyor belts due to fixed footfall patterns, IFM adapted its gait effectively.
Tracking Performance: On flat terrain, IFM(SR) achieved the lowest linear tracking errors, outperforming both MPC and Vanilla RL. Vanilla RL(SR) exhibited poor tracking due to "pronking" (excessive vertical motion), a problem mitigated by IFM's pre-training.
Hardware Success Rates:
- Obstacle Crossing: IFM(CR) achieved a 90% success rate in stepping over 7.5cm obstacles, whereas the hardware-tuned MPC (WBIC) and DAgger failed completely at this height.
- Slippery Terrain: IFM policies successfully traversed a surface with a friction coefficient of 0.22, while MPC and DAgger fell due to reliance on Raibert's heuristics which could not handle significant slippage.
Motion Quality: IFM policies demonstrated lower Cost of Transportation (CoT) and Phase-Portrait Index (PPI) compared to Vanilla RL, indicating more energy-efficient and symmetric gaits. The phase portraits of IFM showed near-perfect overlap between left and right legs, unlike the asymmetric motions of Vanilla RL.
Efficiency: IFM converged in approximately 2,000 iterations, significantly faster than Vanilla RL (approx. 4,000+ iterations).

Significance and Claims

The paper claims that the IFM framework offers a balanced solution to the trade-off between the interpretability and robustness of model-based control and the adaptability of learning-based methods. By using MPC as a pre-training expert, the framework:

Reduces the search space for RL, allowing the agent to find high-quality, symmetric gaits without complex reward engineering.
Overcomes the limitations of fixed-model controllers (like MPC) in dynamic and uncertain environments.
Enables real-time control by replacing heavy online optimization with a lightweight neural network.

The authors conclude that IFM provides a structured approach to regulating motion styles, alleviating the challenge of reward engineering while significantly improving the robustness and efficiency of quadrupedal locomotion on challenging terrains. Future work is suggested to extend this method to vision-based locomotion, potentially requiring more complex network architectures like LSTMs or CNNs.