💻 computer science

Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model-Homotopy-Inspired Transfer

This paper presents a dynamic policy learning framework for legged robots that combines simplified single-rigid-body pretraining with a model-homotopy-inspired continuation strategy to efficiently transfer and refine complex dynamic behaviors, such as flips and wall-assisted maneuvers, from reduced-order models to full-body dynamics and real-world deployment.

Original authors: Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dongyun Kang, Min-Gyu Kim, Tae-Gyu Song, Hajun Kim, Sehoon Ha, Hae-Won Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot dog how to do a backflip or jump off a wall. If you just tell the robot, "Go figure it out," it usually crashes, spins wildly, or gives up. It's like trying to learn to ride a bicycle by being thrown off a cliff; the task is too complex, and the robot doesn't know where to start.

This paper presents a clever new way to teach these robots, which the authors call a "continuation-based learning framework." Here is how it works, broken down into simple steps using everyday analogies:

1. The Problem: The "Model Gap"

Robots are heavy, complex machines with many moving parts (legs, joints, motors). Simulating all of this perfectly in a computer is slow and hard to control.

The Old Way: Scientists often use a "simplified model" to teach the robot first. Imagine teaching a robot using a cartoon version of itself—a single floating block with no legs. It's easy to learn on, but when you try to use that knowledge on the real, heavy robot, it fails because the physics are totally different. This difference is called the "model gap."
The Challenge: How do you take what the robot learned in the "cartoon world" and make it work in the "real world" without it falling over?

2. The Solution: A "Gradual Transition" (The Homotopy Path)

Instead of jumping straight from the cartoon to the real robot, the authors created a sliding scale (or a "continuation path") that slowly morphs the cartoon into the real thing.

Think of it like training for a marathon:

Step 1 (The Simplified Model): You start by learning to walk on a flat, soft treadmill. This is the "Single Rigid Body" (SRB) model. The robot learns the core idea of moving its body and pushing off the ground, but without the confusion of heavy legs or complex joints. It's like learning the rhythm of running without worrying about your shoes or the terrain.
Step 2 (The Magic Bridge): This is the paper's big innovation. Instead of switching to the real robot immediately, the system slowly starts adding "weight" and "complexity" to the simulation.
- Imagine the robot's legs are made of helium balloons at the start (very light).
- As the robot gets better, the balloons slowly fill with water, making the legs heavier and heavier.
- Simultaneously, the robot's main body (the trunk) gets lighter to keep the total weight the same.
- By the end of this process, the "balloon legs" have turned into real, heavy metal legs, and the robot has been practicing with them the whole time.

This gradual change is called "Model-Homotopy-Inspired Transfer." It's like a video game where the difficulty increases level by level, rather than throwing the player into the hardest boss fight immediately.

3. What Did They Achieve?

The researchers tested this on a real quadruped robot (a four-legged robot) and in computer simulations. They taught it to do some very fancy moves:

Backflips and Sideflips: The robot learned to tuck and rotate in the air.
Wall-Assisted Maneuvers: The robot learned to push off a wall to jump higher or turn sharply, using the wall as a springboard.

The Results:

Faster Learning: The robot learned these tricks much faster than if they tried to teach it from scratch or just jumped straight to the real physics.
More Stable: Because the robot practiced on the "sliding scale," it didn't get confused when the physics changed. It kept its balance better.
Real-World Success: They successfully deployed these learned moves on a physical robot (a Unitree Go1). The robot could actually do a backflip and run on the real floor, whereas other methods often resulted in the robot crashing or landing on its back.

Summary

In short, the paper says: Don't try to teach a robot complex acrobatics all at once. First, teach it the basics on a simple, simplified version of itself. Then, slowly and smoothly "upgrade" the simulation to look more like the real robot, letting the robot adapt step-by-step. This method acts as a bridge, allowing the robot to carry over its skills from the simple world to the complex real world without falling apart.

Technical Summary: Dynamic Policy Learning for Legged Robot with Simplified Model Pretraining and Model-Homotopy-Inspired Transfer

Problem Statement
Generating highly dynamic motions for legged robots remains a significant challenge due to the complex interplay between body momentum, leg inertia, and contact forces. While Reinforcement Learning (RL) has shown promise, training policies for complex behaviors often requires extensive reward engineering or high-quality expert demonstrations, both of which are resource-intensive. Conversely, approaches utilizing reduced-order models (e.g., Single Rigid Body or SRB models) offer faster training and simpler reward design but suffer from a "model gap" when transferring policies to full-body dynamics. Direct transfer often leads to instability, suboptimal convergence, or a complete failure to replicate the learned behaviors in the real world due to discrepancies in dynamics, particularly regarding leg inertia and contact interactions.

Methodology
The authors propose a continuation-based learning framework that bridges the gap between simplified models and full-body dynamics through two primary stages:

Simplified Model Pretraining (SRB Pretraining):
The policy is first pretrained in an idealized environment using a Single Rigid Body (SRB) model. In this model, the robot is approximated as a floating rigid body with massless legs and ideal point contacts. The control inputs are Ground Reaction Forces (GRFs) during the stance phase and residual foot targets during the swing phase. This stage allows the policy to efficiently learn core motion patterns (such as body momentum control) without the computational burden and instability of full-body dynamics. The environment utilizes predefined contact plans and nominal swing trajectories (e.g., Bezier curves) to structure the learning of specific tasks like gaits, flips, and wall-assisted maneuvers.
Model-Homotopy-Inspired Transfer (MHIT):
To transfer the pretrained SRB policy to the full-body environment, the authors introduce a parametric continuation path. Instead of a direct jump, they define a continuous transition parameter $\lambda \in (0, 1]$ .
- Parametric Transition: The system gradually redistributes mass and inertia between the trunk and the legs. At $\lambda \approx 0$ , the legs have negligible mass (SRB model); as $\lambda$ increases to 1, the legs acquire their full mass and inertia, and the trunk mass is adjusted to maintain constant total mass.
- Continuation Strategy: The policy is trained along this path, starting from $\lambda = 0.01$ and incrementally increasing to $\lambda = 1.0$ over 900 iterations. This allows the policy to adapt to increasingly complex dynamics while remaining in regions of high success probability.
- Action Conversion: During the transfer, the GRF outputs from the SRB policy are converted into joint torques using Jacobian transposes during the stance phase, while task-space PD control tracks foot targets during the swing phase.
- State Estimation: Since privileged information (true linear velocity and contact states) is available in simulation but not in reality, a concurrent estimation network (MLP) is trained to infer these states from proprioceptive history, enabling sim-to-real deployment.

Key Contributions

Continuation-Based Framework: The paper presents a novel framework that integrates simplified model pretraining with a model-homotopy-inspired transfer strategy to efficiently generate and refine complex dynamic motions.
Principled Transition Path: The authors introduce a specific parametric transition path that smoothly interpolates inertial properties between the SRB and full-body models. This approach is shown to improve convergence speed and learning stability compared to direct transfer or imitation learning baselines.
Validation on Complex Tasks and Real Hardware: The method is validated on a diverse set of challenging dynamic tasks, including backflips, sideflips, yaw spins, and wall-assisted maneuvers (jumps and turns). The learned policies are successfully deployed on a real Unitree Go1 quadrupedal robot.

Results

Simulation Performance: In comparative studies against Direct Transfer (DT), Imitation Transfer (IT), and Vanilla RL, the proposed MHIT method demonstrated the fastest convergence and the most consistent learning outcomes. For instance, in the wall-assisted backflip task, MHIT converged twice as fast as IT and achieved a 19.0% higher normalized return. It also exhibited superior stability, with a significantly smaller variation in performance across different random seeds compared to DT.
Robustness: MHIT policies showed greater robustness to external force and torque disturbances compared to IT policies, attributed to the preservation of reactive behaviors learned during pretraining and the ability to explore a wider state space without the constraints of early termination used in imitation learning.
Real-World Deployment: On the Unitree Go1 robot, the MHIT policy successfully executed a backflip, a task where the direct SRB policy failed due to insufficient angular momentum caused by unmodeled leg inertia. The transferred policy achieved a 90% success rate over 10 consecutive trials and demonstrated superior velocity tracking accuracy (lower RMSE) compared to the unrefined SRB policy.

Significance
The paper claims that its primary significance lies in providing an effective and stable refinement process to bridge the model gap between simplified and full-body dynamics without relying on expensive expert demonstrations. By leveraging a continuation strategy, the framework enables the discovery of optimal behaviors that surpass the limitations of the initial simplified guidance. The successful deployment on real hardware demonstrates the practical applicability of this approach for generating complex, dynamic legged locomotion behaviors that are difficult to achieve with standard end-to-end RL or hierarchical tracking methods alone. The authors note that while the current approach relies on predefined contact sequences, the framework offers a promising evolutionary starting point for future work aimed at automating contact schedules and adapting to diverse robot morphologies.

1. The Problem: The "Model Gap"

2. The Solution: A "Gradual Transition" (The Homotopy Path)

3. What Did They Achieve?

Summary

More like this