💻 computer science

Sampling Strategies for Robust Universal Quadrupedal Locomotion Policies

This paper investigates and compares various joint gain sampling strategies, including performance-based filtering and uniform randomization, to train a single robust reinforcement learning policy for universal quadrupedal locomotion that successfully bridges the sim-to-real gap through zero-shot deployment on the ANYmal robot.

Original authors: David Rytz, Kim Tien Ly, Ioannis Havoutis

Published 2026-06-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David Rytz, Kim Tien Ly, Ioannis Havoutis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a dog how to walk. If you only train a tiny Chihuahua, it won't know how to walk like a Great Dane. If you only train a Great Dane, it might trip over its own paws if you suddenly shrink it down.

This paper is about teaching a computer brain (an AI) to control four-legged robots of all different shapes and sizes, without having to retrain the brain every time you swap the robot.

Here is the breakdown of their discovery, using simple analogies:

The Problem: The "One-Size-Fits-None" Trap

Most robot controllers are like custom-tailored suits. If you design a controller for a specific robot (say, a 12kg robot), it works great. But if you try to put that same "suit" on a heavier robot (like a 50kg robot), it falls apart.

To fix this, scientists usually use Reinforcement Learning (RL). Think of this as training a video game character by letting it fail thousands of times until it learns the right moves. The problem is, if you only train the character on one specific body type, it gets confused when you give it a new body.

The Solution: The "Gym Class" Strategy

The authors realized that to make a robot that can walk on any body, you have to train it in a "gym class" where the robots constantly change their weight, leg length, and muscle strength.

However, there was a catch: How do you change these settings?

If you just pick random settings (like rolling dice), you might accidentally give a robot "super-muscles" that are too strong for its joints, or "weak muscles" that can't move it. This is like telling a student to run a marathon with shoes that are either made of lead or made of jelly. It won't work.

The Three Strategies They Tested

The team compared three ways to "mix up" the robot's settings during training:

The "Math Formula" Approach: They tried to guess the right muscle strength based on the robot's weight using a simple math line (like "heavier robot = stronger muscles").
- Result: It worked okay for small robots, but failed miserably for big ones. The math formula suggested muscle strengths that were too aggressive, causing the robot to shake violently or break.
The "Total Random" Approach: They just picked random numbers for everything.
- Result: This was better, but sometimes the robot got a "bad roll" of the dice where the settings were impossible to learn.
The "Smart Coach" Approach (Their Winner): This is their new method. Imagine a coach who watches the student.
- If the student is struggling, the coach says, "Okay, let's make the weights slightly lighter so you can get the hang of it."
- If the student is doing great, the coach says, "Okay, let's make it a little harder to see how far you can go."
- They also used a technique called a Particle Filter. Think of this as keeping a list of the "best practice sessions." If a certain combination of weight and muscle strength worked well, the coach remembers it and tries variations of that specific setup, rather than starting from scratch every time.

The Big Discovery: "Muscle" Settings Matter Most

The most surprising finding was about the PD Gains. In robot language, this is basically the "stiffness" or "reflex" of the robot's joints.

Old methods tried to calculate these reflexes based on weight. The paper found this dangerous. It often told the robot to be so "stiff" that it would slam its joints into the ground, causing noise and potential damage.
Their method realized that to make a robot robust (tough), you need to train it with wildly different reflex settings. You need to teach it to walk with "jelly legs" and "steel legs" so that when it hits the real world, it doesn't care what its legs feel like.

The Real-World Test

They took their AI, which had only ever seen simulations (computer games), and put it on a real robot called ANYmal (which weighs 50kg).

The Result: The robot walked perfectly.
The "Zero-Shot" Magic: They didn't tell the robot, "Hey, you are now a 50kg robot." The robot had never seen a 50kg robot during training. It just knew how to walk because it had been trained on every possible variation of a robot.
The Bonus: They even put a 13kg backpack on the robot (making it heavier than the manufacturer's limit), and it still walked without falling over.

Summary

The paper claims that to build a universal robot walker, you can't just use a simple math formula to adjust the robot's "muscles." Instead, you need a smart, adaptive training system that constantly tweaks the difficulty and remembers what worked best. This allows a single AI brain to control a tiny robot, a giant robot, or a robot carrying a heavy load, all without needing to be retrained.

Technical Summary: Sampling Strategies for Robust Universal Quadrupedal Locomotion Policies

Problem Statement
The increasing diversity of commercial quadrupedal robots, ranging from lightweight units (e.g., 12 kg A1) to heavy-duty platforms (e.g., 50 kg ANYmal), presents a scalability challenge for locomotion control. Existing control designs are typically tailored to specific morphologies, requiring extensive retraining and manual parameter tuning when the robot's dynamics or structure change. While Reinforcement Learning (RL) has advanced quadrupedal agility, a significant barrier remains in achieving "universal" policies that generalize across diverse platforms without per-robot adaptation. A critical bottleneck is the "sim-to-real" gap: policies trained in simulation often fail on hardware, particularly when the training distribution does not adequately cover the physical variations (mass, inertia, kinematics) and controller gains (PD gains) of unseen robots. Current sampling strategies for domain randomization often rely on fixed uniform ranges or simple mass-dependent mappings, which may not provide sufficient robustness for larger, heavier platforms or may lead to unsafe controller gains during deployment.

Methodology
The authors propose a framework for training a universal RL locomotion policy that generalizes across a wide range of quadrupedal morphologies and controller configurations. The core of the methodology lies in the sampling strategy for robot configurations and joint Proportional-Derivative (PD) gains during the training phase.

Control Architecture: The system utilizes an asymmetric actor-critic architecture based on MorAL.
- Estimator (Red): A neural network estimates base linear velocity and observes robot-specific morphology parameters (mass, center of mass, link offsets) from a history of states.
- Actor (Blue): Generates desired joint positions based on estimated velocities, morphology parameters, and nominal joint positions. These positions are executed via a PD controller.
- Critic: Receives privileged information available only in simulation (e.g., foot contact states, friction, exact PD gains) to guide the policy.
Configuration Generation: The training set is generated by randomizing kinematic and dynamic properties of four reference models (Unitree A1, Aliengo, ANYmal B, ANYmal C). This includes sampling base center of mass, body masses, and leg geometries (knee orientation).
Sampling Strategies Compared: The paper investigates three fundamental approaches for sampling joint PD gains and morphology parameters:
- Mass-Dependent Mappings: Linear or polynomial functions mapping total robot mass to PD gains (similar to GenLoco and MorAL).
- Uniform Randomization: Sampling gains uniformly from a fixed range (similar to PAL and URMA).
- Performance-Based Adaptive Filtering (Proposed): A novel curriculum learning approach that dynamically adjusts the sampling range based on policy performance. This involves:
  - Adaptive Clipping: Adjusting the Sampling Range (SR) of uniform randomization based on the policy's ability to track velocity commands (linear and zero-velocity thresholds).
  - Particle Filter Resampling: Using a Sequential Importance Resampling (SIR) particle filter to weight and resample configurations. Configurations yielding mid-to-upper-range performance are prioritized to focus training on "challenging but solvable" parameter sets.
  - Replay Buffer: Sampling from a buffer of previously successful configurations to prevent parameter set degeneration.
Training: Policies are trained using Proximal Policy Optimization (PPO) in the RaiSim simulation environment. The reward function encourages velocity tracking, stability, and smoothness while penalizing collisions and excessive joint torques.

Key Contributions

Novel Morphology Sampling: Introduction of a particle-filter-based, performance-adaptive curriculum for sampling robot morphologies and control parameters, moving beyond fixed uniform randomization.
PD Gain Tuning Analysis: A comparative study demonstrating that significant randomization of joint controller gains is necessary for robust sim-to-real transfer, particularly for heavier platforms.
Zero-Shot Hardware Deployment: Successful deployment of a single policy on the 50 kg ANYmal robot without any fine-tuning or adaptation, covering a range of unseen morphologies (including a 13 kg payload).

Results

Simulation Robustness: The proposed adaptive particle filter strategy consistently achieved higher success rates ( $SR^*$ ) under perturbations (base force, friction changes, mass variations) compared to fixed uniform sampling and mass-dependent mappings. It demonstrated larger contiguous regions of stable performance across varying PD gain combinations, especially for the larger ANYmal platform.
Sim-to-Real Transfer:
- Success: The proposed method and the PAL baseline (uniform sampling with SR=1) were the only approaches that achieved safe, stable, and reliable walking on the ANYmal hardware.
- Failure of Alternatives: Mass-dependent mapping strategies (GenLoco, ManyQuadrupeds) and narrow-range sampling (URMA) failed to generalize to the hardware. GenLoco and ManyQuadrupeds required PD gains that exceeded hardware-safe limits, leading to actuator impacts and mechanical knocking. MorAL and URMA (adaptive) exhibited degraded performance or instability on hardware.
Performance Metrics: The adaptive particle filter method achieved Root Mean Square Error (RMSE) in velocity tracking comparable to the PAL baseline (e.g., ~0.095 m/s for heading, ~0.228 rad/s for yaw). It also successfully maintained locomotion with an additional 13 kg payload.

Significance and Claims
The paper claims that the primary barrier to universal quadrupedal locomotion is not just the policy architecture, but the sampling strategy used to expose the policy to diverse configurations during training. The authors demonstrate that:

Fixed uniform sampling or simple mass-dependent gain mappings are insufficient for scaling to heavier, more complex robots.
Significant randomization of joint controller gains is a prerequisite for closing the sim-to-real gap.
A performance-based adaptive curriculum that reshapes the distribution of training configurations (focusing on challenging but solvable regimes) significantly enhances robustness.
This approach enables zero-shot deployment on hardware platforms that were not explicitly seen during training, including heavy-duty robots operating beyond nominal specifications.

The work concludes that while zero-shot transfer to highly out-of-distribution morphologies remains challenging, performance-adaptive sampling substantially improves robustness within a broad and practically relevant range of quadrupedal configurations.