🤖 AI

ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Control

ZEST is a streamlined reinforcement learning framework that enables robust, zero-shot deployment of diverse, contact-rich athletic skills on heterogeneous humanoid and quadruped robots by training on heterogeneous data sources—including motion capture, monocular video, and animation—without requiring contact labels, state estimators, or extensive reward shaping.

Original authors: Jean Pierre Sleiman, He Li, Alphonsus Adu-Bredu, Robin Deits, Arun Kumar, Kevin Bergamin, Mohak Bhardwaj, Scott Biddlestone, Nicola Burger, Matthew A. Estrada, Francesco Iacobelli, Twan Koolen, Alexan

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jean Pierre Sleiman, He Li, Alphonsus Adu-Bredu, Robin Deits, Arun Kumar, Kevin Bergamin, Mohak Bhardwaj, Scott Biddlestone, Nicola Burger, Matthew A. Estrada, Francesco Iacobelli, Twan Koolen, Alexander Lambert, Erica Lin, M. Eva Mungai, Zach Nobles, Shane Rozen-Levy, Yuyao Shi, Jiashun Wang, Jakob Welner, Fangzhou Yu, Mike Zhang, Alfred Rizzi, Jessica Hodgins, Sylvain Bertrand, Yeuhi Abe, Scott Kuindersma, Farbod Farshidian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to dance, do a backflip, or climb a box. Traditionally, this has been like teaching a human to juggle by writing a 500-page manual for every single finger movement, then spending months tweaking the instructions until the robot doesn't fall over. It's slow, fragile, and requires a different manual for every new trick.

The paper introduces ZEST (Zero-shot Embodied Skill Transfer), which is like a "universal dance instructor" for robots. Instead of writing a manual for every move, ZEST lets the robot learn by watching videos, motion-capture recordings of humans, or even animated cartoons, and then figuring out how to do it itself.

Here is how it works, broken down into simple concepts:

1. The "Watch and Learn" Approach

Think of ZEST as a robot that has a superpower: it can learn from any video source.

High-Definition Motion Capture: Like watching a professional dancer on a stage with perfect cameras.
Shaky Phone Videos: Like watching a friend record a dance on their phone in a park (the video might be shaky or blurry).
Cartoons/Animation: Like watching a character in a video game do a move that a human body physically couldn't do (like spinning 360 degrees in mid-air).

ZEST takes all these messy, different sources and says, "Okay, I see the goal. I'll figure out the physics." It doesn't need a human to tell it exactly which foot touches the ground or when to push off. It learns that on its own.

2. The "Training Wheels" That Disappear

One of the hardest parts of teaching a robot to do something wild (like a backflip) is that it falls down a lot during practice. Usually, engineers have to manually design a "curriculum" (a step-by-step lesson plan) to help the robot.

ZEST uses a clever trick called an Automatic Assistive Wrench. Imagine a gymnastics coach holding a student's waist while they learn a backflip.

At first, the coach holds on tight to keep the robot from crashing.
As the robot gets better at the move, the coach lets go a little bit more.
Eventually, the coach lets go completely, and the robot does the flip on its own.

ZEST does this automatically. It calculates how hard a move is, gives the robot a virtual "push" to help it stay upright, and slowly removes that help as the robot gets smarter. No human has to write the lesson plan; the robot creates its own.

3. The "One-Size-Fits-All" Brain

Most robot controllers are like a custom suit tailored for one specific person. If you change the robot's size or shape, the suit doesn't fit.

ZEST is like a stretchy, magical hoodie.

It was tested on a giant, heavy robot (Boston Dynamics' Atlas, about the size of an adult human).
It was tested on a smaller, lighter robot (Unitree G1).
It was even tested on a four-legged dog robot (Spot).

The same "brain" (software) learned to make the giant human walk, the small human dance, and the dog do a backflip. It didn't need to be re-tuned for each robot; it just adapted.

4. The "Zero-Shot" Magic

The term "Zero-shot" is the most exciting part. Usually, if you want a robot to learn a new trick, you have to spend days or weeks training it specifically for that trick.

With ZEST, once the robot is trained, you can show it a video of a new move (like climbing a box), and it can do it immediately, without any extra training. It's like watching a video of someone climbing a ladder and then instantly being able to climb a ladder yourself without practicing.

What Did They Actually Achieve?

The paper shows that this system works on real hardware, not just in computer simulations. They successfully made robots do:

Complex Human Moves: Breakdancing, army crawling, cartwheels, and forward rolls.
Video-to-Robot: They took a video of a human dancing or climbing a box, fed it to the robot, and the robot did it perfectly on the first try.
Impossible Moves: They used animation to teach the four-legged Spot robot to do a continuous backflip, a move a real dog (or robot) couldn't naturally do without this specific training.

The Bottom Line

ZEST is a streamlined way to turn human movement (whether it's a perfect dance, a shaky phone video, or a cartoon) into robot skills. It removes the need for engineers to manually program every detail of how a robot should move, making robots much more agile, adaptable, and ready to learn new tricks instantly.

Technical Summary: ZEST (Zero-shot Embodied Skill Transfer for Athletic Robot Control)

Problem Statement

Achieving robust, human-like whole-body control on humanoid robots for agile, contact-rich behaviors remains a central challenge in robotics. Traditional approaches often rely on heavy per-skill engineering, brittle controller tuning, and complex pipelines involving offline trajectory optimization followed by online tracking. These methods typically demand precise environment modeling, contact labeling, state estimation, and extensive reward shaping. Furthermore, they struggle with high uncertainty, complex contact scenarios (e.g., multi-contact interactions beyond the feet), and the "sim-to-real" gap, where policies trained in simulation fail to transfer to hardware due to modeling discrepancies and actuation limits. While Reinforcement Learning (RL) has shown promise in agility, standard "tabula rasa" RL is sample-inefficient and sensitive to reward design, often requiring careful regularization to avoid unnatural behaviors.

Methodology: The ZEST Framework

ZEST (Zero-shot Embodied Skill Transfer) is a streamlined, unified motion-imitation framework designed to train policies via Reinforcement Learning from diverse, heterogeneous data sources and deploy them to hardware with zero-shot transfer. The framework eliminates the need for complex, domain-specific techniques such as contact labels, reference/observation windows, state estimators, and multi-stage training pipelines.

Core Components

Heterogeneous Data Sources: ZEST trains on a single-stage pipeline using three distinct types of motion references:
- Motion Capture (MoCap): High-fidelity data (e.g., Xsens, Vicon).
- Video-Captured (ViCap): Noisy, monocular video data reconstructed via 3D pose estimation (e.g., MegaSaM, TRAM).
- Keyframe Animation: Non-physics-constrained animations, enabling motions impossible for humans or tailored to specific morphologies.
- Note: The framework assumes no contact labeling for any source, relying solely on kinematic information.
MDP Formulation and Architecture:
- Goal-Conditioned MDP: The policy is trained to track a reference target $g_t$ (the next target state from the reference) using only instantaneous proprioceptive signals (joint positions/velocities, IMU data, gravity vector, previous action).
- Asymmetric Actor-Critic: The actor (policy) uses only onboard sensor data to ensure deployability without state estimators. The critic utilizes "privileged information" available only in simulation (e.g., true base velocity, contact forces, full root state) to accelerate learning and improve value estimation.
- Action Space: The policy outputs residual actions added to the reference joint positions. These are sent to a joint-level PD controller. This approach leverages the reference for feedforward control, reducing the exploration burden on the RL policy.
Training Enhancements:
- Adaptive Sampling: To handle long-horizon clips and large motion libraries, trajectories are split into fixed-duration bins. A categorical sampler biases training toward bins with higher failure rates (estimated via an Exponential Moving Average of failure scores), ensuring difficult segments are learned without catastrophic forgetting of easier ones.
- Automatic Assistive Wrench Curriculum: For highly dynamic behaviors (e.g., backflips, handstands) that cause immediate early termination, a model-based virtual assistive wrench is applied to the robot's base. Its magnitude is modulated by the difficulty of the current bin and automatically annealed to zero as tracking performance improves, acting as a "coach" that gradually reduces support.
- Actuator Modeling: To bridge the sim-to-real gap for closed-chain actuators (e.g., knees, ankles), ZEST employs progressive approximations of Parallel-Linkage Actuators (PLAs), including Locally Projected, Dynamic Armature, and Nominal Armature models. PD gains are derived from approximate analytical armature values to ensure consistency between simulation and hardware.
Domain Randomization: Training is performed entirely in simulation with moderate domain randomization (e.g., impulsive pushes, sensor noise, friction variations, mass perturbations) to ensure robustness.

Key Contributions

Unified, Minimalist Recipe: ZEST provides a generic, single-stage training pipeline that processes heterogeneous data (MoCap, ViCap, Animation) without contact labels, state estimators, or extensive reward shaping.
Zero-Shot Hardware Deployment: The framework successfully deploys policies to physical hardware without per-task fine-tuning, demonstrating robustness across diverse data imperfections (e.g., pose jitter in ViCap, physically infeasible animation).
Multi-Contact and Dynamic Skills: It achieves state-of-the-art performance in dynamic, multi-contact behaviors (e.g., army crawl, breakdancing, cartwheels) on full-size humanoids, moving beyond standard foot-only gaits.
Morphological Generality: The same policy structure and training recipe are successfully applied across distinct morphologies, including the Boston Dynamics Atlas (humanoid), Unitree G1 (humanoid), and Boston Dynamics Spot (quadruped).

Results

The paper evaluates ZEST on three robots: Boston Dynamics' Atlas, Unitree's G1, and Boston Dynamics' Spot.

MoCap-Derived Skills: On Atlas, ZEST executes complex multi-contact skills including forward rolls, rolls on all fours, army crawls, cartwheels, and breakdancing. On the G1, it successfully transfers cartwheels and table-tennis motions. The framework preserves human nuance (e.g., heel-toe rollover) and handles intermittent contacts without predefined schedules.
ViCap-Derived Skills: ZEST demonstrates robustness to noisy video data. On Atlas, it executes soccer kicks and dance snippets. On the G1, it performs athletic ballet sequences and scene-interactive skills (jumping onto, climbing up, and climbing down a box) using only proprioceptive inputs, without precise box location feedback. It achieves high success rates even with initial pose variations and payload additions.
Animation-Derived Skills: For Atlas, it performs handstand inversions. For Spot, it executes acrobatic maneuvers like continuous backflips, barrel rolls, and handstand balances, leveraging animation to achieve configurations impossible for human anatomy.
Quantitative Metrics: Tracking errors (MAE for joint angles, MAD for base orientation) remain low across skills. While dynamic multi-contact skills show higher orientation errors than simple gaits, they remain within feasible bounds.
Comparison with MPC: In simulation, a multi-skill RL policy trained with ZEST outperforms a state-of-the-art Model Predictive Control (MPC) baseline on dynamic behaviors (e.g., cartwheels, jogging) and handles motions involving non-foot contacts (e.g., rolls, crawls) that the MPC baseline fails to execute due to its reliance on predefined contact schedules and end-effectors.

Significance and Claims

The paper claims that ZEST establishes a scalable interface between biological movements and robotic counterparts. Its primary significance lies in:

Democratizing Complex Control: By removing the need for intricate, handcrafted pipelines (contact labeling, multi-stage training), ZEST makes it feasible to transfer a vast repertoire of skills from diverse data sources to robots with minimal engineering overhead.
Robustness to Real-World Uncertainty: The framework's ability to handle noisy ViCap data, physically infeasible animations, and complex multi-contact interactions demonstrates a level of robustness previously difficult to achieve with model-based controllers or standard RL.
Generality Across Embodiments: The successful transfer of skills across humanoids and quadrupeds with different sizes and actuation mechanisms highlights the framework's potential as a universal control interface for legged robots.

The authors acknowledge limitations, including the lack of evaluation on generalization to unseen skills outside the training distribution, the assumption of flat terrain, and the reliance on manual system identification for actuator modeling. However, the work positions ZEST as a significant step toward robust, human-level physical intelligence in legged robots.