OmniRetarget: Interaction-Preserving Data Generation for Humanoid Whole-Body Loco-Manipulation and Scene Interaction

OmniRetarget is an interaction-preserving data generation engine that leverages an interaction mesh to overcome embodiment gaps and preserve critical human-object/environment relationships, enabling the efficient creation of high-quality training data for robust, long-horizon humanoid loco-manipulation and parkour skills without complex learning curricula.

The Gist

Imagine you want to teach a robot to do parkour, carry a heavy chair up a hill, or climb a wall. You could try to program every single muscle movement by hand, but that's like trying to teach someone to swim by writing a physics textbook about water resistance. It's too hard and too slow.

Instead, the researchers behind OmniRetarget decided to let the robot learn by watching a human do the moves. But here's the catch: Humans and robots look very different. If you just copy a human's movements onto a robot, the robot might end up with its feet sliding across the floor (like ice skating), its legs passing through a chair, or its joints twisting in impossible ways.

OmniRetarget is a new "magic translator" that fixes this problem. Here is how it works, using simple analogies:

1. The "Interaction Mesh" (The Invisible Spiderweb)

Think of the human, the robot, the chair, and the ground as being connected by an invisible, stretchy spiderweb.

• The Problem: When a human moves, the web stretches naturally. If you just copy the human's pose onto a robot, the web snaps or tears because the robot's body parts are in different places.
• The Solution: OmniRetarget builds a digital "interaction mesh" (a 3D web) that connects the human's joints to the objects they touch. When converting the human motion to the robot, the system stretches and shrinks this web to fit the robot's body, but it keeps the web intact.
• The Result: If the human's hand is touching a chair, the robot's hand must touch the chair in the new version. If the human's foot is planted firmly on the ground, the robot's foot stays planted. It prevents the robot from "ghosting" through objects or slipping.

2. The "Strict Coach" (Hard Constraints)

In the past, these translation systems were like a coach who said, "Try to look like the human, but it's okay if you fall through the floor a little bit."
OmniRetarget is a strict coach. It uses a set of unbreakable rules (mathematical constraints) that say:

• "Your feet cannot slide."
• "Your body cannot pass through the chair."
• "Your joints cannot bend backward."
  It solves a complex puzzle to find a way for the robot to move that looks like the human but obeys all the laws of physics.

3. The "One-to-Many" Photocopier (Data Augmentation)

Usually, to teach a robot to pick up a red box, you need a human to demonstrate picking up a red box. To teach it to pick up a blue box, you need a new demonstration.
OmniRetarget is like a smart photocopier.

• You show it one human demonstration (e.g., picking up a box).
• The system automatically generates hundreds of new variations: picking up a tall box, a short box, a box in a different spot, or even climbing a higher platform.
• It does this by mathematically reshaping the "spiderweb" to fit these new scenarios while keeping the core movement logic intact. This creates a massive library of training data from just a few human videos.

The Big Win: From Simulation to Reality

The researchers used this system to train a robot (a Unitree G1 humanoid) using Reinforcement Learning (a trial-and-error learning method).

• The Training: Because the data was so clean and physically correct, the robot only needed 5 simple rules (rewards) to learn. It didn't need complex, messy instructions to fix mistakes.
• The Result: The robot learned to do a 30-second parkour course that included carrying a chair, stepping on it, jumping off, and rolling.
• Zero-Shot Transfer: This is the most impressive part. The robot learned entirely in a computer simulation using these translated motions. When they turned it on in the real world, it worked immediately without any extra tuning. It didn't need to "re-learn" how to walk on real concrete; it just knew what to do.

Summary

In short, OmniRetarget solves the "embodiment gap" between humans and robots. It takes human movements, wraps them in a protective, physics-compliant "web" that ensures they make sense for a robot, and then uses that to generate endless practice scenarios. This allows robots to learn complex, agile skills like parkour and object manipulation quickly and successfully transfer those skills from the computer to the real world.

Technical Summary

Technical Summary: OmniRetarget

Problem Statement

The development of complex whole-body loco-manipulation skills for humanoid robots is currently constrained by a fundamental data bottleneck. While deep reinforcement learning (RL) has shown promise in robot control, training policies from scratch is inefficient due to high-dimensional action spaces and the sensitivity of exploration to reward engineering. Existing approaches that attempt to bridge this gap by retargeting human motions to robots face two primary challenges:

1. The Embodiment Gap: Humanoids differ significantly from humans in shape, proportion, and degrees of freedom. Simple kinematic adaptation often results in physically implausible artifacts, such as foot-skating, body penetration, and loss of contact stability.
2. Neglect of Interactions: Common retargeting pipelines focus on keypoint matching but fail to explicitly preserve rich human-object and human-environment interactions. This leads to low-quality reference trajectories that complicate downstream RL training, often requiring extensive, ad-hoc reward engineering to compensate for artifacts.

Current strategies, such as teleoperation, are labor-intensive and do not scale, while existing offline retargeting methods often rely on unconstrained or softly-penalized optimization, failing to guarantee physical feasibility or interaction preservation.

Methodology

The authors introduce OmniRetarget, an open-source data generation engine designed to transform human demonstrations into diverse, high-quality kinematic references for humanoid control. The core of the methodology is an interaction-preserving motion retargeting framework based on an interaction mesh.

1. Interaction Mesh with Hard Constraints

OmniRetarget models the spatial and contact relationships between the agent, manipulated objects, and the terrain using a volumetric interaction mesh. This mesh consists of key joints (human or robot) and points sampled from objects and the environment.

• Optimization Objective: The system minimizes the Laplacian deformation energy between the source (human) mesh and the target (robot) mesh. This preserves the relative spatial configurations and contact relationships of the original demonstration.
• Hard Constraints: Unlike prior methods that use soft penalties, OmniRetarget enforces strict physical feasibility through a constrained optimization problem solved via Sequential Quadratic Programming (SQP). Constraints include:
  • Collision avoidance (non-penetration).
  • Joint limits and velocity bounds.
  • Foot sticking (preventing foot-skating by enforcing zero horizontal velocity for stance feet).
  • Temporal smoothness.

2. Systematic Data Augmentation

A key feature of the framework is its ability to generate a large-scale dataset from a single human demonstration. By parametrically altering object configurations, shapes, and terrain features, the system re-solves the optimization problem to produce diverse, kinematically valid trajectories.

• Robot-Object Augmentation: The system modifies object poses (translation/rotation) and scales object dimensions. To prevent trivial rigid transformations, it anchors specific body parts (e.g., the lower body) to the nominal trajectory while allowing the upper body to discover new coordination strategies.
• Robot-Terrain Augmentation: Environmental features, such as platform height and depth, are scaled, with additional constraints added to ensure stable ground contact.

3. Minimal RL Training Formulation

The high quality of the generated references allows for a minimal RL training setup.

• Observations: The policy uses a purely proprioceptive observation space (joint positions/velocities, pelvis state) and reference motion data, without explicit scene or object information.
• Rewards: The training requires only five reward terms: body tracking, object tracking (where applicable), action rate, soft joint limits, and self-collision penalties. No complex reward engineering or curriculum learning is needed.
• Domain Randomization: To improve generalization, the system randomizes object physical parameters (mass, inertia, shape) and robot parameters (torso COM, joint defaults, random pushes) during training.

Key Contributions

The paper outlines four primary contributions:

1. Interaction-Preserving Framework: The first humanoid retargeting framework that explicitly handles rich robot-object-terrain interactions while enforcing hard physical constraints (collision avoidance, joint limits, foot stability).
2. Systematic Data Augmentation Pipeline: A method to transform a single human demonstration into a large-scale, diverse set of high-quality kinematic trajectories across various robot embodiments, object shapes, and environments.
3. Large-Scale Open-Source Dataset: The release of a dataset containing over 8 hours of retargeted, kinematically-feasible loco-manipulation trajectories derived from OMOMO, LAFAN1, and in-house MoCap datasets.
4. Zero-Shot Sim-to-Real Transfer: Successful deployment of proprioceptive RL policies on a physical Unitree G1 humanoid, demonstrating complex, long-horizon skills (up to 30 seconds) including parkour, object carrying, and wall-flipping without manual reward tuning.

Experimental Results

The authors evaluated OmniRetarget against state-of-the-art baselines (PHC, GMR, VideoMimic) across kinematic quality and downstream RL performance.

• Kinematic Quality: OmniRetarget significantly outperformed baselines in minimizing penetration and foot-skating while maximizing contact preservation. For instance, in robot-object interaction tasks, OmniRetarget achieved 0.00% penetration duration and 0% foot-skating duration, compared to significant violations in baselines.
• Downstream RL Performance: Policies trained on OmniRetarget data achieved superior success rates in simulation.
  • Robot-Object Tasks: 82.20% success rate (vs. 71.28% for PHC and 50.83% for GMR).
  • Robot-Terrain Tasks: 94.73% success rate (vs. 52.63% for PHC and 78.94% for GMR).
• Sim-to-Real Transfer: The trained policies successfully transferred to the Unitree G1 humanoid in the real world. Demonstrated behaviors included:
  • A 30-second parkour sequence involving chair moving, stepping, vaulting, jumping, and rolling.
  • Object transportation and crawling on slopes.
  • High-dynamic wall-flipping (reaching 15 rad/s angular velocity).
  • Platform climbing and sitting.

Significance and Claims

The paper claims that OmniRetarget represents a paradigm shift in humanoid learning. Instead of attempting to patch low-quality reference motions with complex, fragile reward engineering, the authors argue for solving the problem at the source through principled, interaction-preserving data generation.

The significance of this work lies in its ability to:

• Decouple Data Generation from Reward Engineering: By providing artifact-free references, the need for extensive reward tuning is eliminated, allowing for minimal, unified reward formulations across diverse tasks.
• Enable Scalable Data Generation: The augmentation pipeline allows for the creation of massive, diverse training datasets from minimal human demonstrations, addressing the data scarcity bottleneck in whole-body control.
• Facilitate Robust Sim-to-Real Transfer: The high fidelity of the kinematic references enables proprioceptive policies to learn complex, dynamic skills that transfer zero-shot to physical hardware, demonstrating that natural, human-like behaviors can be achieved without manual post-processing or task-specific reward design.

The authors conclude that while their current frame-by-frame optimization is effective, future work could explore joint trajectory optimization to handle noisier sources (like video) and further accelerate progress toward versatile humanoid robots. All code, datasets, and trained policies are released publicly.