OmniTrack: General Motion Tracking via Physics-Consistent Reference

1. Problem Statement

The core challenge addressed by OmniTrack is the "embodiment gap" between human demonstrators and humanoid robots. While learning motion tracking from large-scale human motion data (e.g., LAFAN1, AMASS) is a dominant paradigm for humanoid control, direct application of this data leads to significant issues:

• Physical Infeasibility: Due to differences in kinematics, body proportions, and degrees of freedom, retargeted human motions often contain physically impossible artifacts such as floating (feet not touching the ground), penetration (limbs passing through the ground), and foot skating.
• Training-Execution Conflict: Standard reinforcement learning (RL) policies must simultaneously learn to track these inaccurate references and maintain physical stability. This creates an intrinsic conflict where the robot is forced to compensate for impossible targets, leading to dynamic failures, instability, and a lack of generalization.
• Partial Observability: In real-world deployment, robots lack "privileged information" (e.g., global velocity, exact contact states) required to judge the feasibility of a reference motion, causing them to blindly pursue infeasible tracking rewards.

2. Methodology: The Two-Stage Framework

OmniTrack proposes a novel two-stage learning pipeline that explicitly decouples physical feasibility from general motion tracking. This allows a single policy to execute diverse, complex behaviors while remaining stable.

Stage I: Physical Motion Generation (Simulation)

• Goal: Convert raw, physically inconsistent reference motions into physically plausible trajectories.
• Mechanism: A "generalist" policy is trained in a physics simulator with full privileged information (global pose, velocity, contact states, etc.).
• Process:
  • The policy performs trajectory rollouts to reconstruct the reference motion while strictly adhering to the robot's dynamics and actuator limits.
  • It acts as a motion filter, removing artifacts like floating and penetration.
  • Key Design: No domain randomization is used here to maximize reconstruction fidelity. Instead, noise is injected into motion commands to ensure robustness across diverse sources.
  • Output: A dataset of high-quality, dynamics-consistent reference trajectories (including joint states, global poses, and contact masks) that serve as the ground truth for the next stage.

Stage II: General Motion Tracking (Real-World Deployment)

• Goal: Train a general control policy to track the physically feasible trajectories generated in Stage I.
• Mechanism: The policy is trained under partial observability, mimicking real-world conditions (using only onboard sensors like joint encoders, IMUs, and previous actions).
• Process:
  • The policy learns to track the refined references without needing to compensate for physical impossibilities, as the references are already dynamically valid.
  • Domain Randomization: Extensive randomization (friction, mass, IMU offsets, external pushes) is applied to ensure robust Sim-to-Real transfer.
  • Reward Design: Includes tracking rewards, action smoothness penalties scaled by velocity, and desired-contact rewards to ensure coherent interaction with the environment.

3. Key Contributions

1. Decoupled Framework: The first framework to explicitly separate physical feasibility generation from general motion tracking, enabling a single policy to handle hour-long, diverse motion sequences without instability.
2. Physically Consistent Dataset: The construction of a high-quality benchmark dataset where dynamic artifacts are removed from raw human motion data, providing a standardized foundation for general humanoid control.
3. Robust Sim-to-Real Transfer: Demonstration of zero-shot transfer to the Unitree G1 humanoid robot, achieving stable execution of highly dynamic acrobatic skills (flips, cartwheels) and robust online teleoperation.

4. Experimental Results

The authors validated OmniTrack on the Unitree-retargeted LAFAN1 and AMASS datasets, as well as on real hardware.

• Motion Quality: The physical motion generation stage eliminated 100% of penetration and floating artifacts found in raw retargeted data while maintaining stylistic fidelity (only a moderate increase in MPJPE due to necessary dynamic corrections).
• Training Stability & Scalability:
  • Training on raw motions led to performance degradation as dataset size increased (success rate dropped from 95.1% to 88.2%).
  • Training on physically plausible motions maintained high stability (success rate ~92.6%) and high fidelity even with the full dataset.
• Tracking Performance (Simulation):
  • OmniTrack achieved a 96.13% success rate on seen motions and 96.88% on unseen motions, significantly outperforming baselines like Dagger, OmniH2O, and BeyondMimic.
  • On high-dynamic subsets (e.g., flips, cartwheels), OmniTrack achieved an 84.81% success rate, whereas baselines dropped to ~48-70%.
• Real-World Deployment (Unitree G1):
  • Long-Horizon Stability: The robot successfully tracked diverse motions (walking, dancing, martial arts) for over one hour continuously.
  • Dynamic Skills: Successfully executed 17 consecutive side flips and complex acrobatic maneuvers without failure.
  • Teleoperation: Demonstrated robust online teleoperation using both full-body motion capture and VR headsets. The system successfully filtered noise, jitter, and latency from user inputs, producing smooth, human-like motions in real-time.
  • Sim-to-Real Alignment: Tracking errors in joint position, orientation, and angular velocity were comparable between MuJoCo simulation and the real robot.

5. Significance

OmniTrack represents a paradigm shift in humanoid control by addressing the root cause of instability in motion tracking: infeasible references. By moving the burden of physical consistency to a pre-processing generation stage, the control policy is freed to focus purely on tracking and adaptation.

• Scalability: It enables the use of massive, diverse human motion datasets without the risk of training collapse due to physical artifacts.
• Generalization: It allows a single policy to master a wide spectrum of behaviors, from static balance to high-impact acrobatics, bridging the gap between everyday movements and complex interactions.
• Practicality: The framework supports real-time, human-style teleoperation, making it a viable solution for practical deployment of humanoid robots in unstructured environments.

In conclusion, OmniTrack demonstrates that explicitly resolving physical feasibility before policy learning is crucial for achieving scalable, general, and robust control in humanoid robots.