💻 computer science

Unified Motion Retargeting for Humanoids with Learned Point Cloud Correspondence

This paper presents Unified Motion Retargeting (UMR), a framework that learns dense point cloud correspondence to automatically and scalably transform diverse human motion data into high-fidelity robot trajectories without relying on manual human-robot mappings.

Original authors: Hanyang Cao, Yuetong Fang, Taesoo Kwon, Runyi Yu, Ji Ma, Jing Tan, Yangchen Zhou, Baoze Du, Yi Gu, Yukang Gao, Ruoli Dai, Lei Han, Renjing Xu

Published 2026-09-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanyang Cao, Yuetong Fang, Taesoo Kwon, Runyi Yu, Ji Ma, Jing Tan, Yangchen Zhou, Baoze Du, Yi Gu, Yukang Gao, Ruoli Dai, Lei Han, Renjing Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Humanoid robots are designed to move through our world with the same fluidity and adaptability that humans possess. To teach them how to walk, pick up objects, or navigate stairs, engineers often look to the vast library of human movement for inspiration. However, simply copying a human's motion onto a robot is not as straightforward as pressing a play button. Humans and robots are built differently; they have different body shapes, different numbers of joints, and different limits on how far those joints can bend. If a robot tries to mimic a human pose exactly, it might twist itself into an impossible position or lose its balance. For years, the solution has been to manually map specific human joints to specific robot joints, creating a custom instruction set for every new robot design. This process is slow, requires expert knowledge, and often fails to capture the subtle details of how a human body interacts with its environment.

A team of researchers has developed a new approach called Unified Motion Retargeting, or UMR, which changes how this translation happens. Instead of relying on a manual map of joints, the system treats the human body and the robot as two surfaces covered in millions of tiny points. Imagine the skin of a person and the skin of a robot as two clouds of dots. The researchers taught a computer to learn how these two clouds of dots correspond to each other, finding a match for every point on the human surface to a point on the robot surface. This learning happens once, in a neutral standing pose, and the resulting connection is saved. When the human moves, the system uses this pre-learned map to guide the robot's surface points to follow the human's surface points, rather than trying to match specific bones or joints. This allows the robot to reproduce complex poses and, crucially, to understand exactly where and how to touch objects or the ground, just as the human did.

The power of this method lies in its ability to handle differences in body shape without needing a new manual setup for each robot. In their tests, the researchers applied this system to a wide variety of human motion sources, including data from motion capture suits, scanned human meshes, and computer-generated animations. They then retargeted these movements onto five different humanoid robots, which varied significantly in height and body proportions. The system successfully transferred the motion across all these combinations, proving that the learned point-to-point map works regardless of the specific robot being used. This suggests that the method can scale to any new robot design without the need for engineers to spend weeks redefining how the robot's body parts relate to a human's.

Beyond just moving the body, the system also preserves the details of physical contact. When a human kicks a ball or climbs a staircase, specific parts of their body touch specific parts of the environment. Older methods often struggled to transfer these contact points accurately, leading to robots that might miss a step or fail to grasp an object correctly. Because the new system tracks the entire surface, it can transfer the contact map directly. If a human's hand touches a ball at a specific angle, the robot's hand is guided to touch the ball in the same way. In experiments involving tasks like carrying objects, kicking, and climbing stairs, the robots trained with this new method performed significantly better than those trained with previous techniques. They were more successful at completing the tasks and made fewer errors in their joint movements.

The researchers also tested the system in a real-world setting, deploying the learned behaviors on a physical robot. The robot was able to perform high-dynamic movements, such as a spinning kick, and navigate complex environments like stairs and slopes. The system handled these challenges with a level of stability and accuracy that matched or exceeded the best existing methods, even when the robot was operating in a simulated environment with random variations to test its robustness. The results indicate that by focusing on the surface geometry rather than the underlying skeleton, the system creates a more natural and reliable bridge between human motion and robot action. This approach offers a scalable path forward, allowing engineers to use large datasets of human movement to train a wide variety of robots without being limited by the specific design of the machine.

Technical Summary: Unified Motion Retargeting for Humanoids with Learned Point Cloud Correspondence

Problem Statement

Humanoid robots require large-scale, high-quality motion data to learn complex whole-body behaviors such as walking, manipulation, and physical interaction. While human motion data offers a scalable corpus, directly using it as reference trajectories for robots is infeasible due to significant morphological differences, including body proportions, joint topology, degrees of freedom (DoF), and kinematic constraints.

Existing retargeting methods are predominantly skeleton-centric. They rely on manually defined, one-to-one mappings between specific human joints and robot links. This approach presents several limitations:

Lack of Scalability: Switching to a new robot embodiment requires redefining skeletal correspondences and retuning fitting parameters.
Sparse Guidance: Joint-level representations constrain motion only at discrete points, providing insufficient geometric information for fine-grained pose alignment and reliable contact transfer.
Manual Dependency: The quality of retargeting heavily depends on manual semantic design, limiting the ability to generalize across diverse motion sources and robot morphologies.

While mesh-based representations offer denser geometric information, current methods often struggle to establish consistent surface correspondence across different embodiments without predefined skeletal maps.

Methodology: Unified Motion Retargeting (UMR)

The authors propose Unified Motion Retargeting (UMR), a framework that decouples retargeting from source-specific skeletal semantics and robot-specific topology by treating exterior point clouds as a unified interface. UMR operates in two primary stages:

1. Point Cloud Correspondence Learning

This stage establishes a reusable, dense, and ordered correspondence between the source human surface and the target robot surface.

Input: A canonical T-pose for both the human source (e.g., SMPL-X, SOMA, scanned meshes) and the target robot.
Process:
- The human point cloud ( $X_h$ ) is ordered, providing fixed indices.
- The robot point cloud ( $X_r$ ) is unordered.
- A PointNet-style encoder ( $E_\theta$ ) processes $X_r$ to create a latent representation.
- An MLP decoder ( $D_\theta$ ) predicts a deformation vector for each indexed human point, reconstructing the corresponding robot point cloud ( $\hat{X}_r$ ).
- Training Objective: The model minimizes a composite loss ( $L_{corr}$ $L_{cor r}$ ) comprising:
  - Chamfer Distance ( $L_c$ ): Ensures the reconstructed points cover the target geometry.
  - Repulsion Term ( $L_r$ ): Prevents multiple points from collapsing into the same region.
  - Edge Smoothness ( $L_e$ ): Enforces coherent deformation vectors between connected points on a geodesic graph.
Outcome: A fixed set of paired points that persist as the bodies move, allowing robot points to inherit human body segment labels without manual mapping.

2. Correspondence-Guided Retargeting

This stage optimizes the robot's generalized coordinates ( $q_t$ ) frame-by-frame to match the source motion while satisfying constraints.

Optimization Objective: The system minimizes a stack of residuals:
- Pose Residual ( $r_p$ ): Matches both the position and local surface orientation of paired points.
- Contact Residual ( $r_c$ ): Transfers contact maps directly. Contact vectors are defined as the difference between a body point and its nearest point on the environment (or another body segment for self-contact).
Constraints: The optimization is formulated as a constrained Gauss-Newton Quadratic Programming (QP) problem. It enforces:
- Joint limits ( $q^-, q^+$ ).
- Floor clearance (preventing foot penetration).
- Trust-region radius for stability.
Mechanism: The learned point pairs serve as geometric anchors. As the source mesh deforms, the paired robot points provide fine-grained targets for the optimizer, enabling surface-level alignment and direct contact transfer without additional hand-designed associations.

Key Contributions

Unified Surface-Centric Framework: UMR learns dense source-robot correspondence from surface point clouds, eliminating the need for manually defined skeletal or body mappings. It supports heterogeneous motion representations (SMPL-X, SOMA, scanned meshes) and diverse robot embodiments.
Efficient Retargeting Pipeline: The framework utilizes a reusable correspondence setup shared across motions. It enables surface-level pose matching and direct contact-map transfer, decoupling the retargeting process from specific source semantics.
Systematic Downstream Evaluation: The authors provide a comprehensive evaluation across whole-body tracking, large-scale policy learning, and contact-rich interaction tasks, demonstrating the framework's versatility.

Experimental Results

Experiments were conducted primarily on the Unitree G1 humanoid robot using the LAFAN1 dataset for tracking and BONES-SEED for policy learning.

Motion Tracking: UMR outperformed state-of-the-art baselines (GMR and Unitree's proprietary retargeting) in tracking success rates, particularly for challenging motions like "Fall and GetUp" and "Fight." Under domain randomization and Sim2Sim settings, UMR maintained superior performance across all motion categories.
Tracking Accuracy: UMR achieved lower mean joint-angle errors and competitive body-part position errors compared to baselines.
Large-Scale Policy Learning: In the SONIC framework, policies trained on UMR-retargeted data showed consistent improvements in total reward and tracking errors compared to those trained on GMR-retargeted references, even without the auxiliary SMPL encoder.
Contact-Rich Interactions: In tasks involving robot-object (Carry, Kick, Push) and robot-scene (Stair, Slope) interactions, UMR significantly outperformed OmniRetarget.
- UMR preserved intended contact geometries (e.g., hand-object contact), whereas OmniRetarget often shifted contacts to incorrect surface regions.
- UMR-trained policies reduced joint tracking errors by 40–56% in object interaction tasks.
- UMR demonstrated robustness with noisy human motion sources (reconstructed from videos), outperforming baselines on stair traversal and slope tasks.
Real-World Deployment: High-fidelity references generated by UMR enabled successful policy training and deployment on physical robots for high-dynamic motions (spin kicks) and diverse interactions (ball pickup, stair traversal).

Significance and Claims

The paper claims that UMR provides a scalable foundation for transforming large-scale, heterogeneous human motion references into robot-ready training data. By shifting from sparse, manually designed skeletal correspondences to learned dense surface correspondences, UMR:

Decouples retargeting from specific source semantics and robot topologies, allowing a single framework to handle diverse embodiments.
Enables fine-grained geometric alignment and direct transfer of interaction contacts, which are critical for complex manipulation and scene interaction.
Demonstrates that dense surface correspondence offers an effective interface for bridging the morphology gap between humans and humanoids, leading to higher motion fidelity and plausibility than existing state-of-the-art methods.

The authors acknowledge current limitations, specifically the assumption of access to mesh-based source geometry and a canonical template, and identify future work in extending the framework to unstructured observations and more articulated embodiments like dexterous hands.