🤖 machine learning

Weave: Learning Whole-Body Dexterous Loco-Manipulation from Human-Object Interactions

Weave is a unified framework that learns whole-body dexterous loco-manipulation for humanoid robots from human demonstrations by converting them into executable references via contact-aware retargeting and employing a geometry-aware policy to achieve high success rates in both trained and unseen interaction scenarios.

Original authors: Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, Mengdi Xu

Published 2026-09-16

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Liu Cao, Xingze Wu, Jingzhi Cui, Botian Xu, Mingzhi Pei, Ruoqu Chen, Mengdi Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To build a robot that can move through a human world, engineers face a problem of coordination that is far more complex than simply teaching a machine to walk or to pick things up. A robot must do both at once, often while holding an object that shifts its weight and changes how it balances. This is known as loco-manipulation, a term that describes the difficult task of moving the body while simultaneously manipulating an object with the hands. For a robot to succeed, it cannot treat walking and grasping as separate jobs; if it reaches for a chair without adjusting its feet, it will topple. If it walks toward a table without planning how its fingers will touch the surface, it will fail to lift it. The challenge lies in teaching a machine to understand that its entire body, from its feet to its fingertips, must work as a single, balanced unit to interact with the physical world.

Researchers have long tried to teach robots by showing them human movements, a process called imitation learning. However, copying a human directly often fails because robots and people have different body shapes and different ways of moving their joints. A human can twist their wrist in a way a robot cannot, or a robot's fingers might be too stiff to mimic a human's soft touch. Furthermore, most previous attempts to teach robots to handle objects focused on either the big movements of the body or the fine movements of the hands, but rarely both together in a single, unified system. This gap meant that while robots could learn to walk or learn to hold a cup, they struggled to do the complex, everyday task of walking up to a heavy object, grabbing it securely, and carrying it somewhere else without dropping it or falling over.

A team of researchers has introduced a new system called WEAVE to solve this specific problem. Their approach starts by recording real humans interacting with everyday objects like chairs, boxes, and tables. These recordings capture the full motion of the person as they approach, grab, and move the item. The researchers then take these human movements and translate them into a format that a robot can understand and execute. This translation is not a simple copy-paste; it is a careful reconstruction that respects the robot's physical limits. The system first fills in the missing parts of the motion, such as the steps the robot needs to take to get close to the object, which were not present in the original human recording. Then, it adjusts the human's hand movements to fit the robot's specific fingers, ensuring that the robot's grip is physically possible and strong enough to hold the object. Crucially, the system pays close attention to exactly where the fingers touch the object, preserving the contact points that make the grasp stable.

Once these reference movements are created, the researchers train a single computer program, or policy, to learn from them. This program is designed to control every joint in the robot's body, including the twenty-nine joints in its torso and legs and the twelve joints in its two hands. Instead of training a separate program for each type of object, the researchers trained this one program on a wide variety of items and interaction styles at the same time. The robot learns in a simulated environment, a digital world where it can practice thousands of times without breaking anything. It learns to watch the reference motion and try to match it, adjusting its balance and finger pressure in real time to keep the object secure. The training is rigorous, involving random changes to the robot's friction and weight to ensure it can handle unexpected conditions.

The results of this training are impressive. When tested on the specific objects and movements it had seen during training, the robot succeeded in completing the task 92.5% of the time. More importantly, the system showed a strong ability to generalize. When the researchers tested the robot on sequences of movements it had never seen before, using the same objects but approached from different angles or with different motions, it still succeeded 65.0% of the time without any additional training. This suggests that the robot learned a general strategy for interacting with objects rather than just memorizing specific movements. The researchers also released a large dataset of these successful movements, totaling about 23 hours of recorded robot activity, to help other scientists study how robots can learn to interact with the physical world.

The study highlights that learning to handle objects is most effective when the robot learns to coordinate its whole body at once. By training on many different objects together, the robot discovered common patterns, such as how to balance its weight when lifting a heavy box or how to position its feet when reaching for a tall lamp. This unified approach proved more successful than training separate specialists for each object. The researchers found that while a specialist might be slightly better at mimicking the exact pose for one specific item, the generalist robot was much better at actually completing the task across a wide range of situations. This indicates that the ability to adapt to new shapes and situations is more valuable than perfect imitation of a single movement.

Despite these successes, the researchers are clear about the limits of their work. The entire study was conducted in a computer simulation, meaning the robot never physically touched the real world. The system relied on perfect knowledge of where the robot and the object were, which is not always available in the real world where sensors can be noisy. Additionally, the robot used in the simulation has hands that are not fully flexible, limiting the complexity of the grasps it can perform. The researchers also note that their system requires a pre-planned sequence of actions to follow; it does not yet have the ability to decide on its own what to pick up or where to take it. These are the next steps for the field, where future systems will need to combine this physical skill with the ability to understand complex instructions and navigate the unpredictable nature of the real world.

Technical Summary: WEAVE

Problem Statement

Learning whole-body dexterous loco-manipulation for humanoid robots is a high-dimensional, heterogeneous control challenge. It requires coordinating balance, locomotion, and dexterous hand contacts to manipulate objects while maintaining stability. Existing approaches face two primary limitations:

Contact Modeling Gap: Reference motion tracking methods often supervise body and arm motion but leave finger-level hand-object contact unmodeled. Conversely, dexterous interaction learning often targets simulated characters or fixed-base hands with kinematics and actuation that differ significantly from underactuated humanoid hands.
Generalization vs. Specialization: Differences in object geometry and interaction sequences (approach direction, posture) require distinct control behaviors. Prior methods typically train separate policies per object or per clip, failing to learn a shared control strategy that adapts to specific objects while capturing common coordination patterns.

The core challenge is to learn a unified policy that tracks robot and object motion, explicitly learns hand-object contact, and maintains whole-body balance under the robot's dynamics, all derived from human demonstrations.

Methodology: WEAVE Framework

WEAVE is an end-to-end framework for learning whole-body dexterous loco-manipulation from captured human-object interactions. It operates in two main stages:

1. Interaction-Preserving Reference Construction

The system converts captured human-object interaction data (SMPL-X motion and object trajectories) into executable robot-object reference trajectories through:

Whole-Body Retargeting: Uses inverse kinematics to align robot links with human counterparts, constraining the pelvis in the horizontal plane and optimizing height based on ground contact to handle morphological differences.
Contact-Aware Hand Refinement: Jointly refines arm and finger configurations to ensure stable grasps. This minimizes a loss function combining:
- Contact Attraction: Aligning fingertips with object surfaces.
- Force-Closure Objective: Maximizing a differentiable approximation of force-closure quality ( $Q_{FC}$ ) to ensure the grasp can resist arbitrary disturbance wrenches.
- Penetration Penalty: Preventing fingers from penetrating the object.
Approach-Motion Completion: Uses the Kimodo model to synthesize locomotion prefixes that approach the initial interaction pose from various azimuths, creating complete trajectories spanning approach, grasping, and transport.
Annotation: The resulting trajectories include contact labels ( $\{-1, 0, 1\}$ ) indicating separated, neutral, and contact states for each robot link.

2. Contact- and Geometry-Aware Policy Learning

A unified policy is trained in simulation using Proximal Policy Optimization (PPO) to track the constructed references across multiple objects and interaction clips.

Observations: The policy receives robot proprioception (joint positions/velocities, base angular velocity), object state (relative pose, contact flags, BPS-SDF geometry descriptor), and a short-horizon reference command.
Architecture: An asymmetric actor-critic setup where the critic uses privileged information (e.g., base linear velocity). The network utilizes SimBaV2 layers.
Reward Function: The reward combines:
- Tracking: Pelvis, body, and object position/orientation errors.
- Grasp Quality: Hand opposition (thumb vs. opposing fingers) and contact matching (aligning actual contact forces with reference labels).
- Regularization: Penalties for foot sliding, action rate, and joint limit violations.
Training Strategy: The policy is trained jointly on a dataset of 9 objects (7,869 training trajectories) with domain randomization on friction, mass, and actuator properties.

Key Contributions

Unified Contact-Aware Policy: A single policy trained jointly across diverse objects and interaction clips that coordinates locomotion, whole-body balance, and finger-level grasping, explicitly learning contact patterns rather than just body motion.
Interaction-Preserving Pipeline: A method to convert human demonstrations into robot references that preserves task-relevant hand-object contacts and object motion across the morphology gap, augmented by approach-motion completion.
Systematic Evaluation and Dataset: Extensive experiments quantifying the benefits of joint multi-object learning and the release of a curated dataset of simulation rollouts (~9,000 rollouts, ~23 hours) with contact annotations for downstream research.

Experimental Results

The framework was evaluated on a Unitree G1 humanoid with 29 body DoFs and two Inspire hands (12 actuated finger DoFs).

Performance on Training Data: The unified policy achieved a 92.5% success rate on trained interactions (D_train) and a 96.3% progress rate, with tracking errors decreasing significantly over 100k iterations.
Generalization: Without additional training, the policy achieved a 65.0% success rate on unseen interaction sequences (D_test) involving the same objects but different approach directions and locomotion prefixes.
Joint vs. Specialist Training: A unified policy trained on all 9 objects outperformed a collection of 9 single-object specialist policies (95.3% vs. 91.5% pooled success rate on D_test). Joint training improved performance on 6 of the 9 objects and matched specialists on the other 3, demonstrating that shared whole-body structures and contact patterns act as mutual augmentation.
Optimization: The use of the Muon optimizer significantly improved sample efficiency compared to AdamW, attributed to its orthogonalized momentum updates which prevent gradient dominance by body-tracking terms over contact terms.

Significance and Limitations

Significance:
WEAVE demonstrates that a unified policy can learn complex, contact-rich loco-manipulation skills from human demonstrations by explicitly modeling hand-object contacts and object geometry. It bridges the gap between human demonstration and physically feasible robot behavior, providing a reusable low-level capability for future humanoid systems. The released dataset offers a resource for training interaction policies and modeling physically consistent human-object interactions.

Limitations (as stated by the authors):

Simulation-Only: Evaluation is conducted entirely in simulation using ground-truth odometry and object pose. The released dataset consists of simulation rollouts. Real-world deployment requires state estimation or distillation to perception-based inputs, and the sim-to-real gap for contact-rich interactions remains unaddressed.
Embodiment Constraints: The policy is limited by the underactuated nature of the Inspire hands (controlling only proximal joints) and cannot yet handle heavy or highly articulated objects outside the scope of the reference data.
Reference Conditioning: The policy executes interactions supplied to it; it does not autonomously select or compose tasks. Integrating this with high-level planners (e.g., Humanoid VLA systems) is identified as future work.