💻 computer science

Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration

This paper introduces CHORD, a framework that leverages object-centric contact wrench guidance derived from human demonstrations to enable scalable reinforcement learning for long-horizon dexterous manipulation, achieving high success rates across a large-scale simulation benchmark and demonstrating robust generalization and real-world transfer.

Original authors: Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi F

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinghao Zhu, Zixi Liu, Shalin Jain, Chenran Li, Milad Noori, Huihua Zhao, John Welsh, Michael Andres Lin, Wei Liu, Tingwu Wang, Xingye Da, Zhengyi Luo, Vishal Kulkarni, Naema Bhatti, Yuke Zhu, Linxi Fan, Bowen Wen, Danfei Xu, Soha Pouya, Yan Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot with incredibly dexterous hands (like a human's) how to perform complex tasks, such as opening a tricky box, stirring a thick batter, or turning the pages of a book. You have a video of a human doing these tasks perfectly. The problem is that the robot's hands are shaped differently, have different joints, and touch objects in different spots than the human does. If you just tell the robot, "Copy the human's hand movements exactly," it will fail because its hands can't physically reach or move the same way.

This paper introduces a new method called CHORD (Contact Wrench Guidance from Human Demonstration) to solve this problem. Here is how it works, explained simply:

The Core Problem: "Copying the Wrong Thing"

Think of a human hand and a robot hand as two different musical instruments. If a human plays a piano, their fingers press specific keys. If you try to make a violin play the exact same finger movements, it won't make the right sound.

Previous methods tried to teach robots by matching the location of the human's fingers to the object. They said, "If the human's thumb touches the top-left corner of the box, your robot thumb must touch the top-left corner too."

The Flaw: Sometimes, touching the top-left corner with a robot's thumb pushes the box the wrong way, or slips off, because the robot's finger is shaped differently. The location is the same, but the effect is wrong.

The Solution: Focus on the "Push," Not the "Touch"

CHORD changes the question. Instead of asking, "Where did the human touch?", it asks, "What did that touch do to the object?"

The authors use a concept called Contact Wrench Space. Imagine every time a hand touches an object, it creates a specific "push and twist" force.

The Analogy: Think of a door. You can push the door handle (location A) or push the edge of the door (location B). These are different locations, but if you push them both in the right direction, they both make the door swing open.
CHORD's Trick: It ignores where the robot touches and focuses on the force and twist (the "wrench") that the touch creates. It tells the robot: "You don't need to touch the exact same spot as the human. You just need to create the same swinging motion on the door."

This allows the robot to find its own unique way to touch the object, as long as the result (the object moving correctly) matches the human's goal.

How They Taught the Robot (The "Gym")

To train the robot, the researchers didn't just use a few videos. They built a massive digital gym (a simulation) containing 4,739 different tasks.

These tasks range from simple things (picking up a cup) to complex, long sequences (opening a box, taking out a tool, and using it).
They used videos of humans doing these tasks as the "reference guide."
The robot practiced millions of times in this simulation, trying to match the "force and twist" of the human's actions rather than just copying their finger positions.

The Results: A Master of Many Trades

When they tested the robot on 1,831 different tasks:

Success Rate: The robot succeeded 82% of the time on average. This is a huge improvement over previous methods.
Versatility: It worked on rigid objects (like a hammer), articulated objects (like a door with a hinge), and even tasks requiring two hands working together.
Real World: They took the robot out of the computer and put it on a real robot. It worked in both "open-loop" (doing a pre-planned sequence) and "closed-loop" (reacting to the world as it happens).

Bonus: Teaching the Whole Body

The method is so flexible that it works even if you only show the robot videos of a human's hands (without showing their body). The robot can figure out how to move its whole body to get its hands into the right position to create the necessary forces. It even worked with videos taken from a third-person perspective (watching someone from across the room), even though the hand movements in those videos were a bit blurry.

Summary

In short, CHORD teaches robots to be creative problem solvers rather than mindless copycats. Instead of forcing the robot to mimic the human's exact finger placement, it teaches the robot to mimic the physics of the action. This allows the robot to use its own unique body to achieve the same result as a human, making it much better at handling complex, real-world objects.

Technical Summary: Learning Dexterous Manipulation Using Contact Wrench Guidance From Human Demonstration (CHORD)

Problem Statement

Transferring human demonstrations to robotic policies for dexterous manipulation remains a significant challenge. While human demonstrations offer a rich source of skill data, existing methods often struggle with scalability and transferability due to morphological differences between humans and robots. Simply replaying human hand motions is ineffective because differences in kinematics, geometry, and contact points require the robot to utilize different motions to achieve the same functional effect on an object.

Current approaches face two primary limitations:

Brittle Assumptions: Many optimization-based methods rely on rigid assumptions about how demonstrations should be transferred, often failing when the robot's morphology differs significantly from the human demonstrator.
Scalability Issues: Representation-learning approaches typically require aligned human-robot data for every specific task and object, limiting their application to curated settings rather than general, long-horizon tasks.

The core problem addressed is how to learn transferable and scalable task knowledge from human demonstrations that accounts for contact-rich, long-horizon manipulation of both rigid and articulated objects.

Methodology: CHORD Framework

The authors propose CHORD (Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation), a reinforcement learning (RL) framework that bridges the human-robot gap by focusing on the object-centric contact wrench space rather than raw kinematic positions.

1. Object-Centric Wrench Space Representation

Instead of matching 3D contact locations (which vary based on hand morphology), CHORD represents interactions by the forces and torques (wrenches) they induce on the object.

Wrench Matrix: For each contact point, the system approximates the Coulomb friction cone using a polyhedral cone of unit-magnitude edge forces. These generate primitive wrenches ( $w \in \mathbb{R}^6$ ) representing the force-torque directions capable of inducing object motion.
Support Function: To compare human and robot contacts despite differences in the number of contact points or ordering, the method uses a support function. It projects the wrench matrices onto a set of pre-sampled unit directions to generate a support vector ( $\sigma$ ). This allows for a direct comparison of the motion-inducing capability of the contacts.

2. Reward Formulation

The policy is trained using a composite reward function $r = r_{task} + r_{imit} + r_{contact} + r_{VOC}$ :

Task Reward ( $r_{task}$ ): Encourages the rollout to follow the reference object motion (poses and inter-part relations). For multi-object tasks, a relative reward is added during interaction phases to ensure precise geometry (e.g., insertion, pouring).
Imitation Reward ( $r_{imit}$ ): Regularizes the robot motion toward the retargeted human motion.
Contact Wrench-Space Reward ( $r_{cws}$ ): This is the core contribution. It compares the robot's support function ( $\sigma_{r,k}$ ) against the human reference ( $\sigma_{h,k}$ ) using a relative tolerance $\beta$ . It rewards contacts that induce wrenches within a specific range of the human demonstration, penalizing both missed contacts and unintended extra contacts.
Virtual Object Controller (VOC): An auxiliary controller is used during early training to apply wrenches that move the object along the reference trajectory, smoothing the exploration landscape. This is annealed over time.

3. Robustness and Generalization

Noisy Data Handling: For demonstrations reconstructed from RGB videos where contact estimates are noisy, CHORD switches to a reduced force-closure objective. This rewards the generation of positive support along wrench-space basis directions without requiring precise matching, effectively learning to achieve force closure.
Whole-Body Extension: The framework extends to whole-body manipulation. For hand-only references (e.g., egocentric), an inpainting module predicts full-body motion. For third-person references, the reduced force-closure objective is used to handle inaccurate finger reconstruction.

4. Benchmark Construction

The authors constructed a large-scale simulation benchmark containing 4,739 bimanual dexterous manipulation tasks. These tasks were derived from open-source motion-capture datasets and reconstructed in-house videos, covering:

Single and multiple rigid objects.
Articulated objects.
Long time horizons and dense contact events.

Key Contributions

CHORD Framework: A novel RL framework that transfers human demonstrations to dexterous robot policies using contact wrench-space guidance, enabling scalable learning across diverse morphologies.
Large-Scale Benchmark: A curated dataset of 4,739 long-horizon, bimanual manipulation tasks spanning rigid, articulated, and multi-object interactions, significantly larger and more complex than prior benchmarks.
Empirical Validation: Comprehensive evaluation demonstrating state-of-the-art performance in simulation and successful transfer to real-world hardware.

Results

The paper reports the following quantitative and qualitative results:

Simulation Performance: Evaluated on 1,831 benchmark tasks, CHORD achieved an average success rate of 82.12%. It outperformed prior baselines (ManipTrans, DexMachina, SPIDER) across their original task suites and the new benchmark.
Whole-Body Manipulation: When extended to whole-body manipulation (using hand-only or third-person demonstrations), CHORD achieved a 90.77% success rate.
Reward Ablation: Experiments comparing "Contact Wrench" vs. "Position Only" vs. "No Contact" rewards showed that wrench-space guidance is critical. For example, on the "mixer use" task, CHORD achieved 89.4% success, while Position Only dropped to 62.4% and No Contact to 42.3%.
Correlation: A strong positive correlation (Pearson $r \approx 0.80$ ) was observed between the contact-wrench-support reward and task success, validating the reward as a reliable proxy for manipulation quality.
Long-Horizon Stability: CHORD maintained high tracking accuracy (ADD-AUC $\approx 0.85-0.98$ ) over horizons of up to 48 seconds, whereas baseline methods degraded significantly as task length increased.
Real-World Deployment: Policies trained in simulation successfully transferred to a real Dexmate robot with two Sharpa dexterous hands in both open-loop and closed-loop settings, handling rigid and articulated objects.

Significance and Claims

The paper claims that CHORD addresses the fundamental scalability bottleneck in dexterous manipulation learning. By shifting the alignment metric from kinematic position (which is embodiment-specific) to contact wrench (which is embodiment-agnostic regarding the induced object motion), the framework allows robots to learn from human demonstrations without requiring task-specific robot data or perfect morphological alignment.

The authors emphasize that this approach enables:

Scalability: Learning across thousands of diverse tasks without re-engineering reward functions for each.
Robustness: Handling noisy demonstrations and morphological mismatches (e.g., human 5-finger hands vs. robot 3-finger hands) by focusing on the mechanical effect of the contact.
Generalization: Successfully transferring skills from hand-only to whole-body embodiments and from simulation to reality.

The work positions contact wrench guidance as a necessary abstraction for moving beyond simple grasping to complex, contact-rich manipulation tasks that require coordinated, long-horizon strategies.