🤖 machine learning

A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulation

This paper introduces REGRIND, a minimalist pipeline that combines retargeting-guided reinforcement learning with zero-shot sim-to-real transfer to enable dexterous, contact-rich manipulation tasks on multi-fingered robots using only a single human demonstration.

Original authors: Yunhai Feng, Natalie Leung, Jiaxuan Wang, Lujie Yang, Haozhi Qi, Preston Culbertson

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yunhai Feng, Natalie Leung, Jiaxuan Wang, Lujie Yang, Haozhi Qi, Preston Culbertson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot hand how to use a pair of scissors or turn a screwdriver. You could try to program every single finger movement by hand, but that's like trying to write a novel by typing one letter at a time with a blindfold on. Instead, the researchers behind this paper, REGRIND, tried a much simpler trick: they asked a human to do the task once, recorded it, and then taught the robot to copy that single performance.

But here's the catch: a human hand and a robot hand are built differently. If you just tell the robot, "Move your fingers to match the human's shape," you might end up with a robot hand that tries to hold a screwdriver through the table or pinches the air instead of the tool. That's like telling a cat to swim like a dog; the shape might look right, but the physics are a disaster.

The "Interaction-Preserving" Secret Sauce
The paper's main finding is that to make this work, you can't just copy the shape of the hand; you have to copy the relationship between the hand and the object. The authors call this "interaction-preserving retargeting."

Think of it like this: If a human is holding a scissor, their fingers are wrapped around the handles, and the blades are open. If you just tell the robot to put its fingers in the same spots, it might accidentally crush the scissors or drop them because it doesn't "feel" the tension. The REGRIND method builds a digital "safety net" (called an interaction mesh) that keeps the robot's fingers in the right spot relative to the tool, ensuring the robot doesn't try to hold the object in a physically impossible way.

The Recipe: One Demo, Infinite Practice
The process is surprisingly minimalist. They took one single human demonstration (a video of a person using the tool).

Retargeting: They converted that human motion into a robot-friendly path that respects the physics of holding the tool.
The "What If" Game: Since they only had one example, they used a clever trick to create thousands of practice runs. They imagined the tool starting in slightly different positions (up to 5 cm away or 30 degrees tilted) and taught the robot how to adjust its path to still succeed. It's like practicing a basketball shot not just from the free-throw line, but from slightly left, slightly right, and slightly behind, so you don't panic if the ball bounces weirdly.
Simulation Training: The robot practiced millions of times in a video game-like simulation, learning to track the tool's movement perfectly.

Did it Work? The Reality Check
The results were impressive, but with a few important asterisks. In the simulation, the robot was a superstar. On tasks like using a screwdriver with the WUJI hand, it succeeded 99.7% of the time. Even with the trickier scissors task, it hit 98.7% success. The robot learned to move fluidly, almost like a human.

However, when they moved the robot from the video game to the real world, things got a bit bumpier.

The Good News: On three out of four real-world tests, the robot worked great. It successfully used the LEAP hand to cut with scissors (9 out of 10 tries) and turn a screwdriver (10 out of 10). It also worked well with the WUJI hand on the screwdriver task (9 out of 10).
The Bad News: It failed completely on the WUJI-Scissors task (0 out of 10). The authors suspect this was because the real scissors didn't match the digital model perfectly and the robot's motors were too stiff to adjust.
The Comparison: They tested other methods that didn't use their "interaction-preserving" trick. Those methods were like students who memorized the answers but didn't understand the math. In the real world, they failed almost every time (0% success on most tasks), often because they tried to grab the tool in a way that caused the robot to crash into the table.

What This Means (and What It Doesn't)
The paper suggests that if you want a robot to learn a complex, contact-heavy task like using a tool, you need to teach it how the hand touches the object, not just where the fingers go. They proved that a single human demo, when processed correctly, is enough to teach a robot to do these tasks in the real world.

But don't expect a robot butler just yet. The system still needs a motion capture camera to tell it exactly where the tool is in real-time; it can't just "see" the tool with a regular camera yet. Also, if the real-world object doesn't match the digital model perfectly (like the WUJI-Scissors failure), the robot can get stuck.

In short, the authors found a recipe that turns a single human video into a working robot skill, but the robot still needs a little help from a camera and a perfect digital twin of the object to pull it off in the messy real world.

Technical Summary: REGRIND – A Minimalist Retargeting-Guided RL Recipe for Dexterous Manipulation

Problem Statement

Dexterous manipulation involves complex, contact-rich dynamics requiring delicate regulation of contact modes and forces. While humanoid whole-body control has successfully adopted a "retarget-then-learn" recipe (retargeting human motion to robot kinematics and training Reinforcement Learning (RL) policies to track them), this approach has not translated effectively to dexterous manipulation. Current manipulation skills often rely on teleoperated demonstrations from real robots or are confined to simulation.

The core challenge identified is that simple kinematic retargeting, while matching human hand poses, often ignores hand-object interactions. This results in physically implausible trajectories (e.g., hand-object penetrations) that serve as poor references for downstream RL. Furthermore, dexterous manipulation exhibits significantly higher sensitivity to sim-to-real discrepancies (friction, compliance, geometry) compared to locomotion, making zero-shot transfer difficult.

Methodology: REGRIND

The authors propose REGRIND (REtargeting-Guided ReINforcement learning for Dexterous manipulation), a minimalist pipeline that learns dexterous manipulation policies from a single human demonstration. The pipeline follows a real-to-sim-to-real workflow:

1. Interaction-Preserving Motion Retargeting

Instead of standard kinematic retargeting, the method formulates retargeting as an optimization problem that preserves the spatial and contact relationships between the hand and the object.

Interaction Meshes: The method constructs "interaction meshes" connecting semantic keypoints on the human hand and the object (source) and corresponding keypoints on the robot hand and the object (target).
Optimization Objective: The robot trajectory is optimized to minimize the deformation energy between the source and target interaction meshes. This is measured by the change in Laplacian coordinates of the mesh vertices.
Constraints: The optimization includes kinematic constraints, collision avoidance, and a regularization term for temporal smoothness.
Outcome: This generates a robot reference trajectory that maintains valid contact structures, avoiding the physically impossible grasps common in standard Inverse Kinematics (IK) approaches.

2. RL Training with Augmentation

A residual RL policy is trained in simulation to track the retargeted reference trajectory.

Action Space: The policy learns a residual action added to the nominal retargeted configuration ( $q_{target} = \bar{q}_t + \alpha \odot \pi_\theta(\bar{q}_t, o_t)$ ).
Observation Space: The actor uses object pose, robot joint positions, previous action, and a phase variable. To minimize the sim-to-real gap, the policy avoids using noisy sensor measurements like velocity.
Reward Function: The primary reward is a proximity metric based on object keypoints ( $\epsilon_{object}$ ), encouraging the robot to track the object's state relative to the reference. This implicitly encodes interaction priors without needing explicit contact rewards.
Data Augmentation: To address the scarcity of data from a single demonstration, the method dynamically augments the reference trajectory. It perturbs the initial object configuration (position and orientation) and applies a time-varying rigid transformation to the entire retargeted trajectory. This warps the start of the trajectory toward the new initial state while smoothly returning to the original goal, creating diverse training samples without re-solving the expensive retargeting optimization.
Curriculum & Randomization: Training includes domain randomization (friction, motor gains), observation noise, and a curriculum involving random pushes and gradually increasing gravity.

3. Sim-to-Real Transfer

The trained policy is deployed on hardware with zero-shot transfer. The authors emphasize that successful transfer requires not just the retargeting strategy but also careful system identification and the use of object poses from a motion capture system during deployment to eliminate perception errors as a confounding factor.

Key Contributions

A Minimalist Framework: A general pipeline for learning contact-rich dexterous manipulation from a single human demonstration using interaction-preserving retargeting.
Interaction-Preserving Retargeting: A novel retargeting objective that maintains hand-object spatial and contact relationships, addressing the failure of pure kinematic retargeting in contact-rich settings.
Dynamic Data Augmentation: A strategy to synthesize diverse reference trajectories from a single demo by perturbing initial states and warping the trajectory, improving policy robustness and generalization.
Systematic Analysis: An evaluation of key factors governing sim-to-real transfer in dexterous manipulation, highlighting the necessity of interaction-preserving references and careful system identification.

Experimental Results

The method was evaluated on four tasks involving two robot hands (16-DoF LEAP and 20-DoF WUJI) and two tools (Scissors and Screwdriver).

Retargeting Quality: Qualitative results show that REGRIND produces human-like grasps that preserve interaction semantics, whereas baselines (Mink IK, DexMachina) produce physically implausible penetrations or unstable grasps.
Simulation Performance: REGRIND achieved near-perfect success rates (98.7%–99.8%) and low tracking errors across all four tasks. Baselines like SPIDER failed to initialize RL effectively, and DexMachina struggled significantly with the complex contact dynamics of the Scissors task.
Sim-to-Real Transfer:
- REGRIND successfully transferred to real hardware on three of four tasks (LEAP-Scissors, LEAP-Screwdriver, WUJI-Screwdriver), achieving success rates of 9/10 or 10/10.
- Baselines (DexMachina, Mink IK + RL) failed to transfer effectively, often resulting in unstable grasps or collisions, attributed to their reliance on simulation artifacts and lack of interaction-preserving regularization.
- The WUJI-Scissors task failed for all methods, likely due to non-backdrivable motors and inaccurate mesh modeling of the real scissors.
Generalization: Policies trained with REGRIND generalized well to randomized initial object configurations in the real world, maintaining performance similar to the demonstrated configuration.

Significance and Claims

The paper claims that preserving interactions during retargeting is a critical factor for improving RL performance in contact-rich dexterous manipulation. The authors argue that while retargeting-based learning is effective for locomotion, manipulation requires a more nuanced approach due to its sensitivity to modeling errors.

The significance of REGRIND lies in demonstrating that a minimalist pipeline—combining interaction-aware retargeting, residual RL, and simple data augmentation—can achieve robust, human-like dexterous manipulation on real hardware from a single demonstration. The work provides practical guidance for the community, suggesting that successful sim-to-real transfer in manipulation requires a holistic approach involving interaction-preserving references, thoughtful observation/action space design, and careful system identification, rather than just scaling up simulation or reward engineering.