💻 computer science

DexSynRefine: Synthesizing and Refining Human-Object Interaction Motion for Physically Feasible Dexterous Robot Actions

DexSynRefine is a coupled framework that synthesizes physically feasible dexterous robot actions from sparse human-object interaction data by leveraging motion priors for trajectory generation and task-space reinforcement learning with contact-dynamics adaptation, thereby improving real-world success rates by 50–70 percentage points over kinematic retargeting.

Original authors: Hyesung Lee, Hyunwoo Jung, Si-Hwan Heo, Sungwook Yang

Published 2026-06-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hyesung Lee, Hyunwoo Jung, Si-Hwan Heo, Sungwook Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot with human-like hands how to perform a delicate task, like picking up a hammer or pouring water from a watering can. You could try to record a human doing it, but there's a big problem: human hands and robot hands are built differently, and the data we have from humans is often sparse (we only have a few videos) and kinematic (it shows where things move, but not how hard they push or the invisible forces involved).

If you just tell the robot to copy the human's movements exactly, it usually fails because the robot doesn't understand the physics or its own mechanical limits.

The paper introduces DexSynRefine, a three-step "recipe" that turns sparse human videos into successful robot actions. Think of it as a three-stage translation process:

1. The "Creative Director": Synthesizing the Plan (HOI-MMFP)

The Problem: You only have a few videos of a human picking up a bottle. What if the bottle is in a slightly different spot? The robot doesn't know what to do.
The Solution: The system acts like a creative director who watches your few videos and imagines hundreds of new versions of the same task.

The Analogy: Imagine you show a director a single sketch of a person opening a door. The director doesn't just copy that sketch; they use their knowledge of how doors work to imagine that person opening the door from the left, the right, or even if the door is slightly heavier.
What it does: It takes your sparse human data and generates a smooth, consistent "movie" of how a hand should move relative to the object, no matter where the object starts. It fills in the gaps so the robot has a complete plan to follow.

2. The "Physics Coach": Grounding the Plan (Task-Space Residual RL)

The Problem: Even with a perfect movie plan, the robot might try to move its fingers in a way that breaks its own joints or slips on the object because it doesn't understand friction or weight.
The Solution: The system adds a "Physics Coach" that watches the plan and makes tiny, real-time corrections.

The Analogy: Think of a dance instructor. The student (the robot) tries to follow the choreography (the synthesized plan). The instructor doesn't rewrite the whole dance; instead, they gently nudge the student's arm here or adjust their grip there to ensure they don't trip or hit the wall.
What it does: It takes the "movie" from step 1 and adds small "residual" adjustments. It learns that "Okay, the plan says grab here, but because of friction, I need to squeeze a little harder or move my wrist slightly differently." This ensures the movement is physically possible for the robot.

3. The "Intuitive Pilot": Adapting to Reality (Contact & Dynamics Adaptation)

The Problem: In the computer simulation, the robot knows exactly how heavy the object is and if it's touching the table. In the real world, the robot's sensors can't "see" these invisible forces. It's like driving a car with your eyes closed, guessing the road conditions.
The Solution: The system teaches the robot to "feel" the world through its own body movements (proprioception).

The Analogy: Imagine a blindfolded pianist. They can't see the keys, but they can feel the vibration of the strings and the resistance of the keys to know exactly where they are and how hard they are pressing.
What it does: The robot looks at its own history of movements (how its arm and fingers have been moving) to guess what's happening with the object. Did the hammer just hit the nail? Is the water sloshing? It uses these guesses to instantly adapt its grip and force, allowing it to work in the real world without needing a "cheat sheet" from a computer simulation.

The Result

When the researchers tested this three-step process on five difficult tasks (like reorienting a Pringles can or hammering a nail), the results were dramatic:

Old Way (Just copying human motion): The robot succeeded less than 10% of the time. It would drop things or fail to grasp them.
DexSynRefine: The robot succeeded 50% to 70% more often.

In short: DexSynRefine doesn't just tell the robot "copy the human." It imagines a complete plan, teaches the robot the physics of the movement, and trains the robot to feel its way through the real world, turning a few human videos into a reliable robot skill.

Technical Summary: DexSynRefine

Problem Statement

Learning dexterous manipulation from Human-Object Interaction (HOI) data offers a scalable alternative to expensive robot teleoperation. However, directly retargeting HOI demonstrations to robots faces two critical bottlenecks:

Sparsity: HOI datasets are typically small and cannot cover the diverse range of initial object states and task configurations encountered during deployment.
Kinematic Nature vs. Dynamic Reality: HOI data provides geometric motion (kinematics) but lacks the contact forces, friction, and actuation limits required for physical feasibility. Furthermore, the "embodiment gap" between human and robot hands makes direct kinematic retargeting unreliable, especially in contact-rich scenarios.

Existing methods often assume a suitable reference exists for every task instance or fail to systematically address how action representations affect physical grounding. DexSynRefine addresses the challenge of synthesizing task-conditioned interaction priors from sparse data, physically grounding them into feasible robot actions, and enabling deployment without privileged simulation states.

Methodology

DexSynRefine is a coupled framework operating in three coordinated stages:

1. HOI Motion Collection and Object-Centric Augmentation

The system collects sparse HOI demonstrations (e.g., 7 per task) using motion capture and FoundationPose. To address sparsity, it applies object-centric augmentation:

Initial object poses are perturbed via planar SE(2) transformations.
The reaching phase is interpolated, while the manipulation phase is rigidly transformed to preserve relative hand-object geometry.
This generates approximately 300 augmented trajectories per task, expanding pose coverage while maintaining interaction structure.

2. HOI-MMFP: Conditional Motion Prior Synthesis

To generate consistent trajectories for unseen object poses, the authors introduce HOI-MMFP (HOI Motion Manifold Flow Primitives), an extension of Motion Manifold Flow Primitives to coupled hand-object motion.

Manifold Learning: A Transformer autoencoder maps augmented trajectories to a compact latent space ( $z^*$ ), optimized with reconstruction, normalization, and manifold continuity losses (penalizing temporal differences in interpolated latents).
Conditional Flow Matching: A Diffusion Transformer-based vector field is trained to perform conditional flow matching in the latent space. It takes a task embedding ( $u$ ) and initial object pose ( $o_0$ ) as conditioning variables to generate a latent code $z^*$ .
Decoding: The latent code is decoded into a synthesized kinematic trajectory $\hat{\tau}$ , serving as a structured motion prior for downstream refinement.

3. Task-Space Residual Refinement with Contact-and-Dynamics Adaptation

The synthesized kinematic prior is not directly executable. The framework refines it using a two-stage reinforcement learning (RL) approach:

Task-Space Residual RL: Instead of regenerating full motions, the policy learns a residual signal ( $\Delta a_t$ ) added to the synthesized wrist and fingertip targets. This preserves the interaction structure while correcting for embodiment mismatches and contact dynamics.
Privileged Teacher Policy: Trained in simulation using PPO with an asymmetric actor-critic setup. The critic observes privileged states (simulator dynamics, contact forces, object parameters), while the actor uses proprioception and the synthesized reference.
Deployable Student Policy: To bridge the sim-to-real gap without privileged states, a student policy is distilled from the teacher. It infers missing physical context (contact states and dynamics parameters) from proprioceptive history using:
- A GRU-based contact module predicting binary contact and forces.
- A dynamics module inferring latent dynamics parameters.
- The student is trained to match the teacher's actions and inferred dynamics/contact states.

Key Contributions

HOI-MMFP: A generative prior that synthesizes task- and initial-object-state-conditioned hand-object trajectories from sparse demonstrations, ensuring spatial consistency and temporal smoothness.
Task-Space Residual Formulation: A systematic study identifying that refining task-space wrist and fingertip targets (rather than joint-space or absolute actions) provides the most effective physical grounding under embodiment and contact constraints.
Contact-and-Dynamics Adaptation: A deployable module that infers hidden physical context (contact and dynamics) solely from proprioceptive history, enabling robust real-world execution without tactile sensors or privileged simulation data.

Experimental Results

The framework was evaluated on five dexterous manipulation tasks (e.g., Reorient Pringles Bottle, Pick and Hammer, Pick and Pour Watering Can).

Motion Synthesis Quality: HOI-MMFP outperformed baselines (TC-VAE and DiT-Full) in spatial consistency under pose perturbations (lowest wrist error) and temporal smoothness (lowest jerk). It enabled downstream RL to achieve a 60.3% success rate on the "Pick and Hammer" task, compared to 49.1% for DiT-Full.
Action Representation: Task-space residual learning achieved the highest mean success rate (68.1%) across all tasks. Kinematic retargeting alone failed almost entirely (≤5.8% success), confirming the necessity of physical grounding.
Adaptation Ablation: Both contact and dynamics adaptation were shown to be jointly necessary. Removing either component caused success rates to collapse (e.g., on "Pick and Hammer," removing contact adaptation dropped success from 60.3% to 23.4% for the teacher).
Real-World Deployment: On a real 7-DoF arm with a 16-DoF hand, the full DexSynRefine pipeline improved success rates over kinematic retargeting by 50–70 percentage points. For example, success on "Pick and Place Bowl" rose from 2/10 to 9/10, and "Pick and Pour Watering Can" from 0/10 to 5/10.

Significance and Claims

The paper claims that DexSynRefine demonstrates that sparse, purely kinematic human motion priors can be transformed into executable, robust dexterous robot actions when three components are jointly considered:

Motion Synthesis: Generating consistent references for unseen states.
Physical Grounding: Using task-space residuals to correct for embodiment and contact dynamics.
Deployable Adaptation: Inferring physical context from proprioception to operate without privileged simulation states.

The authors conclude that this approach significantly reduces the reliance on expensive, task-specific robot teleoperation datasets, offering a scalable path to dexterous manipulation.

Limitations

The authors acknowledge three primary limitations:

Per-Task Policies: The residual policies are trained per task; a unified multi-task policy remains future work.
Lossy Contact Reconstruction: Without tactile sensors, contact inference relies solely on proprioception, which may miss rapid transitions in highly contact-rich tasks (e.g., hammering).
Static Conditioning: The HOI prior is synthesized only once at the episode start. If the object is displaced mid-execution, the system cannot reactively re-plan.