💻 computer science

Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

The paper presents "Do as I Do," an algorithm that reconstructs hand-object interactions from monocular RGB human videos and retargets them into executable actions for dexterous multi-fingered robotic hands, effectively bridging the human-to-robot embodiment gap to scalably generate manipulation data.

Original authors: Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad Mahi Shafiullah, Jitendra Malik

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhawna Paliwal, Haritheja Etukuru, William Liang, Pieter Abbeel, Nur Muhammad Mahi Shafiullah, Jitendra Malik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to perform a delicate task, like whisking eggs or hammering a nail. Traditionally, you'd have to sit down with the robot, hold its hand, and physically guide it through every single movement. This is slow, expensive, and requires a human expert to do the guiding.

This paper introduces a new method called DO AS I DO that tries to solve this problem by letting robots learn simply by watching everyday videos of humans doing these tasks. Think of it as the robot watching a YouTube tutorial and then trying to replicate the moves on its own.

Here is how the system works, broken down into two main steps using simple analogies:

Step 1: The "3D Detective" (Reconstruction)

The first challenge is that the robot is watching a flat, 2D video (like on your phone), but it needs to understand the world in 3D to move its fingers. The video might be shaky, blurry, or the hand might be hiding the object.

The Problem: If you just look at a video of someone holding a cup, a computer might get confused about how big the cup is, how far away it is, or exactly how the fingers are wrapped around it.
The Solution: The authors built a "3D Detective" system. It looks at the flat video and uses advanced AI to guess the 3D shape of the hand and the object, and how they move through time.
The Trick: To keep the object from "drifting" or changing shape weirdly as it moves, the system picks one clear frame to lock onto the object's shape (like taking a perfect photo of the cup) and then just tracks how that fixed shape moves through the rest of the video. It's like gluing a sticker of the object onto the video and just watching the sticker move, rather than trying to redraw the object every single second.

Step 2: The "Body Translator" (Retargeting)

Once the system knows what the human hand is doing in 3D, it has to translate those moves to the robot. But here's the catch: Human hands and robot hands are built differently.

The Problem: A human has long, flexible fingers with skin; a robot hand might have shorter, stiffer metal fingers. If you just copy the human's finger angles directly to the robot, the robot might try to grab a cup in a way that breaks its own fingers or drops the cup because it doesn't understand physics (like gravity or friction).
The Solution: The system acts like a translator that doesn't just copy words, but translates the intent of the movement.
- The "Warm-up" Trick: Before the robot tries to grab the object, it runs a "warm-up" simulation where it practices holding the object in mid-air. This helps the robot find a stable starting position so it doesn't immediately drop the item when the real action starts.
- The "Shake-Up" Trick: To make sure the robot is robust, the system simulates tiny, random bumps or pushes during the practice. This is like teaching a child to balance on a bike by gently nudging them; it forces the robot to learn a grip that won't fail if things get slightly messy in the real world.
- The "Transition" Check: The system specifically checks the moments when the robot switches from "resting" to "holding." It adds a penalty if the robot tries to pick something up but misses the contact, ensuring the robot actually grabs the object, not just waves near it.

What Did They Achieve?

The team tested this on a massive collection of videos found on the internet, including:

Videos filmed by people holding the camera (first-person view).
Videos filmed by someone else watching (third-person view).
Videos generated by AI.

They found that their method was much better at figuring out 3D hand movements than previous tools. More importantly, they successfully took these reconstructed movements and ran them on a real, physical robot hand (a dexterous hand with 22 moving parts). The robot could successfully perform tasks like whisking, pouring, dusting, and picking up objects based only on watching the videos.

The "Reality Check" (Limitations)

The authors are honest about what the system cannot do yet:

It assumes the objects are solid and rigid (it struggles with soft things like dough or cloth).
It only looks at the hand and the object, ignoring the rest of the room. If there's a table in the way, the robot might not know to avoid it unless the video clearly shows the collision.
It relies on the physics simulator being accurate; if the simulation is slightly wrong, the real robot might struggle.

The Bottom Line

DO AS I DO is a pipeline that turns "watching" into "doing." It takes a messy, everyday video, reconstructs the 3D physics of the action, translates it to a robot's specific body, and produces a set of instructions that a real robot can follow to perform dexterous tasks. It's a step toward letting robots learn from the vast library of human videos already on the internet, rather than requiring humans to manually teach them every single move.

Technical Summary: DO AS I DO

Problem Statement

The field of robotic learning faces a critical bottleneck: the scarcity of experiential data for dexterous manipulation, particularly on multi-fingered robotic hands. While observational data (monocular RGB videos of humans performing tasks) is abundant, converting this "watching" data into "doing" data for robots is hindered by two primary challenges:

Hand-Object Interaction Estimation: Accurately reconstructing 3D hand-object interactions from noisy, in-the-wild monocular RGB videos is difficult. Existing methods often fail under motion blur, occlusion, or when dealing with diverse, unstructured objects.
Embodiment Gap and Retargeting: Even with reconstructed human trajectories, mapping them to a robot with vastly different morphology (finger lengths, articulations) is non-trivial. Prior retargeting approaches often rely on kinematic solvers that ignore physics (leading to penetration or instability) or require clean, ground-truth reference data (e.g., motion capture) which is unavailable for internet-scale video.

The paper asks: How can we close the gap between observational human data and experiential robot data without making restrictive assumptions about the behavior, object classes, or data modality?

Methodology: DO AS I DO

The authors propose DO AS I DO, a two-stage pipeline that reconstructs hand-object interactions from monocular RGB videos and retargets them into physically feasible robot actions.

1. Reconstruction (Hand-Object Tracking)

The first stage aims to recover a consistent 4D (3D + time) trajectory of the hand and object from a single-view video.

Hand Tracking: The system utilizes HaWoR [45], a robust model for world-space hand motion reconstruction, to track the human hand.
Object Tracking via Guided Diffusion: For the object, the authors adapt SAM 3D [11], an image-to-3D generative foundation model. Since SAM 3D generates independent meshes per frame (lacking temporal coherence), the authors introduce a Guided Diffusion tracking mechanism:
- They fix the object's shape $\bar{x}_s$ at an anchor frame.
- For subsequent frames, they sample the pose $x^p_k$ conditioned on the fixed shape and the previous frame's pose $x^p_{k-1}$ .
- This is achieved by modifying the flow-matching inference process (Eq. 1), blending the model's denoising update with target interpolants (the fixed shape and previous pose) using adaptive guidance parameters.
- Adaptive Guidance: The pose guidance strength $\alpha_p$ is dynamically derived from the object's estimated rotational velocity (using 2D point tracks), preventing over-rigidity or spurious flips.
- Sampling & Selection: To handle the stochasticity of diffusion, the system samples $N$ candidate poses per frame and selects the best one using a clustering-based heuristic on a weighted SE(3) distance, rather than computationally expensive log-density ranking.
Alignment: The independently reconstructed hand and object (which may be at different scales) are aligned into a near-metric space. The object's translation is scaled relative to the hand's centroid using depth information from MoGe [10], and the trajectory is aligned with gravity using GeoCalib [68].

2. Retargeting (Dynamics-Aware Optimization)

The second stage maps the reconstructed human trajectory to a dexterous robot hand (Sharpa Wave) and arm (UR3e) using sampling-based optimization (MPPI-style) within a physics simulator (MuJoCo Warp).

Warmup Steps: To address the issue of noisy initial frames that might place the robot in an unrecoverable state (e.g., dropping an object immediately), the system prepends $H$ "warmup" steps. During this phase, the object is held in place (e.g., mid-air) while the robot hand moves freely to find a stable grasp before tracking begins.
Random Force Perturbation: To ensure robustness against local minima and unstable interactions (e.g., balancing on fingertips), the optimization introduces random forces and torques during rollouts. This encourages controls that are robust to minor disturbances.
Transition Reward: To handle the discrete nature of "rest" vs. "in-hand" states, a penalty term is added for failed transitions (e.g., lack of hand-object contact during "in-hand" phases or lack of object-floor contact during "rest" phases).

Key Contributions

Algorithm: Introduction of DO AS I DO, a pipeline that reconstructs and retargets behaviors from arbitrary monocular RGB videos to multi-fingered dexterous hands without assuming specific object classes or grasping priors.
Reconstruction SOTA: A hand-object reconstruction process that outperforms state-of-the-art (SOTA) methods on standard benchmarks (DexYCB, HOI4D) and handles diverse, noisy in-the-wild videos (ego- and exocentric).
Robust Retargeting: A dynamics-aware retargeting process that improves upon existing scalable techniques by introducing warmup steps, force perturbations, and transition rewards to handle noisy, reconstructed references.
Real-World Pipeline: The first known pipeline capable of taking an internet video and generating real-world dexterous hand rollouts, validated on a bimanual robot setup.

Experimental Results

Reconstruction: On the DexYCB and HOI4D datasets, DO AS I DO achieves new SOTA performance in F-score (F-5, F-10) and Chamfer Distance (CD), outperforming joint reconstruction methods (e.g., MCC-HO, G-HOP) and object trackers (FoundationPose, Any6D).
In-the-Wild Evaluation: On a benchmark of 150 internet videos, human raters preferred the object tracking of DO AS I DO over FoundationPose in 67% of cases. Qualitatively, the method maintains temporal consistency where baselines lose the object under motion blur or occlusion.
Retargeting: On the reconstructed in-the-wild dataset, the method achieves a 71% success rate, a significant improvement over the baseline Annealed Sampling method (25%). The "Warmup" component was identified as the primary differentiator. On the clean OakInk2 MoCap dataset, success rates improved from 72% (baseline) to 81%, demonstrating the method's efficacy even with high-quality references.
Deployment: The system successfully executed 10 diverse tasks (e.g., whisking, pouring, hammering) on a real-world bimanual robot setup.

Significance and Claims

The paper claims that DO AS I DO bridges the gap between observational and experiential data for dexterous manipulation. By leveraging recent advances in 3D computer vision (foundation models for depth and 3D generation) and GPU-accelerated physics simulation, the authors demonstrate that it is possible to scale robot data collection by simply observing humans.

The authors position their work as a step toward making human videos "first-class citizens" in the robot learning landscape. They propose an "efficacy playbook" for practitioners, highlighting that raw internet data requires significant filtering (only ~4-5% of sampled clips were deemed useful in their analysis) but that the remaining data is highly valuable for learning complex manipulation skills.

Limitations Acknowledged:
The approach assumes rigid objects and semi-accurate metric depth predictions. It struggles with ambiguous hand-object distances (distinguishing contact from occlusion) and does not reason about full scene constraints (obstacles, other objects), focusing only on the hand and a single target object. Additionally, the fidelity of the real-world performance is bounded by the approximation of the physics simulator.