💻 computer science

What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?

This paper investigates the transferability of robot manipulation policies trained on everyday human videos, revealing that while high-quality hand pose labels are beneficial, successful transfer requires specialized vision and policy networks to bridge the motion gap, ultimately achieving a 29.7% success rate improvement in low-robot-data regimes through a new cotraining approach.

Original authors: Richard Li, Aditya Prakash, Andrew Wen, Saurabh Gupta, Yilun Du, Pulkit Agrawal

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Richard Li, Aditya Prakash, Andrew Wen, Saurabh Gupta, Yilun Du, Pulkit Agrawal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to cook, fold laundry, or tidy up a room. The traditional way is to hire a human to sit down, wear special sensors, and slowly demonstrate the task to the robot over and over again. This is like hiring a personal tutor who only speaks a very specific, robotic language. It's expensive, slow, and you can only get a few hours of lessons.

This paper asks a simpler question: Why can't we just teach the robot by watching YouTube videos of regular people doing these things?

The authors tried this, but they hit a few walls. They built a "training camp" to figure out exactly what makes teaching a robot from human videos work (or fail). Here is what they found, explained simply:

1. The "Hand" Problem: Seeing the Details

When you watch a video of a person pouring coffee, your eyes naturally focus on their hand. But computers are bad at guessing exactly where a hand is in 3D space just by looking at a flat video.

The Old Way: Previous attempts used "monocular" estimation (guessing 3D from a single 2D camera). This is like trying to guess the exact shape of a cloud just by looking at its shadow; it's often blurry and wrong.
The Fix: The authors created a new dataset called TriHands. They took 532 cooking videos and used multiple cameras (like a movie set with cameras on all sides) to mathematically "triangulate" the exact position of the hands.
The Result: It's the difference between a blurry sketch and a high-definition blueprint. They found that the more accurate the hand data, the better the robot learned. Even though their "blurry sketch" method (using standard AI) still helped a little, the "blueprint" method was far superior.

2. The "Camera" Problem: Different Lenses, Different Worlds

Imagine you are teaching a robot using a video taken with a wide-angle fisheye lens (like a GoPro on a helmet) and the robot has a standard camera on its arm.

The Issue: In the video, a cup might look huge because the camera is close and wide. On the robot, that same cup looks tiny because the camera is narrow and far away. If you just feed both images to the robot, it gets confused. It's like trying to learn to drive a car by watching a video of someone driving a go-kart, but the video is zoomed in so much the road looks like a hallway.
The Fix: They developed a way to "resize" the video world to match the robot's world. They mathematically adjusted the video so that objects appear at the same scale and distance as they would to the robot.
The Result: This simple adjustment was a game-changer. Without it, the robot couldn't learn. With it, the robot could actually understand the video.

3. The "Body" Problem: Humans and Robots Move Differently

Humans have flexible arms, wrists, and fingers. Robots usually have stiff, jointed arms with a gripper.

The Issue: If you force a robot to copy a human's exact hand movements, it fails. A human can twist their wrist to grab a cup; a robot arm might not be able to bend that way. It's like trying to teach a penguin to fly by showing it videos of eagles. The penguin will just flap its wings and fall.
The Fix: Instead of forcing the robot to copy the human exactly, they built a special "translator" network. This network learns the goal of the movement (e.g., "get the cup") but lets the robot figure out its own way to get there using its own body. They gave the robot its own "brain" for understanding its own body, separate from the human's.
The Result: This "specialization" allowed the robot to learn the intent of the human without getting stuck trying to mimic impossible human poses.

The Big Payoff

The authors tested this on six different tasks (like stacking bowls, pouring water, or picking up a book).

The "Low Data" Miracle: When the robot had very little practice data of its own (only a few hours), adding the human videos boosted its success rate by nearly 30%. It's like the robot went from barely passing the test to getting an A just by watching the human videos.
The "High Data" Reality: When the robot had tons of its own practice data, the human videos helped less, but they still provided a small boost.

The Bottom Line

Teaching robots from everyday internet videos is possible, but you can't just dump the videos into the robot's brain and hope for the best. You need three things:

Clear Hand Labels: You need to know exactly where the hands are (not just a guess).
Scale Alignment: You need to make sure the video world looks the same size as the robot's world.
Body Specialization: You need to let the robot learn its own way of moving, rather than forcing it to copy human anatomy.

By fixing these three things, the authors showed that we can use the vast library of human videos on the internet to train robots much faster and cheaper than ever before.

Technical Summary: What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?

Problem Statement

Building generalist robotic foundation models is currently hindered by data scarcity. While teleoperated demonstrations are expensive and slow to collect, and reinforcement learning faces challenges in reward design and sim-to-real transfer, Internet videos of human everyday activities offer a potentially scalable source of physical intelligence. However, leveraging these videos for robot manipulation is non-trivial. Prior work either relies on modular pipelines that decompose policies into intermediate components (e.g., 3D poses, affordances) or employs end-to-end cotraining that requires highly curated, aligned data where human motions are orchestrated to match robot kinematics, cameras are identical, and specialized hardware captures accurate 3D hand poses.

The open question addressed by this work is: What factors enable the transfer of manipulation policies from unconstrained, everyday Internet videos to robots? Specifically, the authors investigate the impact of two primary sources of difficulty: (1) the quality of 3D hand pose labels and (2) the "action gap" induced by natural human motion and task mismatches compared to robot behavior.

Methodology

1. The TriHands Dataset

To rigorously study the impact of hand label quality without the noise inherent in monocular estimation, the authors constructed a new dataset, TriHands.

Source: 532 cooking videos from the EgoExo4D dataset.
Annotation: Instead of relying on monocular estimators, the authors used a multi-view pipeline to triangulate 3D hand poses. They utilized extrinsic cameras alongside the egocentric view to generate high-quality 3D labels.
Refinement: The pipeline involved fine-tuning a 3D hand pose estimator (WiLoR) using reprojection loss on partial 2D labels provided by EgoExo4D. They employed Direct Linear Transformation (DLT) with nonlinear refinement, selecting camera subsets that maximize camera count while satisfying reprojection error thresholds.
Scale: The resulting dataset contains over 28 hours of high-quality, natural-motion data with 3,042,406 right-hand frames at 30fps.

2. Framework and Architecture

The authors propose a cotraining framework that balances aligning observations/actions across embodiments while allowing the network to specialize to the specific kinematic and sensory differences between humans and robots.

Image-Space Scale Alignment: A critical, previously overlooked step involves aligning the image-space scale of objects. Since human and robot cameras often have different focal lengths and fields of view (FOV), the authors undistort human fisheye images and adjust the robot camera's extrinsics (rotation and depth translation) to match the vertical 3D extent of the human scene. This ensures objects appear at similar scales in the network's input.
Action Space Alignment: Human 3D hand poses are mapped to the robot's action space (TCP pose and grasp command). To handle the multimodality introduced by head camera motion, per-timestep camera-frame poses are stabilized by transforming them to the current head frame. Action chunks are mean-centered and scaled to $\pm 1$ to resolve distribution biases.
Architecture (Transfusion-inspired): The model uses a conditional flow-matching architecture.
- Token-Level Fusion: Unlike prior works using image bottlenecks (CLS tokens), this architecture retains full ViT image tokens and concatenates them with noisy action tokens via shared attention.
- Embodiment Specialization: The network employs separate FFN experts for visual and action tokens. Crucially, it uses embodiment-specific action encoders and decoders (separate MLPs for human and robot) rather than shared weights. This allows the model to learn invariant visual representations while accommodating the distinct motion dynamics of each embodiment.
Cotraining Algorithm: To prevent the large volume of human data from overwhelming the smaller robot dataset, the authors sample equal numbers of batches from each embodiment per step. This is mathematically equivalent to upweighting the robot loss, ensuring the policy remains grounded in robot kinematics.

Key Contributions

TriHands Dataset: A large-scale dataset of everyday human videos with accurate, triangulated 3D hand labels. The authors demonstrate that this dataset yields better robot transfer performance than large-scale lab datasets (e.g., EgoDex) despite having 5x fewer hours of data, highlighting the importance of scene diversity over sheer scale.
Impact of Hand Quality: The study shows that higher-quality 3D hand labels significantly improve transfer. While state-of-the-art monocular estimators (HaWoR) still provide some benefit over robot-only training, there is a substantial performance gap between monocular estimates and triangulated hands, particularly on complex tasks.
Image-Space Scale Alignment: The authors identify and demonstrate the necessity of aligning image-space scales when cotraining across datasets with different cameras. Simple geometric adjustments to the robot camera's virtual position significantly improve transfer, a factor not currently accounted for in leading Vision-Language-Action (VLA) models.
Architecture for Natural Motion: The paper establishes that standard cotraining recipes designed for lab data (e.g., shared action encoders, image bottlenecks) fail to transfer effectively to natural motion data. Instead, embodiment-specific specialization (token-level fusion and separate encoders/decoders) is necessary to bridge the large motion gap inherent in unconstrained human videos.

Results

The authors evaluated their approach on six real-world manipulation tasks (Pick, Stack, Reorient, Pull, Book, Pour) using a 6-DOF AgileX Piper arm. They trained on 532 human videos and varying amounts of robot data (3, 5, or 10 environments).

Performance Gains: The proposed cotraining recipe yielded consistent improvements across all robot data regimes. In the low-robot-data regime (3 environments), the method achieved an absolute success rate gain of 29.7% (41.7% vs. 12.0% for robot-only) across six tasks.
Scaling Behavior: While the relative benefit of human data diminishes as the amount of task-specific robot data increases (a known phenomenon in cross-task cotraining), the absolute gains remain significant in low-data regimes.
Ablation Studies:
- Architecture: Removing token-level fusion (using CLS tokens) or sharing action encoders/decoders resulted in significant performance drops, confirming the necessity of specialization.
- Scale Alignment: Using a "heavily misaligned" pinhole camera model (without scale correction) resulted in performance comparable to robot-only training, whereas extent-matched alignment doubled performance.
- Hand Quality: Adding Gaussian noise to triangulated hands to simulate monocular estimation errors caused a monotonic degradation in performance, confirming that hand pose accuracy is a limiting factor.

Significance and Claims

The paper claims to chart a path toward robot cotraining on Internet video by identifying and addressing the specific bottlenecks that distinguish such data from curated lab demonstrations. The authors assert that:

Natural motion is a double-edged sword: While it offers scalability, the inherent motion gap requires architectural specialization (embodiment-specific encoders/decoders) rather than simple alignment.
Data quality matters more than scale: Scene diversity in natural videos (TriHands) outperformed larger, more constrained lab datasets (EgoDex).
Future directions: The work suggests that future improvements in hand pose estimation (moving closer to the quality of triangulated labels) will unlock even greater transfer capabilities. The authors note that their current reliance on kinematic retargeting may limit effectiveness on the most complex tasks (e.g., "Pour"), leaving further investigation into representational alignment and action retargeting for future work.

The study concludes that while challenges remain, a combination of high-quality hand labels, careful image-space alignment, and embodiment-specialized architectures makes cotraining on everyday human videos a viable and effective strategy for improving robot manipulation, particularly in data-scarce scenarios.