💻 computer science

Masquerade: Learning from In-the-wild Human Videos using Data-Editing

Masquerade addresses robot data scarcity by editing in-the-wild human videos to synthesize robot demonstrations through arm inpainting and robot overlay, enabling a co-training strategy that significantly outperforms existing methods on long-horizon bimanual tasks.

Original authors: Marion Lepert, Jiaying Fang, Jeannette Bohg

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marion Lepert, Jiaying Fang, Jeannette Bohg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to cook a complex meal, like stacking pots or scraping a potato. The problem is, you don't have thousands of videos of robots doing these tasks. Recording robot data is slow, expensive, and requires special equipment.

However, the internet is flooded with millions of videos of humans cooking. The problem with using these is that humans look and move very differently from robots. If you just show a robot a video of a human hand grabbing a spoon, the robot gets confused because it doesn't have a human hand; it has a metal gripper. This is called the "embodiment gap."

Masquerade is a clever trick to bridge this gap. Here is how it works, broken down into simple steps:

1. The "Digital Mask" (Data Editing)

Think of the researchers as digital editors wearing a "masquerade mask." They take raw videos of humans cooking and run them through a special pipeline:

Step A: They use AI to find exactly where the human's hands are in every frame.
Step B: They use "inpainting" (like a magic eraser) to paint over the human's arms and body, removing them from the video.
Step C: They digitally paste a simulated robot arm into the exact same spot where the human arm used to be.

The result is a video that looks like a robot is doing the cooking, even though it was originally a human. They turned 675,000 frames of human videos into "robotized" demonstrations.

2. The "Apprentice" (Pre-training)

Now, they have a massive library of these fake robot videos. They use this to train the robot's "brain" (a vision encoder).

The Lesson: The robot's brain learns to look at a scene and predict where the robot's gripper will move next, just by watching these edited videos.
The Analogy: It's like a student reading a thousand storybooks about how a chef moves, even if the student has never held a knife yet. They are learning the concept of the movement.

3. The "Mentor" (Co-training)

The robot still needs to learn the specific, real-world physics of its own body. So, the researchers collect a tiny amount of real data: just 50 videos of a real robot doing the task in one specific kitchen.

The Twist: Instead of just training on these 50 real videos, they train on the 50 real videos at the same time as the thousands of edited human videos.
Why? If they only trained on the 50 real videos, the robot would be too rigid and fail in new kitchens. If they only trained on the human videos, the robot wouldn't understand its own metal gripper. By doing both together, the robot learns the general "flow" of cooking from the humans and the specific "feel" of its own body from the 50 real examples.

The Result: A Master Chef in Any Kitchen

The researchers tested this on three difficult tasks (stacking pots, scraping potatoes, sweeping chilis) in brand-new kitchens the robot had never seen before.

The Competition: They compared their method against other top AI models that either learned from raw human videos (without editing) or learned from standard image datasets.
The Win: The Masquerade robot was 5 to 6 times more successful than the others.
The Key Finding: The "magic" wasn't just having more data; it was the editing. When they tried to use the human videos without replacing the human arms with robot arms, the performance crashed. The robot needed to see itself (or a version of itself) in the video to learn effectively.

Summary

Masquerade is like giving a robot a "dream" where it sees itself performing millions of tasks it has never actually done. By digitally editing human videos to look like robot videos, and then mixing that with a tiny bit of real practice, the robot learns to generalize. It can walk into a completely new kitchen and successfully cook, even though it was only trained on 50 real examples.

What the paper does NOT claim:

It does not claim the robot is perfect; the editing isn't always 100% accurate (e.g., if a hand is hidden behind a pot, the robot might look weird).
It does not claim this works for every type of robot (they used a specific two-armed setup).
It does not claim this solves the problem of robots moving around (the robot in the experiment had a fixed camera and base).

The core message is simple: Don't just watch humans; edit the video to show the robot what it would look like doing the same thing, and you get much better results.

Technical Summary: Masquerade

Problem Statement

Robot manipulation research faces a critical bottleneck: data scarcity. Unlike natural language processing (NLP) and computer vision (CV), which leverage massive, diverse datasets to achieve breakthroughs, robotics datasets are orders of magnitude smaller and less diverse. Collecting real-world robot demonstrations is slow and expensive. While "in-the-wild" human videos offer a vast, readily available source of manipulation data, leveraging them for robot policy learning is difficult due to two main factors:

Visual Embodiment Gap: Humans and robots look and move differently; human hands do not directly correspond to robot grippers.
Lack of Action Labels: Raw human videos lack precise action labels required for supervised policy learning.

Prior works have attempted to bridge this gap by pre-training vision encoders on human videos or learning reward functions, but they often fail to explicitly address the visual embodiment gap, relying instead on the model to implicitly learn correspondences. This paper posits that explicitly closing this gap, even imperfectly, can unlock significantly more signal from human videos.

Methodology: Masquerade

Masquerade is a pipeline that edits in-the-wild egocentric human videos to create "robotized" demonstrations, which are then used to pre-train and co-train a robot policy. The process consists of three main stages:

1. Data Editing and "Robotization"

The pipeline converts raw human videos (specifically from the Epic Kitchens dataset) into synthetic robot demonstrations by:

Pose Estimation: Using HaMeR to estimate 21 anatomical keypoints per hand from monocular video frames.
Retargeting: Mapping these 2D hand poses to 3D robot end-effector poses (position, orientation, and gripper width).
Inpainting: Segmenting and removing human arms from the video frames using Detectron2 and SAM2, then inpainting the background using E2FGVI.
Overlay: Rendering a simulated bimanual robot in the same pose as the estimated end-effector trajectory and compositing it into the original frame.
Filtering: Discarding frames with excessive camera motion or invalid keypoint extractions to ensure stability for fixed-base robots.

This results in a dataset of 675,000 edited frames where the video appears to show a robot performing the task.

2. Vision Encoder Pre-training

A Vision Transformer (ViT-Base) is pre-trained on the edited human videos.

Objective: The encoder is trained to predict future 2D robot keypoints (end-effector positions) over a horizon $H$ .
Conditioning: The encoder is conditioned on natural language descriptions of the video clips using FiLM layers.
Loss: A 2D keypoint regression loss ( $L_{2D}$ ) is minimized. Crucially, the 2D keypoints are derived by projecting the smoothed 3D end-effector positions onto the image plane, corrected for egocentric camera motion via homography.

3. Co-training with Real Robot Data

The pre-trained vision encoder is fine-tuned alongside a diffusion-based policy head using a small set of real robot demonstrations (50 per task).

Strategy: The model is co-trained on a mix of the edited human videos (using the auxiliary 2D loss $L_{2D}$ ) and real robot data (using an imitation loss $L_{policy}$ ).
Rationale: Retaining the pre-training objective during fine-tuning is critical. It prevents the encoder from "forgetting" the rich representations learned from the vast human video dataset, thereby maintaining robustness to out-of-distribution (OOD) scene shifts.
Synthetic Consistency: To minimize the visual gap between the training domains, the real robot demonstrations are also processed by inpainting the real robot arms and overlaying the same virtual robot model, ensuring the model always sees the "robotized" appearance.

Key Contributions

Explicit Embodiment Bridging: The paper demonstrates that explicitly closing the visual embodiment gap via simple 2D inpainting and robot overlays significantly enhances the transferability of representations from human videos to robots.
Co-training Pipeline: The authors show that co-training the vision encoder with real robot data (rather than just fine-tuning) is indispensable for preserving the knowledge gained from human videos and achieving OOD robustness.
Scalability: The method scales logarithmically with the amount of edited human video data, suggesting that larger datasets will yield further performance gains.
Zero-Shot Generalization: The approach enables zero-shot deployment in unseen environments without requiring additional robot data collection for those specific scenes.

Experimental Results

The method was evaluated on three long-horizon, bimanual kitchen tasks (Stack Pots, Scrape Potato, Sweep Chilis) across three distinct out-of-distribution scenes per task.

Performance: Masquerade outperformed all baselines (ImageNet, DINOv2, and HRP) by an average of 5–6× in success rates. Specifically, it achieved a 74% success rate compared to baselines ranging from 12% to 20%.
Ablation Studies:
- No Overlay: Removing the robot overlay and training on raw human videos caused a steep performance drop, confirming the necessity of visual alignment.
- No Co-training: Pre-training followed by pure fine-tuning (without co-training on human data) led to a dramatic performance drop, indicating that co-training is essential to retain the pre-trained representations.
Data Scaling: Success rates increased monotonically with the fraction of edited human video data used (from 2% at 0% data to 68% at 100% data).
OOD Robustness: Unlike baselines which suffered large performance drops when moving from in-distribution to OOD scenes, Masquerade maintained similar performance levels, demonstrating superior robustness to scene shifts.

Significance and Claims

The paper claims that Masquerade demonstrates that explicitly addressing the visual embodiment gap—even through imperfect 2D overlays—unlocks a vast, underutilized source of data from in-the-wild human videos. By combining data editing with a co-training strategy, the method allows robots to learn robust manipulation policies from massive human video datasets using only a handful of real robot demonstrations.

The authors emphasize that this approach does not require complex 3D reconstruction or depth data, making it a practical and scalable solution for improving robot policies. They conclude that this pipeline offers a promising path toward "web-scale" robot learning, where the limitations of robot data collection are overcome by leveraging the abundance of human video data.

Limitations Acknowledged:
The paper notes that the method relies on hand-pose estimators that struggle with fast motion or heavy occlusion, necessitating data filtering. The lack of depth data limits the realism of occlusion handling in the overlays. Furthermore, the retargeting from dexterous human hands to parallel-jaw grippers is imperfect. Future work could involve better pose estimators, depth reasoning, and more expressive retargeting for dexterous grippers.