💻 computer science

Phantom: Training Robots Without Robots Using Only Human Videos

The paper presents "Phantom," a scalable framework that enables zero-shot training of general-purpose manipulation robots using only human video demonstrations by converting them into robot-compatible data through hand pose estimation and visual domain alignment, achieving high success rates on diverse tasks without requiring any robot data or fine-tuning.

Original authors: Marion Lepert, Jiaying Fang, Jeannette Bohg

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Marion Lepert, Jiaying Fang, Jeannette Bohg

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to do chores, like sweeping a floor or tying a knot. Usually, to teach a robot, you have to physically hold its arm and guide it through the motions thousands of times. This is like trying to teach a child to ride a bike by pushing them around the neighborhood for hours every day. It's slow, expensive, and hard to do for every possible task.

The paper "Phantom" proposes a clever shortcut: Why not just watch humans do the chores and pretend the robot is the human?

Here is how they did it, using simple analogies:

1. The "Magic Photo Editor" (Data Editing)

The researchers realized that while we have millions of videos of humans doing things on the internet, robots can't just "watch" them. A robot sees the world differently than a human; it has metal arms and grippers, not flesh and fingers. If you train a robot on a video of a human, the robot gets confused because the "teacher" looks nothing like the "student."

To fix this, the authors built a digital "magic photo editor." Here is the process:

Step 1: The Eraser. They take a video of a human reaching for an object. Using AI, they digitally "erase" the human's arm and hand, filling in the background so it looks like the arm was never there.
Step 2: The Puppeteer. They use a computer program to figure out exactly where the human's hand was moving.
Step 3: The Swap. They take a 3D model of a robot arm and "paste" it into the video, moving it in the exact same way the human's hand moved.

The result is a video that looks like a robot doing the task, but it was actually filmed with a human. They call this a "Phantom" demonstration.

2. The "Ghost in the Machine" (Zero-Shot Deployment)

The most surprising part of their discovery is that they didn't need to show the robot a single real-life video of itself doing the task. They trained the robot only on these edited "Phantom" videos.

Think of it like this: If you want to learn to play tennis, you usually watch a pro player. But if you are a robot with a different body shape, watching a human might be confusing. This method is like taking a video of a human tennis player, digitally replacing their body with a robot's body, and then teaching the robot to play by watching that edited video.

When they tested this on real robots (specifically a Franka and a Kinova robot), the robots were able to perform tasks without any prior practice or fine-tuning. It was like the robot walked into the room, saw the task, and immediately knew what to do, having only "studied" the edited human videos.

3. What Could the Robot Do?

The researchers tested this on a variety of tricky tasks, proving the method works even when things are messy or flexible:

Sweeping: Moving a broom to push trash into a dustpan (which involves moving many loose pieces of trash that bounce around unpredictably).
Tying Knots: Tying a rope into a specific sailing knot (which is very hard because ropes are floppy and change shape).
Stacking: Carefully stacking cups that are slightly different sizes.
Rotating: Turning a box over without just knocking it over.

In many of these tests, the robots succeeded up to 92% of the time, even in rooms they had never seen before.

4. Why This Matters (According to the Paper)

The paper argues that this approach removes the biggest bottleneck in robotics: the need for expensive robot data.

Old Way: You need a robot, a lab, and a human to spend hours teleoperating (controlling) the robot to collect data.
New Way (Phantom): You just need a camera and a human. You can film anyone, anywhere, doing a task. The computer does the rest of the work to translate those human movements into robot instructions.

The authors emphasize that this works for "closed-loop" execution, meaning the robot can react to changes in real-time (like if the trash moves unexpectedly), not just follow a pre-recorded script.

What the Paper Does Not Claim

It is important to stick to what the paper actually says:

They did not claim this works for every single type of robot or every possible task. They focused on tasks where a human can use a "pinch grasp" (holding things with a thumb and finger), which matches the grippers on the robots they used.
They did not claim this is perfect. If the human's hand is hidden (occluded) in the video, the computer might guess the hand's position wrong, which can confuse the robot.
They did not claim this replaces the need for robots entirely; they still need a robot to physically do the work. They just removed the need to record data from the robot.

In short, the paper presents a "Phantom" method where we can train robots using only videos of humans, by digitally swapping the human body for a robot body in the footage, allowing robots to learn complex skills without ever needing to be physically demonstrated on a robot.

Technical Summary: Phantom: Training Robots Without Robots Using Only Human Videos

Problem Statement

Advancing general-purpose robotics is currently bottlenecked by data scarcity. While large-scale datasets exist for vision and language models, robotics datasets remain orders of magnitude smaller due to the slow, costly, and difficult process of collecting teleoperated demonstrations with physical hardware. Furthermore, simply increasing data volume is insufficient; diversity across environments is critical for generalization, yet collecting diverse data with robots is a formidable challenge.

Existing approaches to leverage human videos for robot learning face two primary hurdles:

Lack of Action Labels: Human videos do not contain explicit robot action trajectories.
Embodiment Gap: The visual appearance of human arms and hands differs significantly from robot arms and grippers, causing vision-based policies trained on human data to fail when deployed on robots.

Prior methods attempting to bridge this gap typically rely on co-training with robot data or reinforcement learning, which limits scalability. Other approaches using object-centric tracking struggle with deformable objects, granular materials, or multi-object scenarios.

Methodology: The Phantom Framework

The authors propose Phantom, a scalable framework that trains manipulation policies using only human video demonstrations, requiring zero robot data. The core innovation is a data-editing pipeline that converts human demonstrations into robot-compatible observation-action pairs.

1. Action Extraction from Human Videos

Since human videos lack explicit action labels, the method extracts robot actions by estimating human hand poses:

Pose Estimation: The system uses HaMeR to predict 21 hand keypoints and a dense 778-vertex hand mesh from each video frame.
3D Refinement: To address the ambiguity of monocular depth estimation, the system segments the hand using SAM2 and extracts a partial point cloud from the RGB-D depth image. An Iterative Closest Point (ICP) registration aligns the HaMeR mesh with the point cloud to refine the 3D pose.
Occlusion Handling: To prevent unrealistic finger configurations during occlusion (common in grasping), the system constrains the thumb and index finger joints to a single degree of freedom within anatomically feasible ranges.
Action Generation: The refined keypoints define the robot's end-effector pose:
- Position ( $p_t$ ): Midpoint between the thumb and index finger tips.
- Orientation ( $R_t$ ): Derived from a plane fitted to the finger keypoints and a principal axis vector.
- Gripper Width ( $g_t$ ): Distance between the fingertips, with a threshold applied to the bottom 20th percentile to ensure full closure.

2. Visual Domain Bridging via Data Editing

To eliminate the visual domain shift between human and robot embodiments, the authors adapt data-editing techniques (inspired by Rovi-Aug) to the human-to-robot setting:

Training Phase:
1. Segmentation: The human arm is segmented using SAM2.
2. Inpainting: The segmented arm is removed from the image using E2FGVI (a video inpainting model) to restore the background.
3. Rendering & Overlay: A virtual model of the target robot is rendered in the estimated pose ( $p_t, R_t, g_t$ ) and overlaid onto the inpainted image. Depth data is used to ensure realistic occlusions by environmental objects.
Inference Phase:
- To ensure consistency between training (rendered robot) and testing (real robot) distributions, a virtual robot arm is overlaid onto the real robot's observation image at test time.

3. Policy Training

The augmented dataset, consisting of edited images and extracted actions, is used to train a closed-loop imitation learning policy (specifically a Diffusion Policy). This policy is then deployed zero-shot on a real robot without any fine-tuning or additional robot data collection.

Key Contributions

Zero-Robot-Data Training: The paper demonstrates that effective robot manipulation policies can be trained exclusively from human video demonstrations, removing the need for expensive teleoperated robot data.
Data-Editing for Cross-Embodiment: The work successfully adapts data-editing strategies (inpainting and virtual overlay) from robot-to-robot transfer to the more challenging human-to-robot setting.
Versatility: The method handles a diverse range of tasks, including:
- Rigid object manipulation (pick and place, stacking).
- Deformable object manipulation (tying a rope knot).
- Multi-object manipulation (sweeping multiple pieces of trash).
Generalization: The approach supports zero-shot deployment in novel environments and on different robot embodiments (Franka and Kinova) without retraining.

Experimental Results

The authors evaluated Phantom across six tasks on Franka and Kinova robots:

In-Distribution Performance: In a single scene, the Hand Inpaint strategy achieved success rates up to 92% (Pick/Place Book) and 88% (Tie Rope). It significantly outperformed baselines like "Red Line" (EgoMimic style) and "Vanilla" (no editing), which achieved 0% success.
Out-of-Distribution (OOD) Generalization: When tested in unseen environments (e.g., outdoor lawns, indoor lounges with moving backgrounds), the method maintained high success rates (e.g., 84% in an indoor lounge). Performance dropped slightly (20%) on unseen surfaces, attributed to limited training surface diversity.
Ablation Studies:
- Inpainting Quality: High-quality inpainting (E2FGVI) yielded the best results (84%), but low-quality inpainting (OpenCV) still achieved 76%, suggesting the model is robust to artifacts. "Mask Only" (no inpainting) dropped to 60%, highlighting the importance of background restoration.
- Comparison to Hand Mask: An alternative "Hand Mask" strategy (masking the arm black) performed comparably but was 73% slower at inference due to the need for a diffusion model to generate masks at test time.
Robot vs. Human Data: In a single-scene comparison, policies trained on 300 human demonstrations (84% success) approached the performance of those trained on 100 robot demonstrations (88% success), suggesting that the scalability of human data collection can compensate for slight precision losses in pose estimation.

Significance and Claims

The paper claims that Phantom broadens the path to scalable robot learning by demonstrating that human videos alone, when subjected to careful data editing and pose estimation, are sufficient to train general-purpose robot policies.

Scalability: By removing the dependency on physical robot data collection, the method lowers the barrier to creating large-scale, diverse datasets. Anyone with an RGB-D camera can collect training data anywhere.
Asymptotic Setting: The work addresses a future regime where human data is orders of magnitude more prevalent than robot data. It shows that policies can be trained effectively even when robot data is scarce or non-existent.
Limitations: The authors modestly acknowledge limitations, including reliance on the accuracy of current hand pose estimators (which struggle with severe occlusions), the assumption that the robot can follow the same strategy as the human (e.g., pinch grasps), and the focus on quasi-static tasks where latency mismatches are less critical.

In conclusion, Phantom establishes that data-editing-based cross-embodiment learning is a viable and scalable strategy for transferring skills from humans to robots, unlocking the potential of the vast, untapped resource of human video demonstrations.