💻 computer science

EgoAVFlow: Robot Policy Learning with Active Vision from Human Egocentric Videos via 3D Flow

EgoAVFlow is a robot policy learning framework that leverages a shared 3D flow representation from human egocentric videos to simultaneously learn manipulation and active viewpoint control, enabling robust task execution without requiring robot demonstrations.

Original authors: Daesol Cho, Youngseok Jang, Danfei Xu, Sehoon Ha

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daesol Cho, Youngseok Jang, Danfei Xu, Sehoon Ha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to do a task, like folding a towel or spraying a bottle, but you only have videos of a human doing it.

The problem is that humans and robots are built differently. A human wears a camera on their head (their eyes) and moves their head around instinctively. A robot has a camera mounted on a fixed arm. If you just tell the robot, "Copy exactly what the human's head did," the robot might end up looking at the ceiling, getting blocked by its own arm, or losing sight of the object entirely.

EgoAVFlow is a new "brain" for robots that solves this problem. Here is how it works, explained with simple analogies:

1. The Problem: The "Bad Copycat"

Think of a human doing a task like a magician. They move their eyes and head quickly to get the best angle to see the trick. If you try to teach a robot to copy the magician's head movements exactly, the robot might look silly or miss the point because the robot's "body" is different. It doesn't need to move its head like a human; it just needs to see the object clearly to do the job.

2. The Solution: The "3D Flow Map"

Instead of trying to copy the human's head movements, EgoAVFlow learns the physics of the movement.

Imagine the human video is a 2D movie. EgoAVFlow takes that movie and turns it into a 3D hologram (this is the "3D Flow").

It ignores what the object looks like (the color, the texture).
It focuses entirely on where the object is moving in 3D space.
It's like watching a dance and only tracking the dancers' feet and the path they take, ignoring their costumes. This allows the robot to understand the motion without getting confused by the human's specific body shape.

3. The Secret Sauce: The "Smart Camera"

This is the coolest part. The robot doesn't just watch; it actively moves its camera to keep the object in sight.

The Old Way: The robot blindly follows the human's head path.
The EgoAVFlow Way: The robot has a "Crystal Ball." Before it moves, it predicts:
1. Where the object will be in the next few seconds.
2. Where its own arm will be.
3. Will my arm block my view?

If the robot predicts, "Oh no, if I move my arm this way, I'll hide the object from my camera," it instantly adjusts its camera angle to peek around its own arm. It's like a security guard who constantly shifts their position to make sure they never lose sight of the VIP, even if the VIP is moving behind a pillar.

4. How It Learns: The "Practice Run"

The robot learns this skill using a technique called Diffusion Models (think of it as a "denoising" process).

Imagine the robot is trying to draw a picture of the perfect camera angle. At first, the picture is just static noise (random guesses).
It has a "teacher" (the human video) that gives it a general idea of the style.
But, before it finalizes the drawing, it runs a simulation: "If I pick this angle, will I see the object?"
If the answer is "No," it tweaks the drawing. If the answer is "Yes," it keeps it.
It does this thousands of times in a split second to find the perfect angle that maximizes visibility.

Why This Matters

In the real world, robots often fail because they lose sight of the object they are trying to pick up.

Previous methods tried to copy human head movements, which often failed because human heads move differently than robot arms.
EgoAVFlow learns the goal (keep the object visible) rather than the method (move the head like a human).

The Result: The robot can pick up objects, fold towels, and spray bottles much more reliably, even if the camera angle changes constantly, all without needing to be taught by a human holding a robot controller. It learned by watching videos and figuring out the best way to "look" on its own.

1. Problem Statement

The paper addresses the challenge of training robot manipulation policies using egocentric human videos (videos recorded from a human's perspective) without requiring any robot demonstration data. While these videos offer a scalable source of manipulation data, directly transferring them to robots faces two critical hurdles:

The Embodiment Gap: Human videos contain strong human-specific priors (e.g., rapid saccadic eye movements, vestibulo-ocular reflexes) that are suboptimal or irrelevant for robotic cameras. Naively imitating human head motion often leads to poor visibility for robots.
Active Vision Requirement: Successful manipulation requires active viewpoint control. Robots must dynamically adjust their camera angles to maintain visibility of task-critical objects and avoid occlusions caused by the robot arm or environment. Standard 2D visual features fail here because they entangle appearance with viewpoint, making it difficult to predict where objects will be in 3D space to score visibility.

The core question is: How can a robot learn to manipulate objects and independently adjust its camera viewpoint to maximize visibility, using only human video data and without explicit robot demonstrations?

2. Methodology: EgoAVFlow

The authors propose EgoAVFlow, a framework that learns manipulation and active vision simultaneously through a shared 3D flow representation. The system consists of three diffusion-based components and a test-time optimization strategy.

A. Shared 3D Flow Representation

To bridge the human-robot gap, the method unifies data into a 3D flow space:

Data Preprocessing: Human hand poses (from egocentric video) are mapped to robot end-effector poses. 2D pixel trajectories (tracked via CoTracker3) are unprojected into 3D flow using depth maps and camera poses (estimated via DROID-SLAM).
Embodiment Agnosticism: By discarding view-dependent appearance and focusing on geometric 3D motion, the representation allows policies trained on human data to generalize to robots with different physical forms.

B. Three-Component Architecture

The framework utilizes three diffusion models with a prediction horizon of $T=24$ steps:

Robot Manipulation Policy ( $\pi_r$ ): Predicts future robot action sequences based on the history of 3D flow and proprioception.
Flow Generation Model ( $f$ ): Predicts future 3D flows of scene elements. It is conditioned on the robot's planned actions and camera history to predict how the scene will evolve ("what will move where").
View Policy ( $\pi_v$ ): Predicts future camera viewpoints. Crucially, it does not simply sample from the prior; it uses a visibility-aware reward to refine the output.

C. Visibility-Aware Reward Maximization (Test-Time Optimization)

Instead of blindly imitating human camera motion, EgoAVFlow optimizes the camera trajectory at test time to maximize visibility:

Reward Function: A composite reward is calculated based on:
- Visibility ( $R_{vis}$ ): Determined by raycasting from the candidate camera pose to predicted 3D flow points against a reconstructed environment mesh and the robot's own mesh. It checks for Line-of-Sight (LOS) and Field-of-View (FoV).
- Auxiliary Terms: Includes rewards for being close to query points ( $R_{close}$ ), maintaining a safety margin against occlusions ( $R_{marg}$ ), and avoiding collision with the end-effector ( $R_{safe}$ ).
Soft Value-Based Denoising: The system employs a technique where the diffusion model generates multiple candidate viewpoints. These are scored using the visibility reward, and the highest-scoring candidate is selected via importance resampling. This allows the robot to deviate from the human's head motion whenever the geometry requires a better view.

3. Key Contributions

Shared 3D Flow Representation: A novel interface that bridges the human-robot embodiment gap, enabling joint learning of manipulation and viewpoint control without robot data.
Active Vision Strategy: A viewpoint adjustment mechanism that explicitly optimizes for visibility using geometric reasoning, decoupling camera motion from human imitation while maintaining task relevance.
Robust Performance: Demonstrated that policies conditioned on 3D flow significantly outperform 2D-based or human-imitation baselines under actively changing viewpoints.

4. Experimental Results

The method was evaluated on four real-world manipulation tasks (Spray, Doll, Toilet Paper, Towel) using a WidowX robot for manipulation and a Unitree Z1 robot for camera control.

Comparison with Human Viewpoint Imitation (HVI):
- EgoAVFlow achieved significantly higher success rates (e.g., 20/25 vs. 5/25 on the "Spray" task).
- HVI frequently failed because it tried to mimic human head motions that resulted in occlusions or loss of the target object from the Field of View.
- EgoAVFlow maintained high visibility rewards ( $R_{vis}$ ) throughout the episode.
Comparison with Other Robot Learning Baselines:
- Compared against AMPLIFY (2D flow), EgoZero (3D flow but assumes static objects), and Phantom (synthesized robot views).
- EgoAVFlow outperformed all baselines by 1.8x to 2.5x in success rates.
- Failure Analysis: Baselines suffered heavily from "grasping failures" and "object pose failures" due to distribution shifts when viewpoints changed. EgoAVFlow showed superior robustness in manipulation under varying viewpoints.
Ablation on Viewpoint Necessity: Experiments with fixed cameras showed that no single static viewpoint could maintain full visibility throughout a task, proving the necessity of active, continuous viewpoint adjustment.

5. Significance

Scalability: This work provides a pathway to leverage the vast amount of existing egocentric human video data for robot training without the need for expensive and time-consuming robot data collection.
Active Perception: It moves beyond passive observation, demonstrating that robots can learn to "look where it matters" by reasoning about 3D geometry and future motion, rather than just copying human camera movements.
Generalization: The 3D flow representation proves to be a robust abstraction that allows policies to generalize across different embodiments (human vs. robot) and dynamic scene configurations.

In conclusion, EgoAVFlow establishes a new paradigm for robot learning where geometric visibility reasoning is integrated directly into the policy learning loop, enabling robust manipulation in complex, occlusion-prone environments using only human video data.