💻 computer science

EgoAERO: Learning Dexterous Manipulation from a Single Egocentric Video without Object Assets

EgoAERO is a novel framework that enables robots to learn dexterous manipulation from a single egocentric RGB-D video without requiring pre-scanned object assets by reconstructing contact-consistent trajectories and introducing the large-scale EgoDex-R dataset.

Original authors: Yichen Niu, Haoran Lv, Xinrui Zhang, Xueyao Wan, Shiyu Gao, Ying Ai, Hui Xu, Yongqi Hu, Hengyi Zhang, Yang Xie, Zhaxizhuoma, Yue Zhao, Zhenshan Bing, Yan Ding, Jianxing Liu

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yichen Niu, Haoran Lv, Xinrui Zhang, Xueyao Wan, Shiyu Gao, Ying Ai, Hui Xu, Yongqi Hu, Hengyi Zhang, Yang Xie, Zhaxizhuoma, Yue Zhao, Zhenshan Bing, Yan Ding, Jianxing Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a highly skilled robot hand how to perform a delicate task, like peeling an orange or picking up a specific tool. Usually, to do this, you need a perfect 3D blueprint (a CAD model) of the object beforehand, or you need to scan the object in a lab first. This is like trying to teach someone to drive a car by only showing them a video, but without ever telling them what the steering wheel or pedals look like.

EgoAERO is a new system that changes the rules. It allows a robot to learn complex, dexterous tasks just by watching one single video recorded from a person's point of view (like a GoPro on their head), even if the system has never seen that specific object before and has no 3D model of it.

Here is how EgoAERO works, broken down into simple steps:

1. The "Smart Detective" (Semantic Preprocessing)

First, the system watches the video and asks a "Smart Detective" (an AI called an MLLM) to figure out the story.

The Problem: In a video, a person might be holding a cup, but the video doesn't explicitly say, "I am holding a red mug."
The Solution: The AI looks at the video and the task description to identify what object is being used and what the goal is. It then highlights the object in the video, preparing it for the next step.

2. The "3D Sculptor" (Asset-Free Reconstruction)

This is the magic part. Usually, to teach a robot, you need a pre-made 3D model of the object. EgoAERO doesn't have that. Instead, it builds the 3D model on the fly while watching the video.

The Challenge: The person's hand often blocks the view of the object (occlusion), and the object might have a boring, plain surface that is hard to track.
The Solution: The system acts like a sculptor who can guess the full shape of a statue even if parts of it are hidden. It stitches together thousands of snapshots from the video, uses the depth information (how far away things are), and mathematically "fills in the blanks" to create a smooth, 3D mesh of the object. It also figures out exactly how the object is moving and rotating in 3D space.

3. The "Steady Cam" (Ego Motion Compensation)

Since the camera is on the person's head, every time they turn their head, the whole world in the video moves.

The Problem: If the person turns their head, the object looks like it's sliding across the table, even if it's sitting still.
The Solution: EgoAERO acts like a "Steady Cam" operator. It calculates how the person's head moved and subtracts that motion from the video. This ensures the robot sees the object moving only because the person moved it, not because the camera moved.

4. The "Physics Fixer" (Adaptive Contact Optimization)

Sometimes, the video reconstruction might look slightly "wrong" physically. For example, the robot might think the person's finger is floating slightly above the object, or the object is slightly inside the finger (which is impossible in real life).

The Solution: The system runs a quick "physics check." It gently nudges the hand and object positions so they touch realistically, just like a human would. It fixes tiny errors like "floating fingers" or "ghost penetrations" to make sure the data looks physically possible.

5. The "Two-Step Dance" (Policy Learning)

Once the system has a clean, 3D, physics-corrected video of the hand and object, it teaches the robot how to do it. It does this in two stages:

Step 1: The Dance Partner: The robot first learns to simply copy the human's hand movements. It doesn't worry about the object yet; it just learns to move its fingers and wrist like the human did.
Step 2: The Fine-Tuning: Now that the robot knows how to move, it learns a "residual" (a small correction). It uses the 3D object data it built earlier to make tiny adjustments. If the human's hand slipped a little, or if the object needs to be gripped tighter, the robot learns to make those small, precise tweaks to ensure the task is done successfully.

The Result: EgoDex-R

The researchers didn't just build the system; they used it to create a massive library of these "fixable" videos called EgoDex-R. It contains over 4 million frames of people doing dexterous tasks with everyday objects, all without needing pre-scanned 3D models.

Why This Matters

In simple terms, EgoAERO turns a messy, real-world video into a perfect, physics-compliant instruction manual for a robot.

Before: You needed a perfect 3D blueprint of every object you wanted a robot to touch.
Now: You just need a video of a human doing it. The system figures out the rest.

The paper shows that robots trained this way can perform tasks almost as well as robots trained with perfect 3D blueprints, proving that you don't need expensive pre-scanning to teach robots complex hand skills. They tested this in simulations and on a real robot (a Unitree G1 with an Inspire Hand), and it worked.

Technical Summary: EgoAERO

Problem Statement

Egocentric RGB-D videos offer a natural and scalable source for training dexterous manipulation policies, capturing human hand motions, object interactions, and contact-rich behaviors in everyday environments. However, existing ego-centric datasets (e.g., EPIC-KITCHENS, Ego4D) typically lack critical information required for robot learning, such as the 6-DoF pose, geometry, and contact states of manipulated objects. Conversely, datasets with 3D hand-object annotations (e.g., HOI4D, H2O) often rely on pre-scanned object assets, CAD models, or multi-view capture, limiting their scalability to arbitrary daily objects and natural collection settings. Consequently, there is a gap in frameworks capable of converting a single, natural egocentric demonstration into executable robot policies without relying on prior object assets.

Methodology: EgoAERO Framework

EgoAERO is a framework designed to reconstruct contact-consistent hand-object trajectories from a single egocentric RGB-D video without object assets, subsequently transferring these trajectories to a dexterous robot via two-stage policy learning. The pipeline consists of two main components:

1. Asset-Free Egocentric Hand-Object Reconstruction

This module recovers structured trajectories from raw video through four sequential steps:

MLLM-Guided Semantic Preprocessing: A Multimodal Large Language Model (MLLM) performs lightweight semantic parsing on keyframes to identify the target manipulated object, supporting objects, and task goals. It generates prompts for SAM3 segmentation and selects a less-occluded keyframe as an initialization seed. The MLLM does not estimate poses or geometry directly but configures the pipeline.
Asset-Free Object Tracking and Reconstruction: To handle hand occlusions and low-texture surfaces without CAD models, the system employs:
- Coarse Pose Initialization: Back-projects visible object regions into local point clouds and uses RANSAC on RGB-D correspondences to estimate a coarse 6-DoF pose.
- Keyframe Memory-Pool Pose Optimization: Maintains a pool of historical keyframes with reliable observations. It constructs a local pose graph and jointly optimizes object poses using constraints from cross-frame correspondence, geometry, surface consistency, silhouette, and pose priors.
- Neural Field-Guided Coarse-to-Fine Mesh Reconstruction: An online neural object field generates a temporally consistent coarse mesh. This is refined by SAM3D using original RGB-D observations to recover fine surface details, which are then aligned to the canonical frame.
Egocentric Hand Pose Estimation: Utilizes HaWoR for initial MANO-based hand pose estimation. A lightweight correction is applied to the global hand translation using RGB-D depth maps to mitigate monocular depth bias, ensuring the hand mesh aligns stably with the scene geometry.
Ego Motion Compensation: An RGB-D SLAM backend estimates the camera trajectory. All frames are transformed into a fixed table frame to remove spurious drift caused by head motion. Hand masks are used to down-weight dynamic hand pixels during SLAM to ensure background stability.
Adaptive Contact Optimization: This module corrects physical inconsistencies (e.g., fingertip floating, penetrations) without altering the object trajectory or re-solving MANO parameters. It applies bounded corrections to the global hand translation and local finger vertices in active manipulation windows, ensuring contact consistency while preserving the original demonstration motion.

2. Two-Stage Residual Policy Learning

The reconstructed trajectories are converted into robot policies using a two-stage approach:

Stage I (Hand Trajectory Tracking): A policy learns to track the reconstructed human hand motion (wrist and finger keypoints) using kinematic retargeting as a warm start. This establishes a stable hand control prior without requiring successful object manipulation.
Stage II (Object-Contact Residual Policy): A residual policy learns small corrections ( $\Delta a^R_t$ ) to the base hand-tracking action. It incorporates object states (pose, velocity, geometry encoding) and contact feedback. The reward function combines hand imitation with object trajectory tracking and contact constraints, enabling the robot to drive the object along the reference trajectory with stable contact.

Key Contributions

EgoAERO Framework: The first system to learn dexterous manipulation from a single egocentric RGB-D video without requiring object assets (CAD models or pre-scanned meshes).
Asset-Free Reconstruction Pipeline: A robust pipeline featuring MLLM-guided initialization, asset-free object tracking under occlusion, neural-field-guided mesh reconstruction, ego-motion compensation, and adaptive contact optimization.
EgoDex-R Dataset: A large-scale egocentric dataset containing 4.3 million RGB-D frames, 5,600 manipulation sequences, and over 1,000 object instances. It includes contact-consistent hand-object trajectories with annotations for object pose, geometry, depth, SLAM, and contact quality.
Online Quality Assessment: A mechanism to evaluate the physical usability of reconstructed trajectories in real-time, deciding whether to accept, repair, or recapture demonstrations.

Experimental Results

The authors evaluated EgoAERO through simulation and real-world experiments:

Simulation (EgoDex-R): EgoAERO achieved a 49.5% success rate, significantly outperforming baselines using only hand pose (9.8%) and a version without adaptive contact optimization (36.2%). This demonstrates the necessity of object state and contact information for dexterous manipulation.
Simulation (HOI4D Comparison): When compared to trajectories reconstructed using object CAD assets, EgoAERO's asset-free reconstructions achieved comparable performance (44.7% success rate vs. 43.3% for CAD-based raw data). In some metrics, EgoAERO performed better, likely due to the adaptive contact optimization improving physical consistency.
Real-World Execution: The framework was successfully deployed on a Unitree G1 humanoid robot with an Inspire Hand. The system transformed a single human demonstration into a contact-consistent trajectory that the physical robot could execute, validating the transferability of the learned policies.

Significance and Claims

The paper claims that EgoAERO bridges the gap between natural human demonstrations and executable robot policies by eliminating the dependency on object assets. By reconstructing contact-consistent hand-object trajectories from a single video, the framework enables scalable dexterous robot learning in everyday environments. The results suggest that asset-free reconstructions can achieve downstream policy performance close to those derived from high-fidelity CAD-based reconstructions, making natural egocentric data a viable and scalable source for training complex manipulation skills.

Limitations

The authors acknowledge that the current pipeline primarily targets single-hand manipulation. Performance may degrade under conditions of severe occlusion, reflective objects, fast motion, or inaccurate hand/segmentation estimates. Furthermore, the policy learning is currently task-specific and requires simulation training, leaving cross-task generalization and reduced training costs as areas for future work.