💻 computer science

EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks

EgoLive is a large-scale, high-quality egocentric dataset featuring diverse, real-world human task routines and precise multi-modal annotations designed to accelerate the development of generalizable robot manipulation models.

Original authors: Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, Kangliang Chen, Xing Pan, Shuaiyan Liu, Daming Wang, Tao An, Jiayi Li, Shi

Published 2026-04-28

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yihang Li, Xuelong Wei, Jingzhou Luo, Yingjing Xiao, Yibo Bai, Guangyuan Zhou, Teng Zou, Chenguang Gui, Jiajun Wen, He Zhang, Kangliang Chen, Xing Pan, Shuaiyan Liu, Daming Wang, Tao An, Jiayi Li, Shibo Jin, Wanwan Zhang, Tianyu Wang, Boren Wei, Zhixuan Huang, Fangsheng Liu, Ruodai Li, Hui Zhang, Anson Li, Yicheng Gong, Peng Cao, Jiaming Liang, Liang Lin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "First-Person Training Manual" for Future Robots

Imagine you are teaching a toddler how to tie their shoes or fold laundry. You wouldn't just show them a picture of a shoe; you would sit them down, let them watch your hands move from your own perspective, and describe exactly what you are doing: "I’m grabbing the left lace, now I’m looping it..."

Currently, teaching robots is much harder. Most robots learn by being "remote-controlled" by humans (like playing a video game) or by watching videos from a fixed camera on a wall. This is slow, expensive, and doesn't capture the "messy" reality of how humans actually move in the real world.

EgoLive is a massive new project designed to change this. It is essentially a giant, high-definition "training manual" written from a human's point of view.

The Three Big Innovations

To understand why EgoLive is a big deal, think of it through these three metaphors:

1. The "GoPro for Super-Brains" (The Hardware)

Instead of using bulky VR headsets that make people feel dizzy or clumsy, the researchers created a custom, lightweight headset called JoyEgoCam.

The Analogy: Imagine trying to learn to cook while wearing heavy, oversized ski goggles. You’d be awkward! JoyEgoCam is like a pair of lightweight, high-tech sunglasses. It lets people go about their normal day—cleaning a house, working in a shop, or organizing a pharmacy—while recording everything in stunning 3D detail. It sees the world exactly like our eyes do.

2. The "Digital Librarian" (The Annotation)

A video of someone cleaning a window is just a bunch of moving pixels to a computer. To learn, a robot needs to know what is happening. The researchers built an automated "Librarian" (an AI pipeline) that watches the video and writes incredibly detailed notes.

The Analogy: If you watched a movie with the sound off, you might guess what’s happening. But if every scene had subtitles saying, "The right hand is grasping a blue sponge to wipe the glass," you’d learn much faster. EgoLive does this automatically, labeling the hands, the objects, the depth (how far away things are), and even the specific steps of a task.

3. The "Real-World Jungle Gym" (The Diversity)

Most robot datasets are recorded in sterile laboratories or controlled kitchens. They are like training a marathon runner on a flat, indoor treadmill.

The Analogy: EgoLive is like taking that runner out into the actual mountains, through mud, rain, and crowded streets. Because the data was collected in real stores, homes, and workplaces, the robots won't be "surprised" when they encounter a cluttered counter or a weirdly shaped bottle in a real human house.

Why does this matter to you?

Right now, robots are mostly stuck in factories doing the same repetitive motion over and over. They struggle with "generalization"—the ability to take what they learned in one place and apply it to another.

By providing 1,680 hours of high-quality, "first-person" human life, EgoLive gives researchers the fuel they need to build robots that can actually help us. We are moving closer to a world where a robot doesn't just "know how to move," but understands how to interact with our world, our objects, and our lives.

Technical Summary: EgoLive: A Large-Scale Egocentric Dataset from Real-World Human Tasks

1. Problem Statement

The development of generalizable robot manipulation models is currently bottlenecked by the scarcity of large-scale, high-quality datasets. Existing data collection methods face a trade-off between fidelity and scalability:

Real-robot teleoperation (e.g., ALOHA) provides high physical fidelity but is expensive and difficult to scale.
Universal Manipulation Interfaces (UMI) offer better scalability but are often tied to specific robot embodiments and struggle with dexterous manipulation.
Existing Egocentric Datasets (e.g., Ego4D, EPIC-KITCHENS) are either too "generalist" (lacking the geometric/kinematic signals needed for fine-grained manipulation) or too "laboratory-centric" (lacking the environmental diversity and long-horizon complexity of real-world service tasks).

2. Methodology

The authors propose EgoLive, a framework designed to collect and annotate high-fidelity human egocentric data in unconstrained, real-world environments. The methodology is divided into three pillars:

A. Hardware (JoyEgoCam): To ensure natural movement and high data quality, the team developed a custom, lightweight, ergonomic head-mounted device. It features stereo RGB cameras (providing human-like binocular vision with a $130^\circ \times 130^\circ$ FOV) and an integrated 200Hz IMU for precise camera pose estimation. This design minimizes user intrusion, allowing for natural, long-term data collection.
B. Automated Annotation Pipeline: To handle the massive scale of data, the authors developed an automated pipeline that generates multi-modal signals:
- Motion Tracking: Uses a two-stage method (based on HaMeR and MANO) to estimate 6-DoF hand/wrist trajectories, synchronized with camera ego-motion estimated via ORB-SLAM3.
- Semantic Understanding: Employs a combination of object detection (BoT-SORT), segmentation (SAM2), and a fine-tuned Qwen3-VL-32B LLM to generate hierarchical, fine-grained instruction captions (focusing on hand, object, and action).
- 3D Reconstruction: Utilizes FoundationStereo to perform dense depth reconstruction and stereo optimization, ensuring consistent 3D hand keypoints and scene geometry.
C. Data Collection Strategy: Data is collected in "in-the-wild" service-oriented scenarios (home service, retail, pharmacy, etc.), ensuring high ecological validity and task diversity.

3. Key Contributions

Scale: It is the largest open-source annotated egocentric dataset focused on real-world task-oriented human routines, comprising 1,680 hours of stereo video (60 FPS) across 65,866 episodes and 346 tasks.
High-Fidelity Multi-modal Annotations: Unlike generalist datasets, EgoLive provides 6-DoF motion tracking, fine-grained semantic segmentation, 3D scene reconstruction, and structured language descriptions.
Superior Diversity: The dataset covers a wide spectrum of manipulation-intensive domains (cleaning, logistics, organization) with a natural "long-tail" distribution of objects and actions.

4. Results and Evaluation

Hand Reconstruction Accuracy: Qualitative and quantitative evaluations show that EgoLive's 2D and 3D keypoint annotations are significantly more accurate and spatially consistent than previous datasets like EgoDex, effectively handling dynamic occlusions.
Depth Precision: The depth reconstruction model achieves millimeter-level accuracy within the typical human operating range (under 0.9m), as validated by checkerboard calibration tests.
Semantic Richness: t-SNE visualizations and word-frequency analysis confirm that EgoLive occupies a broader representation manifold and exhibits higher semantic coverage (objects, actions, and attributes) than existing datasets.
Captioning Quality: Using an "LLM-as-a-Judge" protocol, the authors demonstrated that their instruction captioning module provides highly consistent and granular descriptions of hand-object interactions.

5. Significance

EgoLive serves as a critical bridge between human behavior and robotic action. By providing a scalable source of high-quality, multi-modal, and diverse human demonstrations, it enables research in:

Human-to-Robot Transfer: Reducing the embodiment gap for policy learning.
Foundation Models: Pre-training Vision-Language-Action (VLA) models with real-world complexity.
Generalizable Embodied AI: Moving robots from controlled laboratory settings to unpredictable, open-world service environments.