Technical Summary: MobileEgo Anywhere

Problem Statement

The advancement of Vision Language Action (VLA) models has created a critical demand for large-scale, long-horizon egocentric datasets. While existing datasets (e.g., Ego4D, Epic Kitchens) provide thousands of hours of video, they are often limited to short episodes (typically minutes) and lack the precise, continuous 6-DoF pose tracking required for robotic policy learning. Conversely, datasets with precise tracking (e.g., EgoScale) often feature very short, disjointed episodes. Furthermore, high-fidelity data collection traditionally relies on specialized, expensive hardware (e.g., Meta Project Aria, motion capture rigs) or teleoperation, creating significant barriers to scaling data diversity and volume. Current scaling laws suggest that reaching the next frontier of generalizable robotics requires an order-of-magnitude increase in data volume and diversity, which is currently hindered by these hardware and temporal limitations.

Methodology

The authors present MobileEgo Anywhere, a framework designed to collect robust, hour-plus egocentric trajectories using commodity mobile hardware (specifically LiDAR-enabled iOS devices like the iPhone Pro).

1. Data Collection System

• Hardware: A head-mounted iPhone Pro captures a first-person perspective of the user's hands and workspace.
• Control: A custom mobile application manages recording via voice commands ("start"/"stop"), enabling hands-free operation for naturalistic daily activities.
• Sensors: The system leverages the device's native sensor suite, synchronizing RGBD streams with high-frequency IMU data.
• Pose Tracking: The framework utilizes ARKit for real-time sensor fusion, generating high-fidelity 6-DoF camera poses and per-frame depth maps. Data is exported in the MCAP format.

2. Processing Pipeline

The authors provide an open-source Python suite to convert raw logs into standardized, training-ready formats:

• 3D Hand Trajectory Estimation: The system employs WiLoR, an end-to-end network optimized for 3D hand pose estimation in unconstrained environments. It uses the MANO parameterization to represent hand joints. Relative 3D coordinates are localized into a global coordinate system by unprojecting 2D keypoints using ARKit depth maps and camera poses.
• Atomic Action Labeling: An automated annotation pipeline partitions video into contiguous spans. A Vision Language Model (VLM) generates short imperative sentences describing the action, including object modifiers (color, material, size) and spatial prepositions, constrained by prompt design to ensure specificity.
• Hierarchical Task Instructions: A language model (DeepSeek V4 Flash) organizes atomic spans into a three-level instruction tree:
  1. Episodes: Groups of temporally contiguous spans sharing a common activity.
  2. Sub-goals: Clusters of related episodes.
  3. Session Goals: A high-level goal grounded in concrete objects across the entire session.

Key Contributions

1. Dataset Release: A novel dataset comprising 200 hours of diverse, long-form egocentric data from 16 contributors, featuring persistent state tracking. The longest continuous session recorded is approximately 108 minutes.
2. Open-Source Software: Release of a mobile application for data collection and a comprehensive processing pipeline for converting raw mobile captures into standardized formats (including 3D hand poses, atomic labels, and hierarchical instructions).
3. Commoditized Infrastructure: A demonstration that commodity smartphones can replace specialized robotics hardware for collecting long-horizon, high-fidelity egocentric data, effectively democratizing the data collection process.

Results and Evaluation

The paper validates the system through several metrics:

• Long-Term Drift Evaluation: Using Aruco markers revisited at temporal midpoints and endpoints of sessions, the system demonstrated minimal drift. Errors were measured at 0.4–0.7 cm for kitchen/living space activities and 1.0–1.5 cm for whole-house activities, representing less than 0.1% of the trajectory length.
• 3D Hand Pose Consistency: Evaluated across 98 sessions (1.19M frames) using ground-truth-free metrics:
  • Bone Length Constancy: The median coefficient of variation (CV) for bone lengths was <1% (excluding the pinky tip, which showed ~7.5% CV due to its short physical length).
  • Joint Angle Plausibility: Over 99.99% of estimated joint angles fell within published biomechanical limits.
  • Wrist Dynamics: Median wrist velocities (0.27–0.34 m/s) and accelerations aligned with typical ranges for activities of daily living, with no systematic artifacts like teleportation spikes.
• Hierarchical Instruction Quality: The pipeline processed 354 sessions, generating 45,415 atomic spans grouped into 5,570 episodes and 1,298 sub-goals. 87% of sessions passed all structural invariants with zero issues. The hierarchical structure naturally exhibited distinct temporal scales (5s atomic spans, 42s episodes, 3.9 min sub-goals, 15.5 min sessions) without artificial prompting.
• Cost Efficiency: The total cost for processing the hierarchical structuring of all 354 sessions was $1.29.

Significance

The paper claims that MobileEgo Anywhere addresses the "bottleneck" of robotics data collection by removing high hardware barriers. By enabling the massive-scale acquisition of long-horizon data across varied global environments using standard consumer hardware, the work aims to accelerate the development of generalizable robotic policies. The authors posit that maximizing spatial and temporal reasoning capabilities during pretraining with such data can significantly reduce the data requirements for resource-intensive downstream fine-tuning, thereby bridging the gap between current institutional capabilities and the data volumes required for next-generation VLA models.