Technical Summary: EgoTraj: Real-World Egocentric Human Trajectory Dataset for Multimodal Prediction

Problem Statement

Accurately forecasting human trajectories from an egocentric (first-person) perspective is critical for applications in humanoid robotics, wearable sensing, and assistive navigation. However, progress in this domain is hindered by a scarcity of real-world egocentric trajectory datasets. Existing trajectory prediction research largely relies on bird's-eye-view (BEV) or static third-person camera datasets (e.g., ETH, UCY, nuScenes), which capture external motion but fail to model how humans perceive, plan, and initiate movement from a first-person viewpoint. Consequently, current forecasting methods rely heavily on motion history and scene context while lacking intention-related cues like gaze. Furthermore, existing egocentric datasets are often limited in scale, restricted to indoor environments, gathered from single participants, or lack synchronized gaze and 6-degree-of-freedom (6DoF) pose data necessary for intention-aware prediction.

Methodology and Dataset Construction

To address these limitations, the authors introduce EgoTraj, a large-scale, multimodal, open dataset recorded in real-world urban environments using Meta Quest Pro (MQPro) headsets.

Data Collection Protocol

• Hardware: Participants wore MQPro headsets operating in full-color passthrough mode. The device integrated a passthrough RGB camera, two infrared eye-tracking cameras, four inside-out tracking cameras, and a 6-axis IMU.
• Participants: The dataset comprises 75 unique participants (ages 18–38, balanced gender, 14 nationalities). Each participant contributed exactly one recording session.
• Environment: Data was collected outdoors in diverse urban settings, including sidewalks, crosswalks, and busy streets. The protocol utilized seven predefined outdoor waypoints (landmarks) to define origin–destination pairs, allowing participants to choose their own routes to encourage naturalistic navigation and spontaneous interactions.
• Duration: The dataset contains 75 sequences totaling 10.7 hours of synchronized data, comprising 1.15 million RGB frames.
• Privacy: All recordings were processed using EgoBlur to automatically detect and blur faces and vehicle license plates.

Multimodal Synchronization

The dataset provides time-synchronized ground truth at 30 Hz, including:

1. 6DoF Head Pose: Position (x, y, z) and orientation (quaternions), along with linear and angular velocity.
2. 3D Gaze Vectors: Per-frame binocular gaze origin and direction vectors.
3. Egocentric RGB Video: H.264-encoded video at 30 fps.
4. Scene Annotations: High-level contextual descriptions generated via a Vision-Language Model (VLM) pipeline using Qwen2.5-VL-7B. These annotations cover environmental context, traffic activity, gaze fixation targets, and inferred short-term movement intent.

Data Processing

A custom pipeline aligned sensor streams (operating at 50 Hz for gaze/pose) with the 30 Hz video timeline using device timestamps. Interpolation techniques (linear for position, SLERP for rotation) were applied to ensure temporal consistency. The final dataset is packaged in HDF5 format with privacy-blurred frames.

Key Contributions

1. EgoTraj Dataset: The first large-scale egocentric trajectory dataset to jointly provide synchronized 6DoF head pose, per-frame 3D gaze vectors, egocentric RGB video, and scene annotations from 75 participants navigating real urban traffic.
2. Benchmark and Evaluation Protocol: The authors established a standardized benchmark for egocentric trajectory forecasting, evaluating several state-of-the-art models on a held-out test split.
3. Multimodal Analysis: Through ablation studies, the authors analyzed the specific contributions of gaze, social interactions, and scene context to trajectory prediction under realistic conditions.

Experimental Results

Quantitative Evaluation

The authors evaluated several baselines, including kinematic models (Constant Velocity, Linear Extrapolation) and learned models (M_Transformer, CXA-Transformer, EgoCast).

• Performance: The adapted EgoCast model achieved the best trajectory forecasting performance with an Average Displacement Error (ADE) of 0.16 m and Final Displacement Error (FDE) of 0.28 m.
• Head Rotation: The CXA-Transformer achieved the lowest head rotation error (0.69), suggesting its cascaded cross-attention mechanism better captures orientation dynamics.
• Baseline Comparison: Learned temporal models consistently outperformed simple kinematic extrapolation, particularly in predicting rotation and handling non-linear motion.

Ablation Studies

Using the CXA-Transformer architecture, the authors analyzed the impact of different input modalities:

• Gaze and Scene: Adding scene segmentation (S) reduced ADE from 0.19 to 0.16. Incorporating gaze (G) further reduced it to 0.15, indicating that visual attention provides anticipatory signals beyond kinematic cues.
• Social Context: Among social representations, full body poses (P) outperformed center points (C) and bounding boxes (B), suggesting fine-grained motion cues are more informative for social interaction modeling.
• Full Multimodal: The combination of ego-motion, pose, scene, and gaze (Y + P + S + G) achieved the best overall performance (ADE: 0.12, FDE: 0.23, Head Rotation Error: 0.58).

Generalization

The models were tested on stricter splits: "Waypoint Held-Out" (unseen origin-destination pairs) and "Unfamiliar" (participants unfamiliar with the area). The full multimodal configuration maintained strong performance across these splits (ADE ~0.14), indicating the model generalizes to new routes and participants rather than overfitting to specific templates.

Qualitative Findings

• Active Transitions: In "active-transition" windows where turning begins within the final 0.5s of observation, multimodal models tracked ground truth significantly better than motion-only baselines. This supports the hypothesis that gaze anticipates locomotor action by 1–2 seconds.
• Failure Cases: The system struggled with abrupt, sharp turns (e.g., ~90° at intersections) where the predictive signal in short-horizon ego-motion history is weak, highlighting the difficulty of deterministic forecasting without strong anticipatory cues.

Significance and Claims

The paper claims that EgoTraj fills a critical gap in the availability of real-world, multimodal egocentric data. By providing synchronized gaze, pose, and scene data, the dataset enables the study of intent-aware trajectory prediction. The results demonstrate that:

1. Visual attention (gaze) is a critical predictive signal that precedes motion, particularly for turns and directional changes.
2. Multimodal fusion (combining ego-motion, social context, scene structure, and gaze) significantly outperforms motion-only baselines in complex urban environments.
3. The dataset supports downstream applications in AR-based perception, assistive navigation for the visually impaired, and socially compliant robotic navigation.

The authors release the dataset, code, and the EgoViz Dashboard (an interactive tool for inspecting synchronized streams) to facilitate future research in egocentric trajectory prediction and vision-language understanding of navigation intent.