EgoCampus: Egocentric Pedestrian Eye Gaze Model and Dataset

This paper introduces the EgoCampus dataset and the EgoCampusNet model to address the challenge of predicting egocentric pedestrian eye gaze in real-world outdoor navigation by leveraging a large-scale, diverse collection of gaze-annotated videos captured via Meta's Project Aria glasses.

The Gist

Imagine you are walking through a busy university campus. You aren't just moving your legs; your eyes are constantly scanning the world. You glance at a friend waving, check a street sign, look out for a bicycle, and stare at a beautiful building. Your brain is making thousands of tiny decisions every second about what to look at and what to ignore.

This paper is about teaching computers to understand that exact process: Where do people look when they are walking outside?

Here is the breakdown of the paper, explained with some everyday analogies:

1. The Problem: The "Blind" Robot

Imagine you are building a robot to walk alongside humans. If the robot doesn't know where humans are looking, it's like a blindfolded person trying to dance in a crowded room. It might bump into people or miss important cues.

Most previous research on "eye tracking" was like studying a person sitting in a dark room staring at a computer screen. That's useful, but it doesn't tell us how people look around when they are actually walking outside, dodging obstacles, and navigating a real world.

2. The Solution: The "EgoCampus" Dataset

The researchers created a massive new library of data called EgoCampus.

• The Analogy: Think of this as a giant "Eye-Tracking Movie Collection."
• How they made it: They gave 82 different people a pair of special glasses (Meta's Project Aria). These glasses are like a high-tech spy gadget. They have cameras on the front to see what the person sees, and tiny sensors inside to track exactly where the person's eyes are looking.
• The Content: They walked 82 people along 25 different paths on a university campus (about 6 kilometers total).
• The Result: They captured 32 hours of video, but more importantly, they captured 3.5 million frames of "where the eyes went." They also recorded how the people moved (GPS, speed, head turns) to understand the context.

Why is this special?
Most other datasets are like short clips of people cooking in a kitchen (indoor, static). This dataset is like a continuous movie of people walking through a city, dealing with weather, other people, and changing scenery.

3. The Brain: "EgoCampusNet" (The Prediction Model)

Once they had the data, they built a computer brain (an AI model) called EgoCampusNet.

• The Analogy: Imagine you are teaching a student to predict where a driver will look next.
  • Old way: You show the student a single photo of a road and ask, "Where will the driver look?" (This is hard because you don't know what happened before the photo).
  • EgoCampusNet way: You show the student a video clip of the road leading up to the current moment, plus the current photo. The AI learns that if the car is turning left, the driver will likely look left. If a pedestrian steps out, the driver will look at them.
• How it works: The model looks at the "movie" of the walk (the video history) and the "snapshot" of the current view. It combines them to guess the next place the eyes will land.

4. The Discovery: What do we actually look at?

The researchers analyzed the data and found some interesting patterns:

• The "Center Bias": When people walk straight, they tend to look slightly toward the center of their vision (where they are going). It's like driving a car; you look down the road, not at the dashboard.
• The "Distraction" Factor: When people turn their heads quickly (like when they hear a noise or see a friend), they aren't looking at random things. They are looking at landmarks (buildings, trees) or navigation cues (other people, path signs).
• The Surprise: Even though we think we look at faces when we see people, in a busy walking scenario, people often keep their eyes on the path or distant features, not necessarily locking eyes with everyone they pass.

5. Why Does This Matter?

This isn't just about making cool videos. It's about Safety and Cooperation.

• Self-Driving Cars: If a car knows where a pedestrian is looking, it knows if that pedestrian is aware of the car or if they are distracted by their phone.
• Robot Helpers: If a robot is walking with you, it needs to know if you are looking at a hazard so it can stop, or if you are looking at a map so it can wait for you to catch up.
• Virtual Reality: It helps make VR worlds feel more real by predicting where your eyes naturally want to go.

Summary

The paper is like a map and a compass for understanding human attention.

1. The Map (Dataset): They created the most detailed map of "where people look while walking" ever made.
2. The Compass (AI Model): They built a tool that uses that map to predict where a person will look next.

By understanding how humans navigate the world with their eyes, we can build robots and cars that are safer, smarter, and more natural to be around.

Technical Summary

1. Problem Statement

The paper addresses the challenge of predicting human visual attention during real-world outdoor navigation. While significant research exists on eye gaze in constrained environments (static images, indoor tasks like cooking, or screen-based videos), there is a lack of large-scale datasets and models specifically focused on pedestrian locomotion in outdoor settings.

Existing datasets often lack synchronized eye gaze data, focus on indoor tasks, or have limited subject diversity. Furthermore, current state-of-the-art (SOTA) saliency and gaze prediction models often fail to generalize well to dynamic outdoor navigation without fine-tuning, largely because they do not account for the specific spatio-temporal dynamics of walking pedestrians.

2. Key Contributions

The authors introduce two primary contributions:

A. The EgoCampus Dataset

A novel, large-scale multimodal dataset designed for studying pedestrian attention in outdoor environments.

• Scale & Diversity: Contains 32 hours of video (approx. 3.5 million frames) from 82 distinct pedestrians traversing 25 unique paths (totaling ~6 km) on the Rutgers University campus.
• Hardware: Data was collected using Meta's Project Aria glasses, which provide:
  • High-resolution front-facing RGB video (1408×1408 @ 30Hz).
  • Synchronized eye-tracking coordinates.
  • Rich auxiliary sensor data: IMU, magnetometer, barometer, GPS, and Wi-Fi/Bluetooth signals.
• Privacy: Implemented strict privacy protocols, including blurring non-consenting passersby using the EgoBlur algorithm.
• Unique Features: Unlike previous datasets (e.g., GEETUP), EgoCampus features continuous path trajectories (avg. 108s/clip) rather than segmented clips, and includes bidirectional traversal (forward and reverse) to enable cross-subject comparison in shared spatio-temporal contexts.

B. EgoCampusNet (ECN)

A novel deep learning model for predicting eye gaze from egocentric video.

• Architecture: A spatio-temporal fusion network.
  • Video Backbone: Uses a pre-trained video encoder (X3D) to extract spatio-temporal features from a sequence of frames, capturing motion and context.
  • Image Encoder: Uses a ResNet block to extract features from the specific "query frame" (the frame for which gaze is being predicted).
  • Fusion & Decoding: Features from the video backbone and the query frame are concatenated and passed through a lightweight CNN decoder to generate a gaze heatmap.
• Training: Trained using Adam optimizer with MSE loss on a single NVIDIA RTX 3090 GPU.

3. Methodology Details

Data Processing

• Alignment: Raw VRS files were processed to align 30Hz RGB frames with 30Hz eye-tracking coordinates and high-frequency (1000Hz) IMU data.
• Preprocessing: Video was downscaled to 224×224 for training. IMU data was averaged to match frame timestamps.
• Analysis: A behavioral analysis of high-velocity head movements (detected via IMU) revealed that structural landmarks (buildings, trees) and navigational cues (other pedestrians, path changes) are the primary targets of attention during head turns.

Evaluation Strategy

The authors introduced a novel evaluation metric strategy to address the center bias inherent in egocentric locomotion (where people naturally look forward).

• Standard Metrics: AUC-Judd, Correlation Coefficient (CC), Kullback-Leibler Divergence (KLD), and Similarity (SIM).
• Prior-Relative Weighting: To prevent models from simply predicting the center of the image (which is often correct due to forward walking), the authors weighted metrics based on the Jensen-Shannon Divergence (JSD) between the prediction and the dataset prior. This penalizes models that rely too heavily on the center bias and rewards those that capture active, off-axis attention shifts.

4. Experimental Results

Quantitative Performance

• Baseline Performance: The proposed ECN model achieved competitive results (AUC-J: 0.972, CC: 0.696) with a significantly smaller parameter count (42.5M) compared to SOTA models like GLC (70.2M) or CV MM (420.5M).
• SOTA Comparison: While GLC [27] achieved the highest absolute scores, it relies on massive pre-trained backbones.
• Impact of Weighting: When applying the Prior-Relative Weighting:
  • Models relying heavily on center bias (e.g., Dataset Prior, Center Prior, DeepGazeIIE) saw performance drops of up to 18%.
  • ECN showed a more modest reduction, indicating it captures active attention shifts better than simple priors.
  • GLC demonstrated superior robustness, actually improving in F1 score by 9% under weighted metrics.

Qualitative Results

• ECN produces focal heatmaps centered on the true gaze point.
• EML-Net and DeepGazeIIE tended to overestimate the gaze area.
• ECN correctly learned that in egocentric locomotion, the gaze is often directed forward (center) rather than at passing pedestrians' faces, a nuance some other models missed.

5. Significance and Future Impact

• Resource for Embodied AI: The dataset provides a critical resource for training embodied agents (robots, autonomous vehicles) to navigate shared human spaces by mimicking human attention patterns.
• Navigation Applications: The data supports the development of spatio-temporal environment-sampling methods that mimic human attention, improving efficiency in navigation systems.
• Human-Robot Interaction (HRI): The paper notes a companion dataset, YOPO-Campus, where a robot traversed the same paths. The combination of EgoCampus (human view + gaze) and YOPO-Campus (robot view) creates a powerful benchmark for studying navigation in human-robot systems.
• Methodological Advance: The introduction of "Prior-Relative Weighting" offers a new standard for evaluating gaze prediction models, ensuring they learn genuine attention mechanisms rather than exploiting dataset biases.

In summary, EgoCampus fills a critical gap in computer vision by providing the first large-scale, synchronized, multimodal dataset for outdoor pedestrian gaze, while EgoCampusNet establishes an efficient baseline for predicting attention in dynamic, real-world navigation scenarios.