🤖 AI

EgoGazeLite: On-Device Egocentric Gaze Prediction for Token-Efficient Multimodal LLM Video Input

EgoGazeLite is a lightweight, on-device dual-process gaze predictor that enables token-efficient egocentric video understanding for multimodal LLMs by accurately cropping video based on predicted gaze, thereby eliminating the need for dedicated eye-tracking hardware while maintaining description quality and operating in real time on consumer hardware.

Original authors: Matteo Stoiber, Niels Buus Lassen

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matteo Stoiber, Niels Buus Lassen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine wearing a pair of smart glasses that can watch the world through your eyes and describe what you are doing. This technology holds the promise of helping a surgeon recall a complex procedure, guiding a mechanic through a repair, or assisting someone with visual impairments by narrating their surroundings. To make this work, the glasses must send a video feed to a powerful artificial intelligence system capable of understanding images and language. However, there is a significant bottleneck: sending high-definition video requires a massive amount of data, far more than a small wearable device can easily transmit or process. The computer systems that understand these videos also struggle when fed too much visual information at once, often getting overwhelmed by the sheer volume of pixels.

Researchers have found a clever way to solve this data problem by focusing only on what the person is actually looking at. Instead of sending the entire wide view, the system can crop the video to show just the small area where the wearer's gaze is fixed. This reduces the amount of data the computer needs to process by about ninety percent, making the task much faster and cheaper. Until now, this approach required a special, expensive piece of hardware built into the glasses to track eye movements precisely. Without this dedicated hardware, the system could not know where to look, leaving the potential of these smart glasses unrealized for everyday consumers.

A team of researchers has now demonstrated that a dedicated eye-tracking sensor is not actually necessary. They developed a lightweight software program that can predict where a person is looking just by analyzing the video stream itself. This program, called EgoGazeLite, runs directly on a standard smartphone and is accurate enough to replace the expensive hardware sensors. The researchers tested this by feeding video clips into an artificial intelligence system, cropping them based on either the real eye-tracking data or the software's prediction. They then asked the artificial intelligence to describe the actions in the video. The results showed that the descriptions generated from the predicted gaze were virtually identical to those generated from the real eye-tracking data. In every test case, the software prediction performed just as well as the hardware measurement, proving that the system can work without the specialized sensor.

The software itself is remarkably efficient. It is small enough to run in real time on consumer-grade hardware, such as the processor in a modern smartphone, completing the prediction and cropping of each video frame in less than twenty-two milliseconds. This speed is crucial because it means the system can keep up with a live video feed without lag. The researchers trained this software on thousands of hours of video from people performing various tasks, from cooking and repairing bicycles to playing soccer and performing CPR. When they tested the system on new, unseen videos, it successfully identified where the person was looking with high precision. The software works by learning two different ways of paying attention: one based on what is visually bright or moving in the scene, and another based on where the person has been looking a moment ago. It combines these two clues to guess the next point of focus.

To ensure their findings were robust, the researchers did not rely on a single method of evaluation. They used two different large language models to generate the video descriptions and then had those descriptions judged by both automated scoring systems and other advanced artificial intelligence models. They compared the quality of the descriptions from the predicted gaze against the ground truth from the hardware sensors and against a simple baseline that just cropped the center of the screen. The results were clear: the predicted gaze produced descriptions that were significantly better than the center crop and statistically indistinguishable from the hardware-based gaze. In fact, the software prediction was so close to the real eye-tracking data that the difference was too small to be measured as significant in any of the ten different test scenarios they ran.

This work removes a major barrier to the widespread adoption of gaze-aware artificial intelligence. By proving that a software model can replace the hardware sensor, the researchers have shown that future smart glasses do not need to be burdened with expensive, power-hungry eye-tracking components to be useful. The system can now be deployed on any device with a camera and a standard processor, opening the door for affordable, real-time assistance in a wide range of daily activities. While the study was conducted on specific datasets and models, the results suggest a fundamental shift in how we can build these systems, moving away from specialized hardware toward intelligent, lightweight software that can understand human attention just by watching what we see.

Technical Summary: EgoGazeLite

Problem Statement

The deployment of Multimodal Large Language Models (MLLMs) for egocentric video understanding on wearable devices is constrained by a strict token budget. MLLMs typically encode visual input via Vision Transformers, where memory and compute costs scale linearly with the number of visual tokens, which in turn depends on video resolution and duration. High-resolution video is prohibitively expensive to transmit and process at scale.

Prior work, specifically GazeLLM, addressed this by cropping video frames around the wearer's gaze point before processing, reducing visual tokens by approximately tenfold while maintaining description quality. However, GazeLLM relies entirely on dedicated eye-tracking hardware, which is absent from current consumer smart glasses. This creates a barrier to entry for token-efficient, gaze-conditioned understanding. The central challenge is to determine if a software-only gaze predictor can accurately substitute for hardware-measured gaze without degrading downstream MLLM performance, while remaining light enough to run on-device within the power and compute budgets of a smartphone.

Methodology

EgoGazeLite Architecture

The authors propose EgoGazeLite, a lightweight dual-process gaze predictor designed for on-device deployment. It builds upon the dual-process structure identified by Huang et al. [10] as a strong non-Transformer baseline, but introduces three architectural modifications to reduce computational cost:

Shared Backbone: Replaces separate spatial and temporal streams with a single shared EfficientNet-Lite4 backbone initialized from ImageNet weights.
Feature-Level Temporal Difference: Replaces dense optical flow with a simple feature-level subtraction ( $\Delta F = \text{BatchNorm}(F_t - F_{t-1})$ ) to capture motion.
Residual Fusion: Replaces nonlinear convolutional fusion with a residual fusion module that operates in logit space, allowing the attention map to sharpen or suppress the saliency prediction without saturation.

The architecture consists of:

Bottom-up Path: Computes a saliency heatmap ( $G^s_t$ ) from visual input alone, incorporating a learnable 2D Gaussian center bias to account for egocentric gaze bias.
Top-down Path: Tracks attention based on fixation history. It extracts channel weights from the previous gaze location, propagates them via a gated LSTM (gated by an online I-DT fixation/saccade decision), and applies them to current frame features to generate an attention heatmap ( $G^a_t$ ).
Fusion: Combines $G^s_t$ and $G^a_t$ using a residual module to produce the final prediction heatmap $G_t$ , from which the gaze coordinate is extracted via argmax.

Gaze-Guided Cropping Pipeline

The system follows the GazeLLM protocol:

Input: 30 fps, 1404×1404 source clips from the Ego-Exo4D dataset.
Processing: Frames are sampled at 1 fps. A fixed 448×448 window is centered on the gaze coordinate (either predicted by EgoGazeLite or ground-truth from an eye tracker).
Conditions: The study compares four conditions: Full-resolution, Ground-truth gaze crop, Predicted gaze crop, and a Center crop (baseline).
Evaluation: Cropped clips are fed into MLLMs (Gemini 2.5 Flash and Pro) to generate step-by-step activity descriptions. These are scored against full-resolution references using automated metrics (BLEU, ROUGE-L, SBERT) and LLM judges (Claude Sonnet 4.6, GPT-4o).

Experimental Setup

Dataset: Ego-Exo4D (Project Aria Gen 1 glasses). Training on seven domains (Cooking, Soccer, Health, Bike Repair, Dance, Bouldering, Music); Basketball was held out.
Substitution Test Set: A held-out set of 138 clips covering six task categories, ensuring no overlap with the training set.
Hardware Evaluation: The full pipeline was tested on an iPhone 15 Pro to measure end-to-end latency.

Key Contributions

Substitution of Predicted for Measured Gaze: The paper demonstrates that gaze predicted by a lightweight model can replace hardware-measured gaze in the GazeLLM pipeline. Across two MLLMs, three automated metrics, and two LLM judges, predicted-gaze crops showed no significant difference from ground-truth-gaze crops in downstream description quality.
Equivalence Confirmation: Using Two One-Sided Tests (TOST), the authors confirmed equivalence in all ten evaluation cells (2 MLLMs × 5 metrics/judges). The difference between predicted and ground-truth conditions was less than half the pooled standard deviation, satisfying the primary equivalence margin.
EgoGazeLite Efficiency: The model achieves this performance with 15.7M parameters and 6.71 GFLOPs, representing an 8.6-fold reduction in FLOPs compared to prior state-of-the-art egocentric gaze models (Huang et al. [10] and Lai et al. [13]).
Real-Time On-Device Performance: The full gaze-and-crop pipeline runs end-to-end in 21.6 ms/frame (approx. 46 FPS) on consumer accelerator hardware (iPhone 15 Pro Neural Engine), making it viable for real-time deployment.

Results

Accuracy: On the in-distribution validation set, EgoGazeLite achieved an Average Angular Error (AAE) of 8.33° and an AUC of 0.9655.
Downstream Quality:
- Both predicted and ground-truth gaze crops significantly outperformed the center-crop baseline across all metrics.
- The difference between predicted and ground-truth crops was statistically non-significant (smallest corrected p-value = 0.054).
- Effect sizes ( $d_z$ ) were consistently small (< 0.17), well within Cohen's convention for negligible effects.
- Equivalence tests (TOST) confirmed that the two conditions are statistically equivalent in all ten cases under a medium-effect margin (0.5× pooled SD), and in four cases under a strict small-effect margin (0.2× pooled SD).
Latency: The EgoGazeLite forward pass accounts for ~41% of the pipeline time (8.9 ms), with the remaining time spent on image processing (cropping, downscaling). The pipeline supports sampling rates up to 30 FPS, far exceeding the 1 FPS requirement for the downstream MLLM.

Significance and Claims

The paper claims that EgoGazeLite removes the strict prerequisite of dedicated eye-tracking hardware for gaze-conditioned egocentric video understanding with MLLMs. By proving that a lightweight, software-only predictor can substitute for hardware-measured gaze without degrading the quality of MLLM-generated descriptions, the work enables token-efficient, gaze-guided video processing on consumer-grade smart glasses and smartphones.

The authors emphasize that this result closes the loop between gaze prediction and MLLM input on-device. While the study is modest about its scope—acknowledging that direct head-to-head accuracy comparisons with prior models under identical splits were not performed, and that generalization to unseen domains or other MLLM families remains to be tested—the core finding is that the substitution is empirically valid for the tested tasks and models. This shifts the paradigm from hardware-dependent gaze systems to software-driven solutions accessible to any first-person camera.