💻 computer science

ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation

The paper proposes ARGaze, an autoregressive transformer model that reformulates online egocentric gaze estimation as a sequential prediction task by conditioning current gaze on visual features and a bounded history of recent gaze estimates, achieving state-of-the-art performance on multiple benchmarks.

Original authors: Jia Li, Wenjie Zhao, Shijian Deng, Bolin Lai, Yuheng Wu, RUijia Chen, Jon E. Froehlich, Yuhang Zhao, Yapeng Tian

Published 2026-02-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jia Li, Wenjie Zhao, Shijian Deng, Bolin Lai, Yuheng Wu, RUijia Chen, Jon E. Froehlich, Yuhang Zhao, Yapeng Tian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a camera on your head, recording your day from your own perspective. You are making coffee, reading a book, or organizing your desk. The goal of this research is to teach a computer to guess exactly where you are looking just by watching the video feed, frame by frame, in real-time.

The paper introduces a new system called ARGaze. Here is how it works, explained through simple analogies:

The Problem: The "Time Travel" Trap

Most previous systems tried to guess where you are looking by looking at a whole chunk of video at once—like watching a movie clip from start to finish to guess what happens in the middle.

The Issue: In the real world (like Augmented Reality glasses), you can't wait for the future. You need to know where the user is looking right now as the video happens.
The Flaw: Old methods often "cheated" by peeking at future frames to make their guess accurate. This is like trying to solve a puzzle while looking at the solution key on the back of the box. It works in a lab, but it fails in real-time.
The Instability: Other methods treated every video frame as a separate, isolated moment. This made the computer's guess jump around wildly, like a shaky camera, even when your eyes were steady.

The Solution: ARGaze (The "Autoregressive" Approach)

The authors propose a new way of thinking: Gaze is a story, not a snapshot.

Imagine you are reading a book. If you know where your eyes were a second ago, you have a very strong clue about where they will be next. You don't need to re-scan the whole room every time; you just follow the trail of your attention.

ARGaze works like a detective solving a mystery step-by-step:

The Clue (Visuals): It looks at the current picture (the video frame) to see what objects are there.
The History (The Context): It remembers the last few places you looked (the "Gaze Context Window").
The Prediction: It combines the current picture with the history to guess where you are looking next.

This is called Autoregressive. Think of it like a game of "Telephone" where the message is passed forward. The system only uses information from the past and the present, never the future. This makes it perfect for real-time devices.

Key Features Explained with Metaphors

1. The "Bounded Memory" (The Fixed-Size Notebook)
Old systems tried to remember the entire video history, which requires a massive amount of computer memory (like trying to carry a library in your pocket).

ARGaze's Trick: It uses a small, fixed-size notebook. It only writes down the last few seconds of where you looked. Once the page is full, it erases the oldest note to make room for the new one.
Why it matters: This keeps the computer's memory usage constant and low, so it can run smoothly on lightweight devices like AR glasses without getting hot or slow.

2. The "Tracking Template" (The Magnifying Glass)
Sometimes, the video is blurry or your hands are moving fast, making it hard to see what you are looking at.

ARGaze's Trick: It takes a tiny, high-resolution "magnifying glass" crop of the area where you were looking a moment ago. It uses this close-up to help the computer stay focused on the right object, even if the camera shakes or your hands block the view.
Why it matters: It stops the computer from getting distracted by your moving hands and keeps it locked onto the actual object you are interested in (like a coffee cup, not your hand holding it).

The Results: Why It's Better

The researchers tested ARGaze on three different datasets (cooking videos, daily life videos, and complex tasks).

Accuracy: It guessed the gaze location more accurately than previous methods.
Speed: It is nearly twice as fast as the best existing systems.
Stability: It doesn't jitter. If you stare at a book, the computer's guess stays steady on the book, rather than jumping around.
Robustness: Even when the computer has never seen the specific room or task before (a "new" environment), it still performs well because it relies on the pattern of how eyes move, not just memorizing specific scenes.

Summary

In short, ARGaze changes how computers predict where you are looking. Instead of trying to see the whole movie at once, it acts like a human observer: it watches the current scene, remembers where you just looked, and uses that flow to predict where you will look next. This makes it fast, memory-efficient, and stable enough for real-world use in Augmented Reality and assistive robots.

Technical Summary: ARGaze: Autoregressive Transformers for Online Egocentric Gaze Estimation

Problem Definition

The paper addresses online egocentric gaze estimation, the task of predicting where a camera wearer is looking using only past and current video frames. Unlike third-person gaze estimation, which relies on observable head pose and eye cues, egocentric settings lack direct facial signals. Instead, models must infer visual attention from sparse, indirect cues such as scene content and hand-object interactions.

Current mainstream approaches frame this as an offline, clip-based spatiotemporal integration problem. These methods typically employ 3D convolutions or bidirectional attention mechanisms that require access to future temporal context. This architecture violates real-time constraints, making them impractical for streaming applications like Augmented Reality (AR) and assistive robotics. Furthermore, these models often treat frames as parallelized clips rather than continuous behavioral sequences, failing to capture the physiological stability of human fixations and leading to erratic predictions under visual perturbations.

Methodology: ARGaze

The authors propose ARGaze, a framework that reformulates gaze estimation as a causal autoregressive sequence generation task. Inspired by vision-conditioned autoregressive decoding in vision-language models, ARGaze predicts gaze heatmaps sequentially, conditioning each prediction on current visual features and a bounded history of prior gaze estimates.

The framework consists of five core components:

Scene Encoder: Utilizes a pre-trained vision foundation model (DINOv3) to extract hierarchical spatiotemporal features from the current frame ( $I_t$ ). It also incorporates Tracking-aware Templates, which are localized, high-resolution crops from historical frames centered at previously predicted gaze coordinates. These templates provide specific visual priors to help the model maintain attention on targets during rapid camera motion.
Autoregressive Heatmap Tokenizer: Converts a bounded window of recent gaze history ( $H_{t-k:t-1}$ ) into high-dimensional temporal tokens. This involves downsampling historical heatmaps, projecting them into the latent space via 1 $\times$ 1 convolutions, and adding temporal and token-type embeddings to distinguish between past fixations and current query tokens.
Tracking-aware Template Module: Dynamically crops historical frames based on the soft-argmax of the previous prediction heatmap. This provides the decoder with localized visual context, acting as a tracking-style cue to disambiguate similar objects.
Causal Transformer Decoder: The central processing unit that integrates historical gaze tokens and visual features (global scene + tracking templates) via self-attention and cross-attention layers. Crucially, it operates strictly causally, accessing only current and past observations. It maintains a bounded history window, ensuring a constant memory footprint regardless of video length.
Spatial Reconstruction Head: A convolutional upsampling branch that maps the generated sequence tokens back to the spatial domain to produce the final 2D gaze target heatmap ( $H_t$ ).

Training and Inference:

Training: The model is optimized using a Kullback-Leibler (KL) divergence loss between predicted and ground-truth heatmaps. To bridge the gap between training (using ground-truth history) and inference (using predicted history), the authors employ scheduled sampling.
Inference: The system operates in a streaming manner. For each arriving frame, the decoder generates a prediction conditioned on visual features and the fixed-size history window, ensuring constant memory usage and real-time responsiveness.

Key Contributions

Causal Autoregressive Formulation: The paper introduces ARGaze, which reframes egocentric gaze estimation as a causal sequence prediction task, enabling strictly online inference without future context.
Bounded Memory Architecture: By utilizing a fixed-length gaze history window, the method ensures constant memory usage, supporting efficient long-horizon streaming without the computational bottlenecks of unbounded sequence models.
Enhanced Robustness and Accuracy: The framework integrates explicit temporal modeling with visual scene understanding, demonstrating improved robustness against hand-induced bias and out-of-distribution (OOD) conditions compared to prior methods.

Experimental Results

The authors evaluated ARGaze on three benchmarks: EGTEA Gaze+, Ego4D, and EgoExo4D.

Performance: ARGaze achieved state-of-the-art performance across all benchmarks under online evaluation.
- On EGTEA Gaze+, it achieved an F1 score of 44.48%, outperforming the adapted causal GLC baseline (41.57%) by nearly 3 percentage points.
- On Ego4D, it reached an F1 of 41.82%.
- On EgoExo4D, it achieved 70.58% F1 and 75.66% precision.
Efficiency: In long-video streaming tests on an NVIDIA RTX A6000 GPU, ARGaze demonstrated significant efficiency gains over the GLC baseline:
- Inference Speed: 1.82 $\times$ speedup (88.59 FPS vs. 48.74 FPS).
- Latency: Reduced mean latency from 20.56 ms to 11.29 ms.
- Memory: Reduced CUDA memory allocation by 3.13 $\times$ (168.2 MB vs. 526.3 MB).
Robustness:
- Out-of-Distribution (OOD): ARGaze maintained strong performance across OOD splits (unseen participants, sites, and tasks), suggesting it captures transferable attention dynamics rather than overfitting to domain-specific cues.
- Hand Dependency: Unlike baselines that suffer from "hand-bias" (erroneously tracking moving hands) or performance collapse when hands are occluded, ARGaze maintained stable fixation on task-relevant objects by leveraging historical gaze trajectories to filter visual distractions.
Real-World Testing: In experiments using a Meta Quest 3 headset, ARGaze consistently outperformed the GLC baseline, improving Heatmap F1 from 0.0411 to 0.4667 across daily activities like desk organizing and coffee making.

Significance and Claims

The paper claims that ARGaze addresses a critical gap in existing literature: the misalignment between current modeling paradigms (which treat gaze as parallelized clips) and the physiological nature of human gaze (characterized by stable fixations and saccades). By enforcing strict causality and bounded memory, the authors argue that autoregressive modeling is essential for robust, real-time prediction in streaming environments.

The authors position ARGaze as a practical solution for Augmented Reality and assistive robotics, where systems must anticipate user intent and provide proactive guidance as each frame arrives. The work emphasizes that accurate, low-latency, and memory-efficient gaze estimation is fundamental for human-centric AI systems that understand user intent without relying on explicit head or eye signals. The authors note that while the work advances algorithmic capabilities, future deployment must prioritize privacy and ethical considerations regarding sensitive user intent data.