💻 computer science

TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues

The paper proposes TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model to provide scalable textual scene cues, enabling robust and annotation-efficient gaze target estimation that balances visual saliency with true gaze intent across diverse datasets.

Original authors: Junhui She, Fei Wang, Kun Li, Yiqi Nie, Yuxin Liu, Zhangling Duan, Xun Yang

Published 2026-07-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junhui She, Fei Wang, Kun Li, Yiqi Nie, Yuxin Liu, Zhangling Duan, Xun Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to guess where a person is looking in a crowded photo. It's like trying to figure out which friend in a group selfie is staring at a hidden cookie. For a long time, computer scientists tried to solve this by building "multi-branch" machines. Think of these as a team of detectives, each with a specific job: one checks the person's head, another measures the depth of the room, a third analyzes their pose, and a fourth looks at their eyes. While this sounds thorough, the paper argues it's a bit like hiring a whole police force just to find a lost key. It requires a massive amount of extra training data (annotations) for every single detective, making the system slow to learn and hard to move to new places.

On the other hand, some newer, simpler methods try to be a "one-person show." They look at the picture and guess based on what looks shiny or interesting. But the paper points out a major flaw here: these simple methods often get distracted by "visual noise." They might think a person is looking at a bright red hat because it's the most colorful thing, even if the person is actually looking at a boring gray wall. The paper calls this "semantic defocus"—the computer sees the object but misses the intent.

Enter TextGaze, the new hero of this story. The authors propose a clever middle ground that doesn't need a huge team of detectives but also doesn't get easily distracted. They use a "Large Vision-Language Model" (LVLM)—think of it as a super-smart, well-read librarian who can look at a picture and describe it in words.

Here's how TextGaze works:

The Librarian's Description: Instead of just staring at pixels, the system asks the LVLM to write a short story about the scene. For example, it might say, "The man in the yellow jersey is looking down at his right side."
The Fusion: The system takes these words and mixes them with the visual picture. It's like giving the computer a map (the text) while it's exploring the terrain (the image). This helps the computer understand why someone is looking somewhere, not just where the shiny object is.
The Safety Net: To make sure the computer doesn't forget the story while it's doing the math, the authors added a special "supervision" step. It's like a teacher checking the student's notes at every step of the lesson to ensure they haven't drifted off-topic.

The paper tested this idea on four different datasets, which are basically huge collections of photos and videos used to train these AI brains. The results were quite promising. On the GazeFollow dataset, the new method achieved an AUC of 0.956 (a score where higher is better, with 1.0 being perfect) and an average error distance (Avg L2) of just 0.099 pixels when using the larger "ViT-L" backbone. This means it was very close to the best existing methods, but without needing the extra "depth" or "pose" data that other top models required.

What's really cool is how well it handled new, unseen situations. When the team tested TextGaze on the GOO-Real dataset (which features people shopping in everyday scenarios) without giving it any extra training, it still performed at the top of the charts, scoring an AUC of 0.918 and an L2 error of 0.159. This suggests the "librarian" approach helps the computer generalize better than just memorizing visual patterns.

The authors are careful to note that while their method is streamlined and effective, it's not a magic wand that solves everything instantly. They explicitly rule out the idea that we need complex, multi-branch systems with extra sensors to get good results. They also argue against relying solely on low-level visual tricks. Instead, they suggest that adding a layer of "textual understanding" is the key to bridging the gap between seeing an image and understanding human intent.

In short, TextGaze shows that giving a computer a little bit of "storytelling" ability can help it figure out where someone is looking much better than just staring at the picture alone. It's a playful, efficient, and surprisingly accurate way to teach machines to read the room.

Technical Summary: TextGaze

Problem Statement

Gaze target estimation aims to infer the position of a person's gaze within a scene or determine if the gaze target lies within the image frame. The paper identifies two primary limitations in existing approaches:

Multi-branch methods: These architectures incorporate auxiliary cues (e.g., depth, pose, head features) via separate branches. While they enrich visual representations, they introduce significant annotation burdens, slow convergence due to structural complexity, and hinder domain transfer. Furthermore, they often function as geometric filters that cannot reason about valid gaze targets, leading to "semantic under-specification."
Streamlined/Prompt-tuning methods: Recent approaches attempt to simplify architectures by using positional prompts with foundation models. However, these methods often suffer from "semantic defocus," where the model relies on low-level visual saliency (spatial proximity, visual salience) rather than true gaze intent, resulting in misalignment between predicted attention and actual gaze targets.

The core challenge is to balance the need for semantic guidance with architectural simplicity, moving beyond purely geometric regression toward intent-level reasoning without reintroducing complex multi-branch structures.

Methodology: TextGaze

The authors propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model (LVLM) as scalable semantic guidance. The framework shifts gaze estimation from direct geometric regression to an intent-recognition process via a semantic layer.

1. Architecture Overview

The model consists of three core components:

LVLM-based Textual Scene Cues Module: A frozen LVLM (specifically Qwen2.5-VL-7B-Instruct in the experiments) is prompted to generate structured textual descriptions of the subject's appearance, behavior, and coarse gaze orientation. These natural language outputs are encoded by a frozen BERT text encoder into dense embeddings ( $S_t$ ). This module provides high-level semantic priors (objects, actions) without requiring explicit gaze target localization from the LVLM, which can be unreliable for out-of-frame targets.
Feature Token Sequence Processing Module:
- Visual Stream: A frozen DINOv2 encoder extracts fine-grained scene features ( $F_v$ ). To enhance sensitivity to the target, a learnable head token embedding is added via masked modulation based on the head bounding box.
- Fusion: Text and visual feature sequences are concatenated. A learnable token ( $z_{in/out}$ ) is prepended for the in-/out-of-frame classification task.
Cross-Modal Fusion and Prediction Module:
- Fusion Transformer: A three-layer stacked transformer with post-normalization integrates visual and textual tokens. It employs Multi-Head Self-Attention (MHSA) to allow unrestricted interaction between modalities.
- Hierarchical Text Supervision: To prevent the degradation of task-relevant textual semantics during deep fusion, the authors introduce a layer-wise text consistency supervision mechanism. This enforces semantic alignment across all transformer layers by projecting intermediate text tokens back to the original text encoder features and minimizing the Mean Squared Error (MSE).
- Prediction Heads: A lightweight decoder produces a gaze heatmap (supervised by pixel-wise BCE loss with Gaussian targets) and a scalar probability for in-/out-of-frame status (supervised by binary cross-entropy).

2. Training Objective

The overall loss function combines the heatmap loss ( $L_{heat}$ ), the in/out classification loss ( $L_{in/out}$ ), and the hierarchical text supervision loss ( $L_{text}$ ):
$L_{all} = L_{heat} + \lambda_1 L_{in/out} + \lambda_2 L_{text}$

Key Contributions

Textual Scene Cues: The paper introduces the integration of textual scene cues into gaze target estimation to mitigate semantic defocus. By using LVLMs to generate auxiliary semantic information (appearance, action, intent), the model gains high-level priors without the complexity of multi-branch designs.
Enhanced Benchmarks: The authors strengthen the GazeFollow and VideoAttentionTarget benchmarks by integrating scene-target-person attention fields with LVLM-oriented prompts, generating robust contextual cues that improve text-aided gaze estimation accuracy.
Unified Dual-Stream Framework: A novel cross-modal fusion framework is designed that combines textual embeddings with fine-grained visual representations. This approach mimics human top-down attention, improving spatial reasoning and scene adaptability while maintaining a streamlined pipeline.

Experimental Results

The method was evaluated on four mainstream datasets: GazeFollow, VideoAttentionTarget, ChildPlay, and GOO-Real.

State-of-the-Art Performance: On GazeFollow, TextGaze (using DINOv2 ViT-L) achieved competitive results, obtaining the lowest average L2 error (0.099) and matching the best minimum L2 (0.043), while maintaining a high AUC (0.956). On VideoAttentionTarget, it outperformed existing approaches across AUC, L2 error, and AP metrics for in-/out-of-frame prediction.
Cross-Dataset Generalization: Without fine-tuning or prompt modification, TextGaze demonstrated robust generalization:
- On GOO-Real (real-world shopping scenarios), it achieved state-of-the-art performance in both AUC and L2 metrics.
- On ChildPlay (children's gaze), it outperformed all baselines in the AP metric for in-/out-of-frame classification, highlighting the effectiveness of LVLM-derived cues for presence detection.
Ablation Studies: Experiments confirmed that the transformer-based fusion strategy outperforms additive and cross-attention fusion. Furthermore, the simultaneous activation of hierarchical text supervision, visual residual connections, and specific positional embeddings yielded optimal performance, validating the complementary nature of these components.

Significance and Claims

The paper claims that TextGaze provides a streamlined alternative to traditional multi-branch designs. By leveraging LVLMs as accessible auxiliary guidance, the method successfully balances the trade-off between the high annotation costs of multi-branch systems and the semantic limitations of streamlined visual-only models.

The authors emphasize that gaze following is inherently an intent-level reasoning problem rather than a purely geometric regression task. TextGaze addresses this by shifting the paradigm from "pixels to coordinates" to "pixels to semantic intent to gaze target." The results validate that LVLMs can effectively connect vision with high-level intentions, offering a scalable solution for gaze estimation in complex scenes without the need for redundant branches or extensive domain-specific annotations.