💻 computer science

Gaze-to-text Generation: Beyond Categorical Decoding of Human Attention

This paper introduces Gazette, a novel framework that leverages multimodal large language models and synthetic "think-aloud" transcripts to decode human gaze scanpaths into free-form natural language descriptions of goals, moving beyond traditional categorical decoding to capture the open-ended nuances of human intentions.

Original authors: Sounak Mondal, Dimitris Samaras, Gregory Zelinsky, Minh Hoai

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sounak Mondal, Dimitris Samaras, Gregory Zelinsky, Minh Hoai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your eyes are like a flashlight beam in a dark room. When you look at something, that beam doesn't just sit still; it darts around, landing on different spots for tiny fractions of a second. This pattern of movement is called a "scanpath." For a long time, scientists have tried to figure out what you are thinking just by watching where your flashlight beam lands. Usually, they've treated this like a multiple-choice quiz. They'd ask, "Is the person looking for a cat or a dog?" and the computer would pick one answer from a fixed list. But human thoughts are messy and creative. Sometimes you aren't just looking for a "dog"; you are looking for "the scruffy golden retriever wearing a red bandana that is hiding behind the couch." The old way of asking questions was too simple to capture that kind of detail. This is where a new field called "gaze decoding" comes in, trying to translate those eye movements into the rich, open-ended stories happening inside our heads.

Enter a new research paper that introduces a clever new way to solve this puzzle. The authors, working at Stony Brook University and Adelaide University, have built a system called Gazette. Instead of forcing the computer to pick from a multiple-choice list, Gazette is designed to write a free-form story about what a person is looking for, using natural language just like a human would. Think of it as upgrading from a rigid checklist to a creative writer who can describe exactly what you see.

The big problem the researchers faced was that everyone's eyes move a little differently, even when they are looking for the same thing. If ten people are asked to find a "red car," ten different people might look at the car in ten slightly different ways. Some might look at the wheels first, others at the windows. If you just show a computer one person's eye movements, it gets confused by all those personal habits. It's like trying to guess a song by listening to one person humming it off-key; you might hear the tune, but the noise makes it hard to be sure.

To fix this, the team came up with a smart trick involving a "think-aloud" strategy. They realized that while everyone's eye movements are unique, the reason they are moving is the same. So, they used a powerful AI (GPT-4) to look at the eye movements of many different people all trying to find the same object. The AI acted like a detective, ignoring the individual quirks and finding the common pattern—the "secret recipe" for how humans look for that specific thing. The AI then wrote a "think-aloud transcript," which is basically a story explaining the strategy: "First, look at the big shapes, then zoom in on the red parts."

They taught their new model, Gazette, to read these AI-generated stories along with the eye movements. This helped the model learn to separate the "goal" (what the person is looking for) from the "noise" (the person's unique way of looking). When they tested it, Gazette was much better at guessing the goal than previous methods. For example, when asked to find a specific car in a picture, Gazette didn't just say "car." It could say, "The black car in the top right corner," and it did this with high accuracy, even when the task was tricky.

The paper shows that by teaching the computer to understand the strategy behind the eye movements, rather than just the movements themselves, we can decode human intentions with much more detail. While the system works best when people agree on what they are looking for (like finding a specific object), it suggests that we are getting closer to a future where computers can understand our goals just by watching where we look, opening doors for better assistive technologies and a deeper understanding of how we interact with the world.

Technical Summary: Gazette – Gaze-to-Text Generation

Problem Definition
The paper addresses the limitation of existing gaze decoding methods, which typically frame the task as a discriminative classification problem over a fixed set of predefined categories (e.g., identifying a specific object from a limited vocabulary). The authors argue that such categorical approaches are inherently constrained, lacking the fine-grained detail and open-ended expressiveness required for real-world applications like e-commerce, educational analysis, and assistive technologies. Instead, the paper proposes a novel learning problem: Gaze-to-Text Generation. The goal is to decode human gaze scanpaths into free-form, natural language descriptions that capture the rich nuances of human goals and intentions, ranging from simple object labels to complex, context-rich referring expressions or questions.

Methodology: The Gazette Framework
The authors introduce Gazette, the first generative framework for gaze-to-text decoding, built upon Multimodal Large Language Models (MLLMs).

Architecture: Gazette extends the LLaVA-1.5-7B foundation model. It takes an input image $I$ and a gaze scanpath $S$ (encoded textually as a sequence of fixations with coordinates and durations) along with a language instruction. The model outputs a natural language description $D$ of the cognitive context, which includes:
- Coarse Behavior Type: The category of goal-directed behavior (e.g., Target-Present/Absent Visual Search, Object Referral, Visual Question Answering).
- Fine-Level Stimulus: The specific goal (e.g., a target category, a referring expression, or a question).
The Challenge of Individual Variability: A core challenge identified is that gaze signals are confounded by individual differences. For the same goal, different participants exhibit different scanpaths. Standard instruction tuning on image-scanpath-goal triplets often fails because the model struggles to disentangle the goal-driven signal from participant-specific noise.
Think-Aloud Transcript Generation (Auxiliary Task): To address the disentanglement problem, the authors propose a novel auxiliary training objective called ThinkAloud.
- Concept: The authors leverage the theoretical insight that while individual scanpaths vary, the underlying goal-driven attention allocation strategy remains invariant across participants performing the same task on the same image.
- Implementation: They use GPT-4 to analyze scanpaths from multiple participants sharing the same goal and image. GPT-4 synthesizes "think-aloud transcripts" that describe the common spatiotemporal patterns and top-down attention allocation strategies, effectively filtering out individual idiosyncrasies.
- Training Data: These transcripts serve as pseudo-annotations for an auxiliary task where the model learns to generate the attention allocation strategy given an image and a single scanpath. The transcripts include three "idea units": (i) top-down attention allocation explanation, (ii) inferred target location (bounding box), and (iii) scanpath length.
- Instruction Tuning: Gazette is fine-tuned using Low-Rank Adaptation (LoRA) on a combination of the primary GazeDec task (decoding the goal) and the auxiliary ThinkAloud task (generating the strategy explanation).

Key Contributions

Task Formulation: The introduction of unconstrained gaze decoding where goals are expressed in open-vocabulary natural language rather than fixed categories.
Gazette Framework: The development of the first text-generative MLLM-based framework specifically designed for decoding top-down attention.
Auxiliary Objective: The proposal of the ThinkAloud transcript prediction task. This strategy leverages the shared invariance of goal-driven signals across multiple observers to help the model filter out individual variability, significantly improving decoding performance.

Experimental Results
The authors evaluated Gazette on three datasets: COCO-Search18 (Visual Search), RefCOCO-Gaze (Object Referral), and AiR-D (Visual Question Answering).

Generative Tasks (Object Referral & VQA):
- Gazette significantly outperformed baselines (including standard LLaVA-1.5 and a "last fixation" baseline) on lexical overlap metrics (BLEU, METEOR, ROUGE, CIDEr).
- Under the LLM-as-a-Judge paradigm (using GPT-4 to evaluate fluency, relevance, and correctness), Gazette achieved higher scores than the variant trained without the ThinkAloud task ("w/o ThinkAloud").
- The results indicate that the ThinkAloud task helps the model learn complex reasoning processes and attention allocation strategies, leading to more accurate and context-aware descriptions.
Predictive Tasks (Visual Search):
- On the COCO-Search18 dataset, Gazette achieved state-of-the-art or near state-of-the-art performance.
- For Target-Present search, Gazette achieved 77.3% accuracy, nearly matching the previous state-of-the-art (77.6% by Mondal et al.).
- For Target-Absent search, Gazette significantly outperformed prior methods (44.3% accuracy vs. 38.7% for Mondal et al.).
- The authors note that improvements in Target-Absent search were more modest, attributing this to lower inter-observer agreement in these tasks, which reduces the efficacy of the shared-pattern extraction strategy.

Significance and Claims
The paper claims that Gazette enables gaze to serve as a powerful, non-intrusive cue for inferring human goals with fine-grained semantic detail, moving beyond the limitations of fixed taxonomies. By successfully decoupling goal-specific dynamics from individual noise through the ThinkAloud strategy, the framework demonstrates generalizability across diverse top-down attentional tasks.

The authors position this work as a step toward real-world applications where understanding what a user is looking for (e.g., specific style preferences in shopping, conceptual focus in education, or navigation intent in AR) is critical. They acknowledge limitations, specifically that the method relies on inter-observer consistency (degrading in tasks with low agreement) and that the GPT-4 generated transcripts may inherit biases from the LLM's training priors rather than true participant cognition. However, they assert that the approach opens new avenues for using MLLMs to understand human attention and mental states.

Technical Summary: Gazette – Gaze-to-Text Generation

More like this