💻 computer science

VEGAS: Human-Aligned Video Caption Evaluation via Gaze

The paper introduces VEGAS, a training-free, cross-modal metric that leverages synchronized gaze data to evaluate and select video captions that better align with individual viewers' attention, thereby improving caption quality and downstream retrieval tasks without requiring model retraining.

Original authors: Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, Ufuk Topcu

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shenghui Chen, Po-han Li, Ximeng Sun, Shijia Yang, Emad Barsoum, Zicheng Liu, Sandeep Chinchali, Ufuk Topcu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a video of someone cooking in a kitchen. A standard AI captioning bot might say, "A person is in a kitchen with a stove, a fridge, and a counter." It's technically true, but it's like reading a map of the whole city when you only care about the one street where the action is happening. The bot ignores what the person actually looked at.

Enter VEGAS (Video caption Evaluation via GAze Score). Think of VEGAS as a super-smart "attention referee" that doesn't just read the video; it watches where your eyes go while you watch it.

The Problem: The "One-Size-Fits-All" Caption

The paper points out that current AI models are trained on a mix of everyone's opinions. They try to describe everything visible in a scene. But humans are picky. If you're watching a cooking video, you might be staring intently at the red pepper being chopped, while ignoring the toaster in the background. A generic AI caption might miss the pepper entirely or bury it in a list of background noise. This makes it hard to find that specific video later if you search for "chopping red peppers."

The Solution: The "Gaze-Filter"

The authors propose a clever trick called VEGAS. Instead of retraining the AI (which is like rebuilding the whole engine of a car), VEGAS acts like a filter at the end of the process.

Here's how it works with a fun analogy:
Imagine the AI is a chef who writes down five different recipes for the same meal.

Recipe A: "I cooked a meal." (Too vague)
Recipe B: "I chopped onions, garlic, and peppers." (Specific, but maybe you didn't look at the garlic)
Recipe C: "I stirred the pot." (You were looking at the pot)

Now, imagine you are wearing special glasses that track exactly where your eyes land. VEGAS takes your eye-tracking data and asks the chef: "Hey, if I only showed you the parts of the video where the viewer was looking, would you still be able to write this recipe?"

If the recipe mentions the red pepper and your eyes were glued to the red pepper, VEGAS says, "Great match! Low score!" (In this game, a lower score is better).
If the recipe mentions the toaster but your eyes never looked at the toaster, VEGAS says, "Wait, how did you know about the toaster? That's extra info you didn't need. High score!" (Bad match).

VEGAS then picks the "recipe" (caption) with the lowest score—the one that fits your specific gaze perfectly.

What They Found (The Real Deal)

The researchers tested this on two very different types of videos:

First-person videos (like someone walking around their house doing chores).
Instructional slides (like a PowerPoint presentation).

The Big Win:
On the first-person videos, VEGAS worked like a charm. When they used the "gaze filter" to pick captions, the descriptions became 13.53% more similar to what humans actually wrote.

The Proof: They ran a statistical test (a Wilcoxon signed-rank test) and got a result of Z = 10.04 with a p-value < 0.001. This means the improvement wasn't a fluke; it was a real, measurable difference.
The Retrieval Boost: When they used these better captions to search for videos, they found the right video 1.14% more often at the very top of the list. If you looked further down the list (at the top 5 or 10 results), the improvement jumped to 4.16% and 4.10% respectively.

The "It Depends" Part:
On the instructional slides, the results were a bit different. The improvement was small (+3.88%) and the paper notes it was not statistically significant (p = 0.0952).

Why? The authors suggest that with slides, there are often many "correct" ways to summarize the text. Even if your eyes look at a specific chart, different people might interpret the meaning of that chart in different ways. Gaze tells you where they looked, but not necessarily how they interpreted the complex ideas. So, VEGAS is great for spotting concrete objects (like a red cap or a dog) but less perfect for abstract concepts.

What They Ruled Out

The paper was very careful to check if VEGAS was just cheating by picking shorter or simpler captions.

Did it just pick short captions? No. They checked the correlation between the VEGAS score and the number of words, and it was basically zero (0.001).
Did it just pick generic captions? No. The correlation with "specificity" (using nouns, verbs, etc.) was also near zero (0.026).
Did it need a new AI model? No. They used existing, powerful AI models (like Gemini and GPT) and just used VEGAS to pick the best one from a list of candidates. No retraining required.

The Bottom Line

VEGAS suggests that if you want a video caption that feels personal and helps you find the video later, you shouldn't just ask the AI "What is in this video?" You should ask, "What did this specific person look at?"

The paper shows that by using eye-tracking data as a filter, we can turn a generic AI description into a personalized one that matches human attention much better—especially when the video is about real-world actions. It's not a magic wand that solves everything (slides are still tricky), but it's a powerful new tool that works without needing to rebuild the AI from scratch.

Technical Summary: VEGAS – Human-Aligned Video Caption Evaluation via Gaze

1. Problem Statement

Vision-language models (VLMs) excel at generating semantically accurate video captions but typically rely on crowd-sourced annotations that aggregate diverse human interpretations. Consequently, these models often produce "globally correct" descriptions that fail to capture individual viewers' specific attention patterns. In applications like caption-indexed retrieval, a generic caption emphasizing unattended background content can make it difficult for a user to retrieve a video based on the specific object or action they attended to.

The core problem addressed is the misalignment between generated captions and individual visual attention. While human gaze provides a measurable proxy for visual attention, prior work has largely used gaze only as a training-time supervision signal. This paper investigates whether gaze can be leveraged at test time to select personalized captions that reflect a specific viewer's focus without retraining the underlying model.

2. Methodology

2.1 VEGAS Metric (Video caption Evaluation via GAze Score)

The authors propose VEGAS, a training-free, cross-modal, information-theoretic metric designed to quantify how well a candidate caption aligns with a viewer's gaze.

Formulation: The goal is to maximize the mutual information between a caption $t$ and a gaze signal $G$ . Since the true marginal distributions $P(t)$ and $P(G)$ are intractable, VEGAS approximates this using a frozen, pretrained VLM's conditional likelihoods.
Core Intuition: A gaze-aligned caption should be highly predictable from the attended regions ( $G$ ) alone, with non-attended regions ( $\bar{G}$ ) contributing minimal additional information.
Definition: VEGAS is defined as the pointwise conditional mutual information between caption tokens $t$ and non-attended regions $\bar{G}$ , conditioned on attended regions $G$ :
$\text{VEGAS}(c, G) \triangleq I(t; \bar{G} | G) = \log \frac{P_\theta(t | V)}{P_\theta(t | G)}$
Where $V$ is the full video and $G$ is the gaze-attended video.
Interpretation: A lower VEGAS score is better. It indicates that the caption is well-explained by the gazed regions alone. A high score implies the caption relies heavily on visual content outside the viewer's focus.
Token-Level Decomposition: The metric decomposes autoregressively over tokens, allowing the identification of specific semantic concepts supported or unsupported by the viewer's gaze.

2.2 Gaze-Conditioned Rejection Sampling

VEGAS is used to select captions from a pool of candidates generated by existing VLMs without model retraining:

Candidate Generation: Multiple VLMs generate $N$ candidate captions for a video $V$ using various decoding temperatures.
Scoring: For each candidate $c_i$ , the VEGAS score is computed using the viewer's gaze signal $G$ .
Selection: The caption with the lowest VEGAS score is selected:
$c^* = \arg \min_{c_1, \dots, c_N} \text{VEGAS}(c_i, G)$
This process effectively filters out captions describing unattended content, favoring those that align with the viewer's specific attentional focus.

3. Dataset Curation

To evaluate the framework, the authors curated a multimodal dataset pairing synchronized human gaze, visual content, and free-form human-written captions across two domains:

Aria Everyday Activities (AEA): 332 egocentric video clips derived from Project Aria smart glasses data. These clips capture dynamic, real-world tasks (e.g., cooking, computing) where gaze helps resolve concrete object ambiguities.
SlideVQA: 300 instructional slides from 30 decks. Participants viewed slides via webcam-based eye tracking and provided captions. This domain tests gaze-conditioning in dense visual media where conceptual abstraction is required.

The dataset includes 2,981 high-quality human annotations, capturing the diversity of attention-driven semantic variations.

4. Key Results

4.1 Semantic Alignment

AEA (Egocentric): VEGAS-selected captions showed a statistically significant improvement in semantic alignment with human annotations compared to naive VLM captions. The mean SBERT similarity increased by +0.0856 (+13.53%) ( $p < 0.001$ ).
SlideVQA: The improvement was smaller (+0.0256) and not statistically significant ( $p = 0.0952$ ). The authors attribute this to the nature of slides, where multiple valid conceptual summaries exist for the same visual, making gaze a weaker selector for abstract interpretations compared to concrete objects.

4.2 Caption-to-Video Retrieval

On the AEA dataset, VEGAS was used to construct an index for caption-to-video retrieval:

Performance: VEGAS-selected indices improved retrieval metrics over random VLM captions.
- Precision/Recall/mAP@1: +1.14% improvement.
- Recall@5: +4.16% improvement.
- mAP@10: +2.46% improvement.
Gap Closure: VEGAS recovered approximately 15% of the gap between random machine selection and human-human retrieval at Rank 1, but recovered over 40% of the gap at Recall@5 and Recall@10. This suggests VEGAS effectively moves the correct video into the top few candidates, even if it does not always reach the top rank.

4.3 Ablation and Diagnostics

Gaze Corruption: Replacing true gaze with random gaze from other clips significantly increased the VEGAS score (worsening alignment), confirming the metric's sensitivity to correct gaze signals.
Length Bias: Correlation analysis showed VEGAS is not driven by caption length or genericness; improvements are not explained by selecting shorter captions.
Token-Level Analysis: Qualitative examples demonstrated that VEGAS correctly penalizes tokens describing unattended regions (e.g., "garden" when the user looked at a "washing machine") and favors tokens aligned with fixation points.

5. Significance and Claims

The paper claims that VEGAS demonstrates the practical utility of incorporating viewer attention during inference. Its primary contributions are:

A Training-Free Metric: It provides a method to evaluate and select personalized captions without fine-tuning or retraining large vision-language models.
Test-Time Conditioning: It shifts the paradigm of using gaze from a training-time supervision signal to a test-time conditioning signal for personalization.
Domain Specificity: The results highlight that gaze-conditioned captioning is most effective when visual attention helps disambiguate concrete referents (e.g., egocentric activities) but less diagnostic when the stimulus supports many valid conceptual summaries (e.g., instructional slides).
Retrieval Improvement: By aligning captions with individual attention, VEGAS improves the semantic indexing of videos, making them more retrievable based on what a specific user actually looked at.

The authors remain modest, noting that VEGAS inherits limitations from the underlying VLMs (e.g., hallucinations) and that gaze indicates where a viewer looks but not necessarily how they interpret complex visual relationships. Future work is suggested to explore implicit attention proxies (like saliency models) and distilling VEGAS into training objectives.