💻 computer science

Leveraging Gaze and Set-of-Mark in VLLMs for Human-Object Interaction Anticipation from Egocentric Videos

This paper proposes a novel approach for anticipating human-object interactions in egocentric videos using Vision Large Language Models, which integrates Set-of-Mark prompting for visual grounding, gaze trajectory analysis for intent understanding, and an inverse exponential sampling strategy for temporal dynamics, achieving state-of-the-art performance on the HD-EPIC dataset.

Original authors: Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniele Materia, Francesco Ragusa, Giovanni Maria Farinella

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a pair of smart glasses that can see exactly what you see. Your goal is to build an AI assistant that doesn't just watch you cook or work, but predicts what you are about to do next before you even do it.

This paper is about teaching that AI assistant to be a "mind reader" for your daily life, specifically in tricky situations like a busy kitchen or a factory floor. Here is how they did it, explained simply.

The Big Problem: The AI is a bit "Clueless"

Imagine you are holding a knife and looking at a cutting board, but the AI only sees a blurry mess of moving pictures. It doesn't know if you are about to chop an onion, slice a tomato, or just wave the knife around.

Current AI models (called Vision Large Language Models or VLLMs) are like very smart students who have read a million books but have never actually seen a kitchen. They can talk about cooking, but when you show them a video, they often guess wrong because they miss the small details of where you are looking and what you are focusing on.

The Solution: Three Superpowers

The authors gave their AI three "superpowers" to fix this. Think of it like upgrading a detective's toolkit.

1. The "Set-of-Mark" Glasses (Highlighting the Scene)

The Metaphor: Imagine you are looking at a crowded room. It's hard to pick out one specific person. Now, imagine someone puts a glowing, colored sticker on everyone's head. Suddenly, it's easy to see who is who.

How it works: The AI takes the video frame and automatically puts "sticky notes" (masks) around every object in the scene (the pot, the egg, the spoon). This forces the AI to stop seeing a blurry blob and start seeing distinct, labeled objects. It's like turning a chaotic sketch into a clear, labeled diagram.

2. The "Gaze Trail" (Following the Eyes)

The Metaphor: Have you ever watched a magician? You know they are about to pull a rabbit out of a hat because your eyes were already staring at the hat. Humans look at what they are about to touch.

How it works: The AI doesn't just look at the current picture; it looks at a trail of breadcrumbs left by your eyes.

If you looked at the egg 2 seconds ago, then the pan 1 second ago, and now you are staring at the stove, the AI sees this path.
They draw this path on the video as a line of dots, fading from blue (old) to red (new).
This tells the AI: "Hey, the user is focusing on this path. They are likely about to interact with the object at the end of the line."

3. The "Time-Travel" Filter (Picking the Right Moments)

The Metaphor: Imagine you are trying to guess the ending of a movie, but you are forced to watch the whole 2-hour film at normal speed. It's boring and you might miss the climax. Instead, imagine a smart editor who skips the slow parts and only shows you the 10 seconds right before the explosion.

How it works: AI computers get overwhelmed if you show them too many video frames. The authors created a special "sampling" rule. Instead of picking frames evenly (like every 1 second), they pick frames heavily weighted toward the very end of the clip.

They use a math trick called "inverse exponential sampling."
Basically, they say: "The last few seconds are the most important. Let's look at those super closely, and ignore the boring stuff that happened 10 seconds ago."

The Result: A Smarter Assistant

The researchers tested this on a dataset called HD-EPIC, which is basically a massive library of videos of people cooking in kitchens.

Before: The best AI could guess the next object correctly about 21% of the time.
After: With these three superpowers (Sticky Notes + Eye Trail + Smart Time Filter), the AI jumped to 27.5% accuracy.

While 27% might not sound like 100%, in the world of AI prediction, jumping 6-7 percentage points is a massive leap. It's the difference between a student who barely passes and one who gets an A.

Why This Matters

This isn't just about guessing what you're cooking. This technology could:

Save lives in factories: Alert a worker before they touch a hot machine.
Help the elderly: Remind someone to put the lid on a pot before it boils over.
Assist the visually impaired: Tell a blind person, "You are reaching for the coffee mug, not the hot pan."

The Bottom Line

The paper shows that to make AI truly understand human behavior, we can't just feed it raw video. We have to teach it to highlight important objects, follow human eyes, and focus on the critical moments. By combining these tricks, the AI becomes a much better "assistant" for our daily lives.

1. Problem Definition

The paper addresses the challenge of Human-Object Interaction (HOI) Anticipation in egocentric (first-person) vision. The goal is to predict which object a user will interact with next and how, based on a short video clip preceding the interaction.

Context: This capability is crucial for intelligent assistive systems (e.g., in kitchens or industrial settings) to guide users, prevent accidents, and understand short/long-term goals.
Specific Task: The authors focus on the Gaze Interaction Anticipation benchmark from the HD-EPIC dataset. The task is formulated as a Visual Question Answering (VQA) problem: given a video clip and a set of candidate objects, the model must select the correct target object the user is about to interact with, distinguishing it from semantic distractors.
Limitations of Existing Approaches: Previous methods often struggle with:
- Visual Grounding: Difficulty in precisely locating and understanding specific objects within a complex scene.
- Intent Modeling: Failure to effectively utilize the user's gaze trajectory as a strong prior for intent.
- Temporal Efficiency: Inefficient processing of video frames, either missing critical temporal dynamics or including too much redundant data.

2. Methodology

The proposed approach is model-agnostic, designed to enhance any Vision Large Language Model (VLLM). It consists of four integrated modules (see Figure 2 in the paper):

A. Set-of-Mark (SoM) Module

Function: Enhances visual grounding and scene understanding.
Mechanism: Applied to the final frame of the input video. It uses Semantic-SAM to generate semantic segmentation masks for objects in the scene.
Implementation: These masks are overlaid on the image (with transparency) to create a "Set-of-Mark" prompt. Unlike standard SoM which uses alphanumeric tags, this method relies solely on masks because the VLLM selects the object name from a provided candidate list in the text prompt.
Goal: To help the VLLM spatially locate and distinguish candidate objects more effectively.

B. Gaze Module

Function: Models user intent and attention.
Mechanism: Projects the sequence of the most recent 15 ground-truth 2D gaze fixations onto the video frames using a sliding window.
Visualization: The trajectory is rendered as connected circles that transition from red (most recent fixation) to blue (least recent).
Goal: To provide a visual cue that explicitly guides the VLLM's attention toward the objects the user has been looking at, serving as a strong prior for the next interaction.

C. Sampling Module (Inverse Exponential Sampling)

Function: Optimizes the temporal input to capture relevant dynamics while reducing computational load.
Challenge: VLLMs process video as long token sequences ( $O(N^2)$ complexity), making dense frame sampling prohibitive.
Strategy: Instead of uniform sampling, the authors propose Inverse Exponential Sampling.
- The final frame is always included.
- The remaining $n-1$ frames are sampled probabilistically with a bias toward the end of the video clip (immediately preceding the interaction).
- Formula: The probability $p_i$ of selecting frame $i$ is defined by an exponential decay function controlled by a hyperparameter $\lambda$ . Higher $\lambda$ values concentrate samples closer to the interaction time.
Goal: To prioritize the temporal window where the user's intent is most clearly forming.

D. VLLM Module

Function: The core predictor.
Input: Receives the $n$ sampled frames (enhanced with SoM masks and Gaze trajectories) and a textual prompt containing the question, candidate answers, and specific instructions to analyze the visual cues.
Output: Predicts the target object noun.

3. Key Contributions

Novel Integration of Gaze and SoM: The paper proposes a unified framework that combines spatial scene information (via Set-of-Mark prompting) with user intent modeling (via gaze trajectory visualization) to significantly boost VLLM performance in HOI anticipation.
Inverse Exponential Sampling Strategy: A new video sampling technique that biases frame selection toward the moments immediately preceding an interaction, optimizing the trade-off between temporal context and computational efficiency.
Model-Agnostic Architecture: The method is designed to work with any state-of-the-art VLLM, demonstrated by experiments on both LLaVA-OneVision and Gemini 2.0 Flash.
State-of-the-Art Results: The approach achieves new benchmarks on the HD-EPIC Gaze Interaction Anticipation task.

4. Experimental Results

The experiments were conducted on the HD-EPIC dataset (specifically the Gaze Interaction Anticipation subset).

Performance Comparison:
- The proposed method achieved 27.2% accuracy with LLaVA-OneVision 7B and 27.5% with Gemini 2.0 Flash.
- This surpasses the previous state-of-the-art (Qwen2.5VL-Ricoh at 22.0%) by a significant margin (>5%).
- It also outperforms the HD-EPIC baseline (Gemini 1.5 Pro at 21.0%) and other top challenge entries (K-Net, T-CoT).
Ablation Studies:
- Module Impact: Combining SoM and Gaze modules yielded the best results (27.2%). Using either alone provided improvements over the baseline but was insufficient for optimal performance.
- Sampling Strategy: The inverse exponential sampling with $\lambda = 1/10$ (biasing toward the end) was optimal. Uniform sampling ( $\lambda=0$ ) performed significantly worse.
- Sample Size: An optimal sample size of 15 frames was identified. Increasing the sample size beyond this point (oversampling) led to performance degradation, likely due to noise or token overload.
- Frame Rate (Gemini): For Gemini 2.0 Flash (which does not support custom frame selection), the best performance was achieved at 2 fps using both SoM and Gaze modules.

5. Significance and Conclusion

Advancement in Egocentric Vision: The work demonstrates that leveraging multimodal cues (specifically gaze) and advanced prompting techniques (SoM) can drastically improve the reasoning capabilities of VLLMs in complex temporal tasks.
Practical Application: The method is highly relevant for real-world assistive technologies where predicting user intent (e.g., reaching for a hot pot) is critical for safety and efficiency.
Efficiency: The inverse exponential sampling strategy offers a computationally efficient way to handle video inputs for large models without sacrificing temporal relevance.
Open Source: The authors have released their code to encourage further research in this domain.

In summary, this paper successfully bridges the gap between raw visual data and high-level intent prediction by enhancing VLLMs with explicit spatial grounding and gaze-based attention mechanisms, setting a new standard for human-object interaction anticipation.