💻 computer science

Do You See What I Am Pointing At? Gesture-Based Egocentric Video Question Answering

This paper introduces EgoPointVQA, a new dataset and benchmark for gesture-grounded egocentric question answering, along with the Hand Intent Tokens (HINT) method that leverages 3D hand keypoints to significantly improve multimodal large language models' ability to infer pointing intent from egocentric videos.

Original authors: Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou

Published 2026-03-17

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yura Choi, Roy Miles, Rolandos Alexandros Potamias, Ismail Elezi, Jiankang Deng, Stefanos Zafeiriou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are wearing a pair of smart glasses that can see exactly what you see. You want to ask your AI assistant a question about the world around you.

The Problem: The "Pointing" Gap
Right now, if you point at a specific pot on your stove and ask, "Should I use this one?", most AI assistants get confused. They might look at the whole kitchen, see two pots, and guess randomly. They don't understand that your finger is the "cursor" telling them exactly which pot you mean.

It's like trying to play a game of "I Spy" with someone who is blindfolded; you say, "I spy something red," but they can't see your finger pointing at the specific red apple on the table, so they guess the red car in the driveway instead.

The Solution: EGOPOINTVQA
The researchers behind this paper built a new training ground called EGOPOINTVQA. Think of this as a massive "driving school" for AI, but instead of learning to drive cars, the AI is learning to understand pointing.

They created thousands of videos (some made by computers, some filmed by real people wearing smart glasses) where someone points at objects and asks questions like:

"What is this?"
"Which of these is closer?"
"How many of these are there?"

The goal was to teach the AI that when a human points, the word "this" or "that" isn't just a random word; it's a command that says, "Look right here, at the thing my finger is touching."

The Secret Sauce: HINT (Hand Intent Tokens)
Even with all these videos, standard AI models still struggled. They could see the video and read the question, but they couldn't "feel" the gesture.

To fix this, the researchers invented a clever trick called HINT (Hand Intent Tokens).

Imagine you are trying to explain a complex dance move to a friend over the phone. You could just describe the steps (the text), or you could send them a video of the dance (the visual). But what if you sent them a special map that highlights exactly where your hands are moving at every second?

That's what HINT does.

The Map: The system uses a special tool to track the 3D position of the user's hand and fingers in every single frame of the video.
The Translation: It turns those hand positions into a secret code (tokens) that the AI can read.
The Mix: It mixes this "hand code" right into the AI's brain alongside the video and the question.

Now, when the AI reads the question "Is this pot black?", it doesn't just guess. It looks at the "hand map," sees the finger pointing at the black pot, and confidently answers, "Yes."

Why This Matters
This research is a big step toward making AI assistants that feel truly natural. In the future, you won't have to say, "Hey AI, look at the black pot on the left." You can just point at it and ask, "Is this one ready?" and the AI will know exactly what you mean.

In a Nutshell:

The Issue: AI is bad at understanding when humans point at things.
The Dataset: They built a giant library of pointing videos to train AI.
The Fix: They gave the AI a "superpower" (HINT) that translates hand movements into a language the AI understands, allowing it to finally "see" what you are pointing at.

1. Problem Definition

The paper addresses a critical gap in Egocentric Video Question Answering (VQA): the inability of current Multimodal Large Language Models (MLLMs) to interpret deictic expressions (e.g., "this," "that," "these") grounded in pointing gestures.

The Challenge: In natural human-AI interaction (e.g., via smart glasses), users often ask questions like "Should I use this one?" while pointing at an object. Current MLLMs struggle because:
1. Data Scarcity: Existing datasets lack natural, gesture-rich egocentric videos where pointing is central to the query.
2. Architectural Limitations: Standard MLLMs process visual and textual inputs globally but lack mechanisms to explicitly encode hand pose, trajectory, or pointing direction. They fail to link the linguistic deictic pronoun to the specific spatial-temporal location of the user's finger.
Consequence: Even state-of-the-art models (e.g., GPT-4o, Qwen3-VL) fail to resolve these references, often guessing based on object saliency or center-bias rather than the actual gesture.

2. Methodology

The authors propose a two-pronged solution: a new dataset (EGOPOINTVQA) and a novel architectural module (HINT).

A. EGOPOINTVQA Dataset

The first dataset specifically designed for gesture-grounded egocentric VQA, comprising 4,400 videos (4,000 synthetic, 400 real-world) and 18,745 question-answer pairs.

Task Taxonomy: Questions are categorized into six reasoning types:
1. Reference: Identifying the pointed object.
2. Counting: Counting instances of the pointed type.
3. Spatial: Determining relative location.
4. Temporal: Resolving order in sequential pointing (e.g., "the second object").
5. Attribute: Identifying properties (color, shape).
6. Feedback: Assessing object function/utility.
Data Generation:
- Synthetic: Generated using AI2-THOR simulator with AI2-THOR and MIXAMO animations. Inverse Kinematics (IK) ensures the index finger aligns precisely with target objects.
- Real-world: Collected using Meta Ray-Ban smart glasses from 20 participants in diverse indoor/outdoor settings.
QA Pipeline: An automated pipeline extracts dense scene metadata, generates structured MCQs, and rephrases them into natural deictic questions (replacing object names with pronouns like "this").

B. Hand Intent Tokens (HINT)

To enable models to understand gestures, the authors propose HINT, a lightweight module that encodes 3D hand keypoints into tokens interleaved with visual inputs.

3D Hand Reconstruction: Uses WiLoR to extract 21 3D hand keypoints ( $K_t$ ) for each video frame.
Keypoint Adapter: A lightweight MLP projects the flattened 3D coordinates into a single Hand Intent Token ( $H_t$ ) per frame. If hand detection confidence is below a threshold ( $\tau=0.5$ ), no token is generated.
Interleaving Strategy: The model input sequence is constructed by interleaving visual tokens ( $V_t$ $V_{t}$ ) with hand intent tokens ( $H_t$ $H_{t}$ ).
- Sequence: [Question] [Vis_1] [Key_1] [Vis_2] [Key_2] ...
Training: The model is fine-tuned using LoRA on the vision encoder and LLM backbone, while the Keypoint Adapter is trained from scratch. This allows the LLM to explicitly attend to the spatial and temporal context of the pointing gesture.

3. Key Contributions

EGOPOINTVQA Dataset: The first benchmark dedicated to deictic reasoning in egocentric videos, bridging the gap between gesture and language.
HINT Architecture: A novel method to inject 3D geometric hand information directly into the LLM's token stream, avoiding the need for artificial visual overlays (like bounding boxes) or complex attention mechanisms.
State-of-the-Art Performance: Demonstrated that explicit gesture encoding significantly outperforms standard fine-tuning and existing MLLMs.

4. Experimental Results

Performance Metrics

Evaluated on the EGOPOINTVQA test set (300 real videos, 672 QA pairs) across six tasks.

Baseline Struggles: Top proprietary models (GPT-4o) achieved only 46.8% average accuracy. Open-source models (InternVL3-78B) reached 66.6%.
HINT Success: The proposed HINT-14B (based on InternVL3-14B) achieved 68.1% average accuracy, surpassing the previous SOTA (InternVL3-14B without HINT) by 6.6%.
- Reference Task: HINT-14B achieved 73.8% (vs. 63.1% baseline), a +10.7% gain.
- Temporal Task: Significant improvements in tracking sequential gestures.

Ablation Studies

Component Necessity: Simply fine-tuning on the dataset (SFT) without HINT yielded marginal gains. Combining SFT with HINT provided the largest boost, proving that explicit gesture encoding is essential.
Data Composition: Mixing synthetic and real data yielded better results than real data alone, highlighting the value of synthetic data for scale.
Alternative Encodings: HINT (learned 3D tokens) outperformed naive visual prompting (drawing arrows on frames) and textual coordinate inputs, suggesting that learning to process geometric hand data is superior to manual prompting.
Robustness: Removing hand visibility from videos caused a massive performance drop (e.g., Reference task dropped from 75.0% to 41.7%), confirming the task relies on gestures, not just scene context.

5. Significance and Impact

Advancing Embodied AI: This work is crucial for the development of next-generation wearable AI assistants (e.g., AR/VR glasses) that must understand user intent through natural, non-verbal cues.
Bridging Vision and Language: It demonstrates that MLLMs can be effectively conditioned on fine-grained 3D geometric signals (hand poses) to resolve linguistic ambiguities, moving beyond simple image-text alignment.
Open Research: The authors release the code, model weights, and dataset, establishing a new standard for evaluating gesture-grounded reasoning in computer vision.

In summary, the paper establishes that pointing gestures are a distinct and necessary modality for egocentric VQA. By introducing a dataset rich in deictic references and a method (HINT) to explicitly encode 3D hand intent, the authors enable MLLMs to accurately interpret "what" a user is pointing at, a capability previously missing from state-of-the-art models.