Do MLLMs Understand Pointing? Benchmarking and Enhancing Referential Reasoning in Egocentric Vision

1. Problem Statement

The paper addresses a critical gap in Egocentric AI (e.g., smart glasses): the inability of current Multimodal Large Language Models (MLLMs) to accurately interpret pointing gestures in first-person views.

• Referential Hallucination: Despite strong general visual capabilities, state-of-the-art MLLMs fail to trace the geometric projection of a pointing finger. Instead, they rely on spurious correlations, such as visual proximity (objects near the hand) or object saliency (prominent background objects), leading to incorrect target identification.
• Data Scarcity: Existing benchmarks (e.g., RefCOCO, Visual Genome) rely on third-person views or explicit bounding boxes, lacking the "wide-angle" nature of egocentric vision. Large egocentric datasets (e.g., Ego4D) focus on action recognition rather than dense "pointing-object" geometry and question-answering (QA).
• Spatial Reasoning Deficit: Current models struggle to separate hand appearance from the spatial intent of pointing, failing to resolve referential ambiguities in naturalistic interactions (e.g., "What is this?" while pointing).

2. Methodology: EgoPoint-Bench

The authors introduce EgoPoint-Bench, a comprehensive benchmark designed to evaluate and enhance pointing reasoning. It consists of a dual-source data construction strategy and a rigorous evaluation framework.

A. Data Construction

The dataset comprises 11,729 QA pairs (10,567 synthetic + 1,162 real-world).

1. Simulation Phase (Point-Sim Framework):

   • Environment: Utilizes 1,838 high-fidelity 3D scenes from AI2-THOR, HSSD, ReplicaCAD, and HM3D.
   • Physics-Based Synthesis: A custom pipeline uses ray-casting to generate noise-free pointing labels. It employs 42 diverse hand models (varying skin tones, clothing, left/right hands) to ensure robustness.
   • Geometric Precision: The system calculates the agent's position and camera orientation to align the index finger's vector precisely with the target object, ensuring the "pointing ray" hits the target without intersecting obstacles.
   • QA Generation: A hybrid "machine-generation, human-verification" pipeline creates questions. To prevent hallucinations during generation, ground-truth bounding boxes are rendered as visual prompts for the generator.

2. Real-World Phase:

   • Collection: 8 volunteers wearing MLVision smart glasses collected data in diverse environments (retail, homes, outdoors, zoos).
   • Annotation: A rigorous Human-in-the-Loop (HITL) workflow involved voice-driven keyframe localization, clarity-aware frame selection (filtering motion blur), and manual bounding box annotation to align with the user's pointing intent.

B. Evaluation Taxonomy

The benchmark evaluates models across five capability dimensions and three levels of deixis (linguistic reference):

• Five Dimensions:
  1. Basic Perception (BP): Category, color, texture.
  2. Function & State (FS): Edibility, operability, dynamic states.
  3. Spatial Context (SC): Localization, scene compatibility, reachability.
  4. OCR: Text extraction from pointed objects.
  5. Adversarial Resilience (AR): Handling counterfactuals and spatial distractors.
• Three Deixis Levels:
  • L1 (Explicit): "The object I am pointing at."
  • L2 (Visual Locative): "That thing over there."
  • L3 (Implicit): "This" / "It."

3. Key Contributions

1. EgoPoint-Bench: The first benchmark unifying true egocentric vision with natural hand pointing mechanics, covering over 11k samples across simulation and real-world domains.
2. Physics-Driven Data Pipeline: A scalable simulation framework that ensures geometric precision (ray-casting) and linguistic diversity (hierarchical deixis), addressing the scarcity of high-quality spatial data.
3. Sim-to-Real Generalization: Demonstrated that models fine-tuned only on high-fidelity synthetic data achieve significant performance gains on real-world test sets, validating synthetic data as a viable solution for data scarcity in egocentric interaction.

4. Experimental Results

The authors benchmarked proprietary models (Gemini 3, GPT-5 series) and open-source models (LLaVA, InternVL, Qwen3-VL, GLM-4) under Zero-Shot and LoRA Fine-Tuning settings.

• Baseline Failure: Off-the-shelf MLLMs struggle significantly, achieving only ~60% accuracy overall. They exhibit "Referential Hallucination," often selecting objects near the hand rather than the pointed target.
• Performance Gains: Fine-tuning on the synthetic EgoPoint-Bench data yielded massive improvements:
  • LoRA Gains: Open-source models saw accuracy improvements ranging from +15.27% to +25.41%.
  • Best Performer: LLaVA-NeXT-7B achieved a 25.41% gain, reaching ~80% accuracy on simulation and ~60% on real-world data.
• Sim-to-Real Transfer: Models trained exclusively on simulation data generalized effectively to real-world data. For example, Qwen3-VL-8B improved its real-world mean accuracy from 58.14% to 71.96%.
• Error Analysis: The primary failure modes in baselines were Proximal Distraction (picking adjacent objects) and Gesture Neglect (ignoring the hand). The fine-tuned models successfully "rescued" 57–72% of these specific error cases.
• Deixis Levels: While zero-shot models performed better on vague locatives (L2/L3) than explicit instructions (L1), fine-tuning balanced performance across all levels, proving the models learned the geometric alignment of pointing.

5. Significance

• Advancing Egocentric AI: This work highlights that general pre-training is insufficient for fine-grained spatial reasoning in first-person views. It establishes that spatially-aware supervision is critical for building precise AI assistants for smart glasses and AR.
• Synthetic Data Viability: It provides strong evidence that high-fidelity, physics-based synthetic data can bridge the domain gap, offering a scalable path to train models for complex human-AI interactions without relying solely on expensive real-world data collection.
• Benchmark Standard: EgoPoint-Bench sets a new standard for evaluating multimodal reasoning, moving beyond simple object detection to complex referential understanding in dynamic, first-person environments.

Conclusion: The paper concludes that while current MLLMs lack an intrinsic understanding of "pointing," they can rapidly acquire this capability through targeted fine-tuning on high-quality synthetic data, paving the way for robust, context-aware egocentric AI assistants.