Technical Summary: PointVG-R

Problem Definition

Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between a visual scene and a pointing gesture. While traditional visual grounding research focuses on semantic descriptions (e.g., "the red cup"), gestural pointing provides a more intuitive reference in shared physical environments. However, existing Multimodal Large Language Models (MLLMs) typically encode images into holistic feature representations and reason primarily through linguistic associations. This approach often overlooks the rich perceptual cues and explicit spatial geometry inherent in images. Consequently, standard MLLMs exhibit "cognitive fragility," failing to explicitly account for the spatial-geometric constraints of pointing behaviors, which leads to hallucinated localization biases, saliency bias, and semantic misinterpretation in complex interactive scenarios.

Methodology

The paper proposes PointVG-R, a reasoning-guided MLLM designed to internalize geometric reasoning into the visual grounding process. The core innovation is the "Thinking with Images" paradigm, which bridges the gap between gesture perception and object localization via a Visual Chain-of-Thought (V-CoT).

1. Visual Chain-of-Thought (V-CoT) Pipeline

Unlike one-step direct predictions, PointVG-R executes an iterative, human-like cognitive process decomposed into four sequential stages:

1. Hand Detection: Localizing the interaction source by predicting the hand bounding box.
2. Keypoint Extraction: Predicting finger root and tip keypoints to parameterize the pointing direction.
3. Pointing Geometry: Constructing a geometric ray from the fingertips to guide spatial sampling. This involves invoking a tool (draw_ray) to explicitly visualize the trajectory and re-encoding the ray-aligned region to fuse with global context.
4. Target Alignment: Using the enriched geometric features to predict the final target bounding box.

2. Two-Stage Training Framework

To ensure the model internalizes this structured reasoning, PointVG-R employs a two-stage training protocol:

• Stage 1: Cold-Start Knowledge Injection (SFT): The model undergoes Supervised Fine-Tuning (SFT) on the EgoPoint-CoT dataset. This dataset features expert-level reasoning annotations (hand boxes, keypoints, rays, and target boxes) to inject foundational geometric priors and establish a stable initial policy.
• Stage 2: Adaptive Reinforcement Learning (RL): The model is further optimized using Group Relative Policy Optimization (GRPO). To address the heterogeneity in learning signal quality during RL sampling (where noisy samples can destabilize coordinate regression), the authors introduce an Adaptive Importance Weighting strategy. This strategy dynamically adjusts reward signals based on Group Variance, up-weighting samples from groups with high variance (indicating strong relative preference signals) and down-weighting low-variance or noisy groups to ensure training stability.

3. Reward Design

The reward function is structured to evaluate the entire V-CoT trajectory, not just the final output. It includes:

• Accuracy Rewards: IoU for the final target box, hand box, and angular consistency for the drawn ray.
• Format Rewards: Ensuring valid output structures and tool calls.
• Penalties: For redundant reasoning steps or hallucinated traces in negative samples (images without pointing hands).

Key Contributions

1. PointVG-R Model: A reasoning-guided MLLM that internalizes explicit spatio-geometric reasoning, moving beyond simple feature matching to structured spatial parsing.
2. EgoPoint-CoT Dataset: A high-quality dataset of ~15K images featuring detailed visual reasoning trajectories (hand, direction, ray, target) to guide the model's iterative cognitive process.
3. Algorithmic Optimization:
   • A multi-dimensional geometric reward function tailored for gesture-based grounding.
   • A Group-Variance-based Adaptive Importance Weighting strategy that resolves signal quality variance issues in RL-based coordinate regression, significantly improving training stability and precision.
4. Paradigm Shift: The introduction of the "Thinking with Images" paradigm, demonstrating that explicit geometric reasoning (via V-CoT) is necessary for robust gesture-based grounding.

Experimental Results

Experiments were conducted on the EgoPoint-CoT benchmark against state-of-the-art MLLMs (including Qwen2.5-VL, LLaVA-OneVision, DeepSeek-VL2, and InternVL).

• Performance: PointVG-R achieves a new State-of-the-Art (SOTA), outperforming the strongest baseline by 15.86 points in mean Intersection over Union (mIoU). Specifically, on the Qwen2.5 (7B) backbone, it reaches an mIoU of 0.7570, a significant improvement over the cold-start baseline (0.6688).
• Ablation Studies:
  • Intermediate Supervision: Removing intermediate V-CoT supervision (hand, ray, keypoint) leads to substantial performance drops, confirming the necessity of decomposing the task into geometric sub-problems.
  • Group Variance: The proposed Group Variance weighting strategy outperforms vanilla GRPO and other statistics (e.g., reward entropy, top-bottom gap), validating its role in stabilizing policy updates for complex spatial reasoning.
  • Scaling Function: Square-root scaling for importance weights was found to be the most effective mapping function, balancing sensitivity to informative groups with training stability.

Significance and Claims

The paper claims that PointVG-R successfully addresses the "cognitive vulnerability" of traditional MLLMs in interpreting gestural spatial relations. By internalizing geometric reasoning through the Visual Chain-of-Thought and optimizing via adaptive reinforcement learning, the model achieves a level of spatial understanding previously unattainable by direct regression or semantic-only approaches. The authors emphasize that their work demonstrates the critical necessity of explicit geometric reasoning for robust gesture-based grounding.

The paper acknowledges a limitation: the iterative V-CoT paradigm introduces non-negligible inference latency due to sequential token generation, which may pose challenges for real-time applications on wearable devices like AR glasses. Future work is suggested to focus on reducing token overhead or accelerating inference while maintaining logical reasoning capabilities.