MolmoPoint: Better Pointing for VLMs with Grounding Tokens

1. Problem Statement

Vision-Language Models (VLMs) require the ability to "ground" language to visual elements, specifically through pointing. Current state-of-the-art VLMs typically perform pointing by generating text-based coordinates (e.g., "x=120, y=340") or discrete coordinate bins as part of their text output. This approach suffers from three main limitations:

• Complexity: The model must learn a complex coordinate system and map it to visual features.
• Inefficiency: Generating coordinates requires a high token count (often 8+ tokens per point), increasing decoding costs and inference latency.
• Decoupling: There is a loose coupling between visual recognition and pointing; the model must recognize an object and then separately calculate its location, rather than directly selecting the visual representation of that object.

2. Methodology: Grounding Tokens

The authors propose MolmoPoint, a novel pointing mechanism that replaces text coordinates with grounding tokens. Instead of generating numbers, the model directly selects the specific visual tokens from the input image or video that correspond to the target concept.

The pointing process is hierarchical and coarse-to-fine, utilizing three special tokens:

1. <PATCH> Token:

   • Function: Selects a coarse-grained patch from the input image (or video frame).
   • Mechanism: The model generates a query vector from the <PATCH> token's hidden state and cross-attends to the hidden states of the LLM's visual tokens (image tokens).
   • Refinement: It scores all image tokens and selects the highest-scoring one.
   • Ordering: To prevent double-pointing and ensure consistency, the model uses Rotary Embeddings (RoPE) to encode the relative position of the current candidate patch against the previously selected patch. It also enforces a sorted order during training/inference.

2. <SUBPATCH> Token:

   • Function: Refines the location within the selected coarse patch.
   • Mechanism: Since standard image tokens are often pooled from multiple ViT (Vision Transformer) patches (e.g., a 28x28 pixel area), this token attends to the finer-grained ViT features within the selected patch to pick a specific subpatch (e.g., 14x14 pixels).
   • Embedding: The selected subpatch's features are added to the token's embedding to carry forward location information.

3. <LOCATION> Token:

   • Function: Specifies the precise point within the selected subpatch.
   • Mechanism: Predicts one of 9 locations arranged in a 3x3 grid within the 14x14 subpatch.
   • Precision: This results in a resolution of approximately 4.7 pixels, which remains constant regardless of the input image resolution (unlike text coordinates which scale with resolution).

4. No-More-Points Class:

   • To prevent degenerate outputs where the model generates excessive points, a special "no-more-points" class is added. The <PATCH> token can attend to this class to signal the end of the pointing sequence.

3. Key Contributions

A. Model Architecture

The authors released three specialized models based on the Molmo2 pipeline:

• MolmoPoint-8B: A general-purpose VLM for image and video pointing.
• MolmoPoint-GUI-8B: Specialized for Graphical User Interface (GUI) pointing.
• MolmoPoint-Vid-8B: A lighter-weight model optimized for video pointing and tracking.

B. New Datasets

To train these models, the authors constructed two major datasets:

1. MolmoPoint-GUISyn: A synthetic dataset of high-resolution GUI screenshots (36k images, 2M points, 10M instructions). It uses code-guided generation (extending CoSyn) to create diverse web, desktop, and mobile interfaces with dense annotations (avg. 54 points/image).
2. MolmoPoint-Track: A tracking dataset comprising:
   • MolmoPoint-TrackAny: Human-annotated point tracks on diverse videos.
   • MolmoPoint-TrackSyn: Synthetic multi-object tracking videos with complex occlusion and motion dynamics generated in Blender.

C. Training Efficiency

The grounding token approach significantly improves sample efficiency. The models learn pointing capabilities faster and with fewer examples compared to text-coordinate baselines. It also reduces the token count per point from ~8 tokens to just 3 (<PATCH>, <SUBPATCH>, <LOCATION>).

4. Results and Performance

The models achieved State-of-the-Art (SoTA) performance across multiple benchmarks, often surpassing much larger proprietary models:

• Image Pointing:

  • PointBench: Achieved 70.7% accuracy (SoTA), beating the previous Molmo2-8B by ~2 points and other open models by significant margins.
  • PixMo-Points: Surpassed proprietary models (GPT-5, Gemini) and previous open models with an F1 score of 89.2%.

• GUI Pointing:

  • ScreenSpot-Pro: Achieved 61.1% (SoTA among fully open models), outperforming text-coordinate baselines by ~9 points.
  • OSWorldG: Achieved 70.0%, demonstrating superior capability in interacting with complex interfaces.

• Video Pointing & Tracking:

  • Human Preference: In a human evaluation on video queries, MolmoPoint-8B won 59.1% of the time against a text-coordinate baseline.
  • Tracking: On the Molmo2Track benchmark, MolmoPoint-8B achieved 62.5 J&F (Jaccard & F-measure), a significant gain over Molmo2-8B (56.7).
  • Counting: Improved accuracy on video counting tasks (BURST-VideoCount) compared to the baseline.

5. Significance and Impact

• Paradigm Shift: The paper demonstrates that token referencing (selecting visual tokens directly) is a more robust and efficient method for grounding than generating numerical coordinates. It tightly couples visual recognition with spatial reasoning.
• Resolution Independence: The method maintains high precision even on ultra-HD images because the final <LOCATION> token operates on a fixed grid relative to the subpatch, avoiding the scaling issues of text coordinates.
• Resource Efficiency: The approach reduces inference latency (fewer tokens) and training data requirements (faster convergence), making high-performance grounding accessible for smaller, open-weight models.
• Generalization: The method shows strong transferability, improving performance not just on pointing tasks but also on high-resolution OCR and GUI interaction tasks, suggesting that grounding tokens enhance the model's overall visual understanding.

In conclusion, MolmoPoint establishes a new standard for VLM pointing by moving away from coordinate regression toward direct visual token selection, resulting in faster, more accurate, and more efficient models for robotics, GUI agents, and video understanding.