💻 computer science

Getting to the Point: Why Pointing Improves LVLMs

This paper demonstrates that incorporating an explicit "point-then-count" mechanism significantly improves Large Vision-Language Models' zero-shot counting accuracy and out-of-distribution generalization by leveraging spatial information to learn robust skills rather than overfitting, despite revealing some spatial biases in the reliability of the generated pointing coordinates.

Original authors: Simone Alghisi, Massimo Rizzoli, Seyed Mahed Mousavi, Giuseppe Riccardi

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simone Alghisi, Massimo Rizzoli, Seyed Mahed Mousavi, Giuseppe Riccardi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly scatterbrained, robot assistant named VisionBot. VisionBot is great at describing pictures ("That's a cat!") and answering simple questions ("Is the cat sleeping?"). But when you ask it a tricky math question like, "How many red stars are in this picture?", VisionBot often gets the answer wrong. It might guess "5" when there are actually "7," or it might just hallucinate a number without really looking.

This paper is about teaching VisionBot a new trick: Pointing before answering.

Here is the simple breakdown of what the researchers discovered, using some everyday analogies.

The Problem: The "Guessing Game"

Usually, when you ask VisionBot to count things, it tries to jump straight to the answer. It's like asking a student to solve a math problem without showing their work. They might get lucky, but if the numbers get bigger or the picture gets messy, they start guessing.

The researchers tested two ways to teach VisionBot:

Direct Counting (The "Gut Feeling" Approach): You ask, "How many stars?" and the bot just says, "Seven."
Point-then-Count (The "Show Your Work" Approach): You ask, "How many stars?" and the bot first has to point to every single star on the screen (like drawing a dot on each one), and then count the dots to give the final answer.

The Big Discovery: Pointing is a Superpower

The researchers found that the "Point-then-Count" method was a game-changer. Here's why, using three simple metaphors:

1. The "Finger-Counting" Analogy (Learning a Skill vs. Memorizing)

Imagine teaching a child to count.

Direct Counting is like teaching them to memorize that "a picture with 3 apples equals the word 'three'." If you show them a picture with 10 apples, they panic because they've never seen that before. They are just memorizing patterns.
Point-then-Count is like teaching them to use their fingers. They point to one apple, say "one," point to the next, say "two." Even if they see 100 apples, they know the process of pointing and counting.

The paper found that when the robots were taught to point first, they actually learned the skill of counting. When the researchers showed them pictures with way more objects than they had ever seen before (like 15 or 18 items), the "Point-then-Count" robots still got it right, while the "Direct Counting" robots failed miserably. Pointing forced them to learn a general rule rather than just memorizing specific answers.

2. The "Map vs. Destination" Analogy (Why it works)

Why does pointing help? The researchers did a cool experiment where they told the robot to point, but instead of giving real coordinates, they just made it say "X, X, X."

Result: The robot got terrible at counting.
Conclusion: The magic wasn't the act of pointing; it was the spatial information (the map).

Think of it like giving someone directions.

Direct Counting: "Go to the store." (The robot has to guess the path).
Point-then-Count: "Walk to the red house, then the blue house, then the green house." (The robot follows a clear map).

By forcing the robot to generate a list of coordinates (the map), it creates a structured path to the answer. It turns a messy visual puzzle into a neat list of text it can easily count.

3. The "Spot the Imposter" Analogy (Handling Distractions)

Real life is messy. Imagine a picture with 5 red stars and 50 blue circles. A "Direct Counting" robot might get confused and count the blue circles too.

The "Point-then-Count" robots were much better at ignoring the noise. Because they had to physically "point" to the specific red stars first, they were less likely to get tricked by the blue circles. It's like a security guard checking IDs one by one; they don't get distracted by the crowd.

The Catch: It's Not Perfect

The researchers also found a few flaws, like a mechanic finding a few loose screws in a great engine:

The "Lazy" Robot: Sometimes, the robot would point to the right stars, but then forget to count them correctly. It was like a student who drew all the right steps on their homework but wrote the wrong final answer.
The "Corner" Bias: Some robots were great at pointing to things in the middle of the picture but got sloppy when objects were in the corners. It's like a camera that has a great focus in the center but gets blurry at the edges.
The "Blind" Robot: In a surprising twist, the researchers found that once the robot generated the list of coordinates, it almost stopped looking at the picture entirely. It just counted the list of text coordinates. It was like a chef who tastes the ingredients, writes them on a list, and then counts the list without ever looking at the pot again.

The Bottom Line

This paper proves that if you want an AI to be smart and reliable, don't just ask for the answer. Ask it to show its work first.

By forcing Large Vision-Language Models to "point" to objects before counting them, we aren't just getting a better answer; we are teaching the AI a fundamental skill that helps it handle new, difficult, and messy situations it has never seen before. It's the difference between a robot that memorizes a script and a robot that actually understands how to solve a problem.

1. Problem Statement

Large Vision-Language Models (LVLMs) have achieved significant success in tasks like image captioning and visual question answering. However, they struggle with complex cognitive tasks requiring spatial reasoning, temporal understanding, and counting. Specifically, LVLMs often fail to generalize to object counts outside their training distribution (Out-of-Distribution or OOD) and lack explainability, as they frequently generate answers without grounding them in specific visual evidence.

While "pointing" (predicting object coordinates before answering) has been shown to improve accuracy, three critical questions remain unresolved:

Skill vs. Task Learning: Does pointing encourage the learning of general counting skills, or does it merely help the model solve narrow, fixed-range tasks?
Explainability: Are the predicted coordinates actually grounded in the image, or are they hallucinated?
Mechanism: What is the underlying mechanism by which pointing improves performance? Is it the spatial information itself, or the decomposition of the task?

2. Methodology

Experimental Setup

The authors use zero-shot counting as a controlled benchmark. The task involves determining the number of specific objects in an image based on a natural language query (e.g., "How many cyan stars are there?").

Models: Four state-of-the-art LVLMs were evaluated:
- Qwen2.5-VL (3B and 7B parameters)
- LLaVA-OneVision-1.5 (8B)
- InternVL3.5 (8B)
Data Generation (CIVET Framework): To avoid data contamination and bias, the authors generated synthetic datasets using the CIVET framework.
- Grid-based: Images are $672 \times 672$ pixels, divided into a $9 \times 9$ grid. Objects are placed in distinct cells to avoid occlusion.
- Balanced: Datasets cover 10 target object types (colored plusses and white shapes) with uniform distributions across counts (1–9) and spatial locations.
- Scenarios:
  - ID (In-Distribution): Counts 1–9.
  - OOD (Out-of-Distribution): Counts 10–18 (testing generalization to higher numbers).
  - Noisy: Images containing up to 9 distractors (irrelevant objects) to test robustness.

Training Approaches

The models were fine-tuned using LoRA under two distinct paradigms:

Direct Counting (DC): The model maps the image and query directly to the final count number.
Point-then-Count (PtC): The model first predicts the grid coordinates $(n, m)$ $(n, m)$ of all target objects, then generates the final count based on these coordinates.
- Output Format: Coordinates: (n1, m1), ..., (ny, my). Answer: y

Evaluation Metrics

Accuracy: Standard counting accuracy.
Grounding Reliability: Measured via F1-score (matching predicted vs. ground truth coordinates), Exact Match (EM) (all coordinates correct), and Consistency (number of coordinates matches the final answer).
Mechanistic Ablations:
- Replacing coordinates with a generic token ("X") to test the importance of spatial info.
- Removing the image input while keeping ground-truth coordinates to test reliance on visual vs. textual cues.
- Leave-one-out coordinate ablation.

3. Key Contributions

Demonstration of Skill Learning: The paper provides evidence that PtC encourages models to learn a generalizable counting skill rather than memorizing specific count ranges. PtC models show significantly better OOD generalization (counts 10–18) compared to DC models.
Validation of Grounding: The study rigorously quantifies the reliability of pointing. Predicted coordinates are grounded in the image in >89% of cases (F1-score), validating their use as visual explanations. However, the authors identify spatial biases in some models (e.g., performance drops in specific grid regions).
Mechanistic Insight: Through ablation studies, the authors reveal that:
- Spatial Information is Key: Replacing coordinates with non-spatial tokens ("X") causes a massive drop in OOD performance, proving that the spatial signal drives generalization.
- Textual Reliance: Once coordinates are generated, models rely almost exclusively on the textual representation of the coordinates to compute the final count, largely disregarding the raw visual modality. This suggests that reasoning over structured text is easier for LLMs than direct visual counting.

4. Key Results

OOD Generalization:
- PtC outperforms DC: On counts 10–18, PtC models achieved significantly higher accuracy. For example, InternVL3.5 (8B) reached 97.33% accuracy with PtC vs. 45.38% with DC.
- DC Failure: DC models often degrade as the number of objects increases, showing a "ceiling effect" tied to training data ranges.
Robustness to Distractors:
- PtC models (specifically InternVL3.5 and Qwen2.5-VL 7B) maintained near-perfect accuracy even with 9 distractors.
- DC models and smaller PtC models (Qwen2.5-VL 3B) degraded more rapidly as distractor counts increased.
Coordinate Reliability:
- High F1: Overall F1-scores were high (>89%), but Exact Match rates were lower (e.g., 88% for Qwen2.5-VL 7B), indicating that models often get the count right but miss or hallucinate specific coordinates.
- Spatial Bias: LLaVA-OneVision showed a distinct left-to-right performance drop. InternVL3.5 was the only model with uniform performance across the grid.
Ablation Findings:
- X-Fine-Tuning: When coordinates were replaced with "X", OOD accuracy plummeted (e.g., Qwen2.5-VL 7B dropped from 97% to 3.36%), confirming spatial cues are essential for extrapolation.
- Image Removal: When the image was removed but ground-truth coordinates were provided, accuracy remained high (>98%), proving the model relies on the coordinates (text) rather than re-scanning the image for the final count.

5. Significance and Implications

Decomposition as a Strategy: The paper validates that decomposing complex vision tasks into intermediate grounding steps (pointing) is a superior training paradigm for LVLMs. It transforms the task from "visual counting" (hard for LLMs) to "counting a list of coordinates" (easy for LLMs).
Interpretability: Pointing provides a verifiable intermediate step. Even if the final count is wrong, the coordinates can be inspected to diagnose whether the error lies in detection (grounding) or aggregation (counting).
Future Directions: The authors suggest that future training paradigms should explicitly separate perception (grounding) from reasoning (aggregation) to improve transferable skill learning. They also note the computational cost of generating coordinates and call for more efficient grounding mechanisms.

In conclusion, this work demonstrates that pointing is not just a heuristic for better accuracy, but a fundamental mechanism that enables LVLMs to learn robust, generalizable counting skills by leveraging spatial information and structured reasoning.