GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations

GroundCount proposes a framework that augments Vision-Language Models with explicit spatial grounding from object detection models to significantly mitigate counting hallucinations, demonstrating that structured prompt-based integration outperforms feature-level fusion and yields consistent accuracy improvements across most architectures.

The Gist

Imagine you have a very smart, well-read assistant (let's call him The Vision-Language Model or VLM). This assistant is great at looking at a picture and writing a story about it. He can tell you the mood of the scene, describe the colors, and even guess what people are feeling.

But there's one thing he is terrible at: counting.

If you show him a picture with five apples, he might confidently say, "I see three apples," or "There are probably seven." He gets so distracted by his own thoughts and the story he's trying to tell that he loses track of the actual number of items. It's like a brilliant professor who can write a beautiful essay but keeps forgetting to count the pages.

This paper, titled GroundCount, introduces a solution to fix this specific problem.

The Problem: The "Daydreaming" Expert

The researchers found that even the most advanced AI assistants (the "State-of-the-Art" models) struggle to count objects. They get confused by the visual details and rely too much on what they think should be there, rather than what is there. This is called a hallucination.

The Solution: The "Eagle-Eyed Spotter"

To fix this, the authors paired the "Daydreaming Expert" (the VLM) with a specialized tool called an Object Detection Model (ODM). Think of the ODM as a hyper-focused security guard or a robotic spotter.

• The Spotter (ODM): This tool is not very creative, but it is incredibly good at one thing: finding things. It scans the image and says, "I see a dog here, a cat there, and two birds over there." It draws invisible boxes around them and gives them a confidence score (e.g., "99% sure this is a dog").
• The Expert (VLM): This is the smart storyteller who usually fails at counting.

How They Work Together: The "Grounding" Trick

The paper proposes three ways to make these two work together, but the best one is surprisingly simple.

1. The "Cheat Sheet" Method (Plan A - The Winner)

Instead of trying to rebuild the Expert's brain, the researchers just hand him a cheat sheet.

• First, the Spotter looks at the image and writes a list: "1 dog in the bottom-left, 2 cats in the top-right."
• Then, they feed this list to the Expert along with the picture.
• The Expert reads the list, looks at the picture to double-check, and says, "Ah, the list says two cats. I see them too. The answer is two."

Why this is amazing:

• It's fast: The Spotter works in a split second.
• It stops daydreaming: Because the Expert has the facts right in front of him, he doesn't waste time guessing or re-checking the image endlessly.
• It's accurate: The accuracy jumped from about 75% to over 81%.

2. The "Brain Surgery" Method (Plan B - Feature Fusion)

The researchers also tried a more complex approach: surgically connecting the Spotter's brain directly to the Expert's brain at the neural level. They tried to make the Expert "feel" the Spotter's data.

• The Result: This was like trying to teach a fish to fly by attaching wings. It was complicated, required a lot of training, and actually performed worse than just giving the Expert a cheat sheet. The two "brains" didn't speak the same language well enough.

3. The "Hybrid" Method (Plan C)

They tried doing both the Cheat Sheet and the Brain Surgery. It was okay, but the simple Cheat Sheet (Plan A) was still the champion.

Key Takeaways (The "Aha!" Moments)

• Less is More: The researchers found that giving the Expert too much information (like low-confidence guesses from the Spotter) actually confused him. It's better to give him a short, high-quality list than a long, messy one.
• Position Matters: Telling the Expert where things are (e.g., "top-left") helps the smartest models, but confuses the weaker ones.
• The "Spotter" is Already Good: The Spotter (the Object Detection model) could actually count the objects on its own with 73% accuracy. But when you combine it with the Expert's reasoning, the team gets 81% accuracy. The Expert adds the "context" (understanding the scene), while the Spotter adds the "facts" (the count).

The Big Picture

This paper teaches us that sometimes, the best way to fix a smart AI isn't to make it smarter or more complex. Sometimes, you just need to give it a second pair of eyes that is really good at the specific task it's failing at.

By pairing a creative, reasoning AI with a precise, counting AI, they created a team that is reliable, fast, and much less likely to make silly counting mistakes. This is a huge step forward for using AI in real-world tasks like inventory management, helping the visually impaired, or educational tools where getting the number right actually matters.

Technical Summary

Here is a detailed technical summary of the paper "GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinations."

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant progress in multimodal reasoning but suffer from persistent hallucinations, particularly in counting tasks.

• The Discrepancy: Despite advances in reasoning capabilities (e.g., iterative reflection, reinforcement learning), counting remains the lowest-accuracy task for state-of-the-art (SOTA) VLMs. On the PhD benchmark, VLM counting accuracy ranges from 64.0% to 74.7%, significantly lower than object recognition (70.3%–89.6%) or attribute identification (83.2%–86.5%).
• The Cause: The authors hypothesize that this stems from an imbalanced cross-modal attention mechanism where language decoders over-attend to textual priors and under-utilize visual tokens. Furthermore, the autoregressive nature of VLMs makes it difficult to maintain accurate object tallies without explicit spatial grounding.
• Limitations of Current Fixes: Existing mitigation strategies (decoding adjustments, layer-level steering) are often ineffective for reasoning-capable VLMs because iterative reasoning processes lack a single "correct" token to steer toward, and reflection mechanisms often fail to filter out spurious textual information in compositional tasks like counting.
• The Opportunity: Convolutional Neural Network (CNN)-based Object Detection Models (ODMs), such as YOLO, excel at spatial localization and instance counting with minimal computational overhead, providing structured outputs (bounding boxes, confidence scores, class labels) that VLMs lack.

2. Methodology: The GroundCount Framework

The authors propose GroundCount, a framework that augments VLMs with explicit spatial grounding from ODMs to mitigate counting hallucinations. They evaluate three distinct implementation strategies:

A. GroundCount A: Prompt-Based Augmentation (Plan A)

• Mechanism: This is a "plug-and-play" approach requiring no architectural changes or training.
• Process:
  1. An input image is processed by an ODM (YOLOv13x).
  2. Detections are converted into structured natural language text.
  3. Spatial Encoding: Image space is discretized into a 3×3 grid (e.g., "upper-left," "middle-center").
  4. Sequencing: Objects are ordered by horizontal position (left-to-right) then vertical (lower-to-upper).
  5. The structured text (e.g., person 1 upper-center: 0.93) is appended to the user prompt as context.
• Benefit: Provides explicit, interpretable spatial priors to the VLM, reducing the need for the model to "guess" object locations.

B. GroundCount B: Feature-Level Fusion (Plan B)

• Mechanism: A deep learning approach that fuses ODM features with VLM vision encoder features.
• Architecture: A lightweight fusion network sits between the VLM's vision encoder and language decoder.
  • Dual-Branch Design:
    • Branch A (FiLM): Uses Feature-wise Linear Modulation to allow CNN features to adaptively modulate ViT patch embeddings.
    • Branch B (Cross-Attention): Allows ViT patches to selectively query relevant CNN features.
  • Information Bottleneck: Dimensionality reduction is applied to force integration of both modalities.
• Training: Requires fine-tuning the fusion network and optionally the VLM/ODM. The training data is constructed using ground-truth annotations formatted to match the ODM output structure.

C. GroundCount C: Hybrid Approach (Plan C)

• Mechanism: Combines the prompt augmentation of Plan A with the feature-level fusion of Plan B.
• Goal: Leverage both implicit feature integration and explicit symbolic reasoning.

3. Key Contributions

1. Systematic Analysis: Demonstrated that counting is the most failure-prone task for SOTA VLMs, even those with advanced reasoning mechanisms, highlighting a fundamental spatial-semantic integration limitation.
2. GroundCount Framework: Introduced a novel augmentation pipeline that significantly boosts counting accuracy (up to 81.3%) with negligible memory overhead.
3. Ablation Insights: Conducted comprehensive studies revealing:
   • Positional Encoding: Critical for stronger models but detrimental for weaker ones.
   • Confidence Scores: Often introduce noise; removing them improved performance in 4 out of 5 models.
   • Detection Threshold: High precision (0.5 threshold) is far more important than recall; lowering the threshold drastically hurts performance due to false positives.
4. Architecture Comparison: Proved that explicit symbolic grounding via prompts outperforms implicit feature-level fusion, challenging the assumption that deeper architectural integration is always superior for multimodal tasks.

4. Experimental Results

The framework was evaluated on the PhD benchmark across five SOTA VLMs (Molmo2-4B, Ovis2.5-2B, R-4B, Qwen3-VL-2B, InternVL3.5-1B).

• Performance (Plan A):
  • Achieved 81.3% accuracy on Ovis2.5-2B (a 6.6 percentage point improvement over the 74.7% baseline).
  • Consistent improvements of 6.2–7.5pp across four of the five models.
  • Efficiency: Reduced inference time by 22% (from 10.0s to 7.8s) for Ovis2.5-2B. The explicit grounding prevents the VLM from entering "hallucination loops" where it endlessly re-checks the image.
• Performance (Plan B & C):
  • Feature fusion (Plan B.4) achieved 78.0% accuracy, which is good but lower than the prompt-only approach.
  • The hybrid approach (Plan C) achieved 78.2% but was the fastest (4.8s) among augmented models.
• Model Variations:
  • InternVL3.5-1B was an outlier, showing degraded performance with full augmentation. This suggests its iterative reflection mechanisms are incompatible with structured prompts.
  • ODM-Only Baseline: YOLO alone achieved 72.8% accuracy in 0.1s, proving that the detection capability exists, but VLMs struggle to extract it without grounding.

5. Significance and Implications

• Paradigm Shift: The paper suggests that for specific compositional tasks like counting, explicit symbolic grounding (via prompts) is more effective than implicit feature fusion. This implies that current VLMs are better at processing structured text than learning complex cross-modal alignments from scratch.
• Practical Impact: GroundCount offers a low-cost, high-impact solution for improving VLM reliability in real-world applications such as inventory management, accessibility tools, and educational technologies.
• Efficiency: By reducing hallucination-driven reasoning loops, the method not only improves accuracy but also speeds up inference, making it viable for real-time applications.
• Future Directions: The work highlights the need for architectural compatibility in augmentation strategies and suggests exploring transformer-based ODMs (like DETR) to further reduce the representational gap between CNNs and ViTs.

In conclusion, GroundCount demonstrates that bridging the gap between the spatial precision of object detectors and the reasoning capabilities of VLMs via structured prompts is a highly effective strategy for mitigating counting hallucinations.