Technical Summary: Thinking with Visual Grounding

Problem Statement

Recent Vision-Language Models (VLMs) have demonstrated the ability to generate natural-language reasoning traces (e.g., Chain-of-Thought) to solve complex visual tasks. However, a critical limitation exists: these textual reasoning traces often leave the supporting image regions implicit. While a model might state, "the red car is near the entrance," it does not explicitly identify which pixels or regions correspond to that claim. This lack of explicit linkage makes the reasoning process difficult to verify, supervise, or debug. A final answer may be correct by chance, or the reasoning trace may appear coherent while relying on hallucinated visual evidence. The paper argues that true visual thinking requires not just step-by-step textual reasoning, but explicit, verifiable links between reasoning steps and the specific visual evidence in the image.

Methodology

1. Visually Grounded Thinking Format

The authors propose a new reasoning format where models interleave natural-language thoughts with explicit coordinate-based groundings. Whenever a reasoning step references a visual object, the model outputs a tag containing the object's name and its spatial coordinates. Two modes are supported:

• Box Mode: <obj> name | [x1, y1, x2, y2] </obj>
• Point Mode: <obj> name | [x, y] </obj>

2. Scalable Data Synthesis Pipeline

To train this behavior, the authors constructed a pipeline to synthesize aligned training data (for both Supervised Fine-Tuning and Reinforcement Learning) from open-source datasets (e.g., TallyQA, VSR, SpatialMQA). The pipeline involves:

• Distilling Reasoning Traces: Using high-capacity VLMs (Qwen3-VL-Plus/Qwen3.5-Plus) to generate correct reasoning traces for image-question pairs.
• Extracting Groundable Objects: An LLM identifies the specific visual objects required for the reasoning (e.g., answer objects, spatial anchors, counted instances) and assigns them a name and a disambiguating context (e.g., "red car" vs. "red car in the back row").
• Agentic Visual Grounding: A SAM3-based agent, powered by a VLM, iteratively grounds these extracted objects. The agent uses tools to propose noun phrases, invoke SAM3 for instance masks, verify masks via rendering and zooming, and select the final masks.
• Deriving Supervision: The resulting Run-Length Encoded (RLE) masks are converted into either normalized bounding boxes or interior points (farthest from the boundary) to create aligned SFT and RL datasets.

3. Grounding-Aware Reinforcement Learning (RL)

The authors introduce a novel reward mechanism that jointly optimizes for answer correctness and grounding quality.

• Grounding Object Routing: Before scoring, a lightweight VLM router matches model-generated grounding objects to the ground-truth objects (based on name and context) to handle variations in wording and disambiguation.
• Dense Grounding Rewards:
  • Box Mode: Uses Intersection-over-Union (IoU) between the union of generated boxes and ground-truth boxes for each matched object.
  • Point Mode: Uses an F1 score based on whether generated points fall inside the corresponding ground-truth masks, preventing duplicate credit for the same instance.
• Final Reward: The total reward combines the dense grounding reward with sparse rewards for answer correctness, thinking format, and grounding format.

Key Contributions

1. Scalable Data Pipeline: A system centered on a SAM3-based agentic grounding agent that synthesizes high-fidelity visually grounded reasoning data for both SFT and RL, eliminating the need for manual coordinate annotation.
2. Grounding Reward Design: A reward function that directly supervises the quality of intermediate visual references, supporting both box and point grounding modes.
3. Empirical Validation: Controlled experiments demonstrating that visually grounded thinking significantly improves performance on counting and spatial reasoning tasks, with 4B models matching or surpassing 27B baselines in specific spatial benchmarks.

Experimental Results

The authors evaluated their approach using Gemma3-4B-IT as the base model across two counting benchmarks (TallyBench, CountQA) and four spatial reasoning benchmarks (VSR-zeroshot, EmbSpatial, SpatialMQA, MultihopSpatial).

• Performance Gains: Visually grounded thinking models consistently outperformed both the base Gemma3-4B-IT and a "non-grounded thinking" baseline.
• Scale Comparison: On spatial reasoning tasks, the 4B visually grounded models achieved performance comparable to, and in some cases (SpatialMQA, MultihopSpatial) exceeding, the much larger Gemma3-27B-IT.
• Stability: The non-grounded baseline suffered from "length collapse" during RL training (responses becoming too short), whereas grounded variants maintained stable rollouts, likely due to the structural constraints of grounding tags.
• Grounding Interface Differences:
  • Counting: Point grounding consistently outperformed box grounding, suggesting that instance-level localization is sufficient for counting tasks.
  • Spatial Reasoning: Box grounding provided clear gains when the grounding reward was applied, likely due to the geometric information (extent, overlap) boxes provide. However, point and box grounding were broadly comparable in overall accuracy on these tasks.
• Reward Impact: The grounding reward provided consistent benefits for box-mode training, particularly on spatial tasks. For point-mode, the reward signal was coarser (discrete F1 score vs. smooth IoU), resulting in less consistent downstream accuracy gains in the current experiments.

Significance and Claims

The paper claims that visual thinking must evolve beyond generating plausible text to producing evidence-backed reasoning. By tying intermediate thoughts to specific image regions, models can be supervised more effectively, and their reasoning becomes verifiable. The authors posit that the next step for visual thinking is not merely "longer thinking," but "thinking tied to the image" in a form that can be checked and improved. Their results suggest that explicit grounding allows smaller models (4B) to achieve the reasoning capabilities of much larger models (27B) on spatial tasks, highlighting the efficiency of grounding-based supervision.