Technical Summary: iVGR – Internalizing Visually Grounded Reasoning for MLLMs with Reinforcement Learning

1. Problem Statement

Multimodal Large Language Models (MLLMs) have made significant strides in general reasoning, yet they struggle with fine-grained perception in high-resolution or complex visual scenes. While Visually Grounded Chain-of-Thought (CoT) has emerged as a promising paradigm to enhance perception by explicitly anchoring reasoning to visual regions (via dynamic crops or bounding boxes), its efficacy during the inference phase remains underexplored.

Empirical analysis in this work reveals a counter-intuitive finding: mandating explicit object boxes or crop tools during inference often degrades performance compared to standard textual CoT.

• Tool-based approaches (e.g., DeepEyes) that invoke external crop tools suffer when crop localization is poor, introducing visual noise that harms answer prediction.
• Explicit grounding approaches (e.g., TreeVGR) that require generating bounding boxes within the CoT consistently underperform textual CoT across all localization quality intervals.

The authors hypothesize that the visual localization capability can be internalized into the textual reasoning process. The mandatory generation of explicit coordinates during inference introduces unnecessary interference, distracting the model from its primary objective of answer prediction.

2. Methodology: iVGR

To address this, the authors propose iVGR (Internalizing Visually Grounded Reasoning), a novel reinforcement learning (RL) framework designed to transfer localization capabilities from a grounded stream into a purely textual reasoning stream.

Dual-Stream Training Strategy

The core of iVGR is a dual-stream training mechanism where, for each training query, the policy MLLM generates two distinct streams of rollouts:

1. Grounded Stream: The model is prompted to explicitly predict bounding boxes (<box>[x1, y1, x2, y2]</box>) when referring to objects. This stream is rewarded for format, answer accuracy, and localization quality (IoU).
2. Textual Stream: The model performs standard natural language reasoning without explicit grounding constraints.

Consistency Reward

The critical innovation is the Consistency Reward, which aligns the textual stream with high-quality reasoning from the grounded stream.

• Reference Selection: A "Rollout Archive" maintains the highest-quality grounded CoT (based on IoU and accuracy) found so far for each query.
• Semantic Alignment: An external LLM judge evaluates the semantic consistency between the textual CoT and the archived grounded reference. The judge assigns a score (0.0 to 1.0) based on whether the textual description matches the visual facts established in the grounded reference (e.g., omitting details, adding hallucinations, or contradicting facts).
• Optimization: The textual stream is optimized to maximize this consistency reward, effectively learning to "see" and localize objects implicitly within its text without outputting coordinates.

Tool-Assisted Test-Time Scaling

While the model is trained to internalize grounding, iVGR retains the flexibility to use external tools. At test time, a workflow is proposed where the model first generates a grounded CoT to extract bounding boxes. These boxes are then used to crop the image (including a "union crop" of all regions), and the model performs reasoning on these multi-view inputs using standard textual CoT.

3. Key Contributions

• Empirical Analysis: The paper demonstrates that explicit visual grounding during inference often harms performance, suggesting that localization capabilities can be implicitly transferred to textual CoT.
• iVGR Framework: A reinforcement learning-based dual-stream strategy that uses a consistency reward to internalize visual localization into textual reasoning, eliminating the need for explicit coordinate generation at inference.
• Scalability and Compatibility: The method is validated on Qwen2.5-VL and Qwen3-VL models, showing significant improvements across fine-grained and general benchmarks while remaining compatible with tool-assisted workflows.

4. Experimental Results

Extensive experiments were conducted on fine-grained VQA benchmarks (V*, HR4K, HR8K) and general reasoning tasks (MME-RW-Lite, POPE, RealWorldQA, etc.).

• Performance Gains: iVGR significantly outperforms existing baselines. On the Qwen2.5-VL-7B model, iVGR achieves an average improvement of 6.5% across benchmarks, with substantial gains in fine-grained tasks (e.g., +10.4% on HR8K).
• Comparison with SOTA: It surpasses both tool-based methods (like DeepEyes) and explicit grounding methods (like TreeVGR) without requiring external tools during the primary inference phase.
• Internalization Evidence: Attention score analysis shows that the iVGR textual stream attends to image tokens more strongly than even the grounded-stream-only variant, confirming that the model has successfully internalized visual grounding.
• Tool Synergy: When equipped with the proposed tool-assisted test-time scaling (using crops), performance improves further (e.g., +1.5% on HR4K for Qwen3-VL-8B), demonstrating the method's flexibility.

5. Significance and Claims

The paper claims that iVGR offers a more efficient and robust paradigm for fine-grained visual reasoning. By internalizing the localization capability, the model avoids the noise and interference caused by mandatory explicit coordinate generation during inference.

The authors position iVGR as a method that:

1. Decouples the capability of visual localization from the necessity of explicit output formats.
2. Enhances fine-grained perception without sacrificing the flexibility of standard textual reasoning.
3. Maintains compatibility with external tools, allowing for a hybrid approach where tools can be leveraged at test time if higher precision is required.

The work suggests that the future of visually grounded reasoning lies not in forcing models to output coordinates, but in training them to reason with an implicit, internalized understanding of visual space.