KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding

1. Problem Definition: Knowledge-Intensive Visual Grounding (KVG)

The paper identifies a critical limitation in current Multimodal Large Language Models (MLLMs): the "Knowledge-Grounding Gap."

• The Phenomenon: MLLMs often possess rich internal knowledge to correctly identify fine-grained entities (e.g., distinguishing a "Clumber Spaniel" from an "American Water Spaniel") but fail to translate this knowledge into accurate spatial localization (bounding boxes) when required.
• The Task: The authors define Knowledge-Intensive Visual Grounding (KVG). Unlike standard Visual Grounding which uses generic descriptions (e.g., "the dog on the left"), KVG requires grounding based on specific, domain-level entity names (e.g., "the Clumber Spaniel") in images containing visually similar distractors.
• The Challenge: Current models struggle because they rely on coarse visual patterns rather than integrating specific entity knowledge with visual evidence. Standard reasoning prompts or generic reinforcement learning (RL) fail to bridge this gap effectively.

2. Methodology: The KARL Framework

To address the knowledge-grounding gap, the authors propose a two-stage training paradigm called KARL (Knowledge-Aware Reasoning and Reinforcement Learning).

Stage 1: Knowledge-Guided Grounding Data Construction (CoT-SFT)

• Data Synthesis: Since high-quality KVG data is scarce, the authors synthesize training data by combining images from Fine-Grained Visual Recognition (FGVR) datasets (e.g., aircraft, cars, birds). They create composite images containing multiple visually similar entities to force fine-grained discrimination.
• Reasoning Generation: They use powerful teacher models (Qwen2-VL-72B and Qwen3-VL-32B) to generate Chain-of-Thought (CoT) rationales. These rationales explicitly guide the model to:
  1. Analyze visual features.
  2. Compare the target entity against distractors using domain-specific knowledge.
  3. Verify knowledge against visual evidence before predicting a bounding box.
• Training: The base model undergoes Supervised Fine-Tuning (SFT) on this CoT data to establish a "knowledge-guided reasoning" initialization.

Stage 2: Knowledge-Aware Reinforcement Learning (KARL)

The core innovation is the Knowledge-Aware Reinforcement Learning framework, which optimizes the Stage-1 model using Group Relative Policy Optimization (GRPO).

• Entity Knowledge Estimation: Before training, the system estimates the model's "knowledge mastery" for each specific entity (e.g., does the model know what a "Fokker 50" looks like?). This is done via a multiple-choice recognition task. Entities are categorized into discrete knowledge levels (from low to high mastery).
• Knowledge-Aware Reward Scaling: Instead of applying uniform rewards, KARL dynamically modulates the reinforcement signal based on the entity's knowledge level:
  • Low-Knowledge Entities: Receive larger positive rewards for correct grounding and weaker negative penalties for errors. This encourages exploration and learning for concepts the model is weak on.
  • High-Knowledge Entities: Receive smaller positive rewards and stronger negative penalties. This prevents the model from becoming complacent and forces it to refine its precision for concepts it already knows.
• Reward Function: The total reward ($R$) combines a Knowledge-Aware IoU (KA-IoU) reward (spatial accuracy modulated by knowledge level) and a Format Reward (ensuring valid output structure).

3. Key Contributions

1. KVG Task & Benchmark (KVG-Bench): The authors introduce a new benchmark spanning 10 domains (e.g., Aircraft, Dogs, Landmarks) with 1.3K test cases and 882 distinct entities. It includes both "Seen" (in-domain) and "Unseen" (out-of-domain) splits to rigorously test generalization.
2. Knowledge-Guided Reasoning Strategy: A novel data construction method that forces models to articulate entity-level knowledge before making grounding predictions, bridging the gap between recognition and localization.
3. KARL Framework: A reinforcement learning approach that adapts optimization signals based on the model's specific knowledge mastery of different entities, leading to more balanced learning across heterogeneous concepts.

4. Experimental Results

The authors evaluated KARL (built upon Qwen3-VL-8B) against strong baselines, including large-scale MLLMs (7B-76B parameters), specialist grounding models (YOLO-World, GroundingDINO), and recent reasoning-guided models.

• Performance on KVG-Bench:
  • Overall: KARL achieved 68.41% accuracy, outperforming all baselines.
  • Unseen Categories: KARL showed superior cross-domain generalization, achieving 70.04% on unseen categories (vs. 58.46% for the next best reasoning model, UniVG-R1).
  • Comparison: It significantly outperformed the base Qwen3-VL-8B (56.94%) and even larger models like Qwen3-VL-32B (63.38%).
• General Capabilities: KARL maintained performance on general multimodal benchmarks (MMStar, RealWorldQA), proving that the specialized training did not degrade general abilities.
• Ablation Studies:
  • Removing the knowledge-aware reward scaling (using standard GRPO) resulted in a significant drop in performance on unseen categories (70.04% $\to$ 66.36%).
  • The two-stage approach (CoT-SFT followed by KARL) was proven essential; direct SFT or standard RL without knowledge awareness yielded suboptimal results.

5. Significance

• Bridging the Gap: The paper provides empirical evidence that the failure of MLLMs in fine-grained grounding is not a lack of knowledge, but a failure to utilize that knowledge during the localization process.
• Efficient Generalization: By adapting reinforcement signals to the model's specific knowledge gaps, KARL achieves better generalization to unseen categories without requiring massive model scaling.
• Paradigm Shift: The work suggests that future visual grounding systems must move beyond generic reasoning and incorporate entity-aware optimization, treating different concepts with different learning pressures based on the model's current proficiency.

In summary, KARL demonstrates that explicitly aligning internal entity knowledge with visual grounding decisions through adaptive reinforcement learning is the key to solving complex, knowledge-intensive visual tasks.