🤖 AI

GRIT: Teaching MLLMs to Think with Images

The paper proposes GRIT, a reinforcement learning-based method that enables Multimodal Large Language Models to generate visually grounded reasoning chains by interleaving natural language with explicit bounding box coordinates, achieving high data efficiency without requiring annotated reasoning or localization labels.

Original authors: Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, Xin Eric Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart robot how to solve a puzzle. Usually, when you ask this robot a question about a picture, it tries to answer by just talking. It might say, "I see a bird," but it doesn't actually point to the bird. It's like trying to describe a location in a dark room using only words, hoping you can guess where you are looking.

The paper "GRIT: Teaching MLLMs to Think with Images" introduces a new way to teach these robots (called Multimodal Large Language Models, or MLLMs) to think differently. Here is the breakdown using simple analogies:

1. The Problem: The "Blind" Thinker

Current AI models are great at talking, but when they look at a picture, they often "think" in pure text. They might guess the right answer, but they can't show you where they looked to find it. It's like a detective solving a crime by guessing the culprit's name without ever showing the evidence or pointing to the fingerprint. This makes their reasoning hard to trust and sometimes inaccurate.

2. The Solution: The "Pointer" Habit (GRIT)

The authors created a method called GRIT (Grounded Reasoning with Images and Text). Instead of just talking, they teach the AI to point while it thinks.

The Analogy: Imagine the AI is a teacher explaining a map to a student. Instead of just saying, "The treasure is over there," the teacher draws a box around the spot on the map and says, "I am looking here [draws box], and I see a rock, so the treasure must be next to it."
How it works: The AI generates a chain of thoughts that mixes normal sentences with bounding boxes (coordinates that draw a rectangle around a specific part of the image). It literally points to the image parts it is using to solve the problem.

3. The Secret Sauce: Learning by Trial and Error (GRPO-GR)

Usually, to teach an AI to do something this complex, you need thousands of examples where humans have written out every single step and drawn every single box. That is like hiring a team of teachers to write a textbook for the robot.

The paper claims a breakthrough: GRIT needs almost no examples.

The Analogy: Instead of reading a textbook, the AI learns like a child playing a video game. You give it a picture and a question. It tries to answer. If it gets the final answer right and follows the rules (like drawing a box), it gets a "high score" (reward). If it fails, it gets a low score.
The Magic: The researchers only used 20 examples (like 20 practice questions) to train the AI. The AI figured out the pattern: "To get a high score, I need to point to the image while I talk." They call this GRPO-GR, a special training method that rewards the AI for getting the answer right and for using the correct "pointing" format.

4. The Results: Smarter and More Honest

When they tested these trained robots:

They became better at math and counting: If you asked, "How many eggs are in the nest?", the robot didn't just guess a number. It pointed to the eggs, counted them one by one in its "thoughts," and then gave the answer.
They unified two skills: Before, the AI was good at talking (reasoning) OR good at pointing (grounding), but not both at the same time. GRIT taught them to do both simultaneously.
They are more reliable: Because the AI has to point to the evidence, it's harder for it to make up lies. If it points to a spot and says "I see a cat," but there is no cat there, the system knows something is wrong.

5. What They Found (and What They Didn't)

Data Efficiency: They proved you don't need a massive library of data. Just 20 examples were enough to teach the robot this new "pointing" habit.
Attention: When the AI points to a spot (draws a box), it actually pays more attention to that part of the image in its next thought. It's like the act of pointing helps the robot focus its eyes.
Limits: The paper notes that while adding more data helps, there is a point of diminishing returns. To get even better at things the robot hasn't seen before, they need more variety in the data, not just more of the same.

In Summary:
GRIT is a new training method that teaches AI robots to "think with their eyes" by forcing them to point at the image while they talk. It's a huge leap because it teaches this complex skill with almost no data (just 20 examples) and makes the AI's reasoning transparent and trustworthy.

, ...`) and the syntactic validity of bounding boxes (detected via regex). This reward does not constrain the semantic accuracy of the text or the specific regions, only the format.
2. Grounded-Target-Counting Reward ( $r_{count}$ ): An optional reward for counting tasks where the number of generated bounding boxes must match the ground-truth count.
3. GPT-Aided Answer-Accuracy Reward ( $r_{ans}$ ): Uses an external VLM (GPT-4o) to judge the correctness of the final answer, supplemented by a BLEU score.

Key Advantage: Because the rewards focus on format and final answer accuracy rather than intermediate reasoning steps or specific bounding box locations, GRIT eliminates the need for training data with reasoning chain annotations or explicit bounding box labels.

Key Contributions

GRIT Framework: A novel method enabling MLLMs to "think with images" by generating reasoning chains that interleave natural language with explicit bounding box coordinates, unifying grounding and reasoning abilities.
GRPO-GR Algorithm: A reinforcement learning algorithm that efficiently trains MLLMs using only image-question-answer triplets. It utilizes novel format and accuracy rewards, removing the dependency on expensive, dense annotations for intermediate reasoning or grounding.
Data Efficiency: The method demonstrates exceptional data efficiency, achieving grounded reasoning capabilities with as few as 20 training samples drawn from existing VQA datasets (VSR and TallyQA).
Unified Capabilities: Experiments show that GRIT-trained models successfully unify grounding and reasoning, which are often disconnected in base MLLMs, leading to coherent and accurate outputs.

Experimental Results

The authors evaluated GRIT-trained models (Qwen 2.5-VL and InternVL 3) against baselines including Direct Query, Chain-of-Thought (CoT), One-shot ICL, and Few-shot SFT across seven testing sets (VSR, TallyQA, GQA, MME, MathVista-mini, OVDEval).

Performance: GRIT-trained models outperformed all baselines in both GPT-as-judge answer accuracy and Grounding IoU (Intersection over Union of generated boxes vs. ground truth). Notably, they generalized effectively to out-of-domain data despite being trained on only 20 samples.
Grounding Quality: Qualitative analysis revealed that GRIT models dynamically determine when grounding is necessary, avoiding false positives for non-existent entities.
Cross-Modal Correlation: A new metric, "Vision-Language Reasoning Cross-Modal Correlation," showed that the image regions selected by GRIT models are highly semantically coherent with the accompanying text, outperforming Zero-shot ICL and Few-shot SFT baselines.
Attention Mechanism: Analysis of attention scores indicated that the presence of bounding boxes in the reasoning chain significantly increases the model's attention to relevant visual tokens in subsequent reasoning steps.
Scaling: Increasing training data from 20 to 7,000 samples improved in-domain performance but showed diminishing returns and generalization challenges for out-of-domain tasks, suggesting that data diversity is more critical than volume for generalization.

Significance and Claims

The paper claims that GRIT represents a significant step forward in enabling MLLMs to perform visually grounded reasoning. By allowing models to generate explicit visual references within their thought processes, GRIT achieves a "successful unification of reasoning and grounding abilities."

The authors emphasize that this is the first open-source approach to achieve interleaved visual-textual reasoning via lightweight RL without needing explicit annotations for intermediate steps. While the study focuses on smaller MLLMs to validate efficiency rather than peak performance, the results suggest that instilling complex grounded reasoning with minimal data is feasible. The paper concludes that future improvements in generalizability will likely depend more on the diversity of training data and model pre-training than simply scaling up data volume.