🤖 AI

DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model

This paper introduces DiG (Differential Grounding), a novel framework that enhances fine-grained visual perception in Multimodal Large Language Models by training them to localize differences between image pairs using an automated 3D rendering data pipeline and curriculum learning, resulting in significant performance improvements across various visual reasoning benchmarks.

Original authors: Zhou Tao, Shida Wang, Yongxiang Hua, Haoyu Cao, Linli Xu

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhou Tao, Shida Wang, Yongxiang Hua, Haoyu Cao, Linli Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a pair of identical twins. They look exactly the same at first glance. But if you look closely, one twin is wearing a red hat instead of a blue one, and the other is missing a button on their shirt.

Most current AI models (Multimodal Large Language Models, or MLLMs) are like people who are great at recognizing that "these are two humans," but they often miss the tiny details like the hat color or the missing button. They see the big picture but miss the fine print.

This paper introduces a new training method called DiG (Differential Grounding) to teach AI to become a super-sleuth, spotting even the tiniest differences between two images.

Here is how it works, broken down into simple concepts:

1. The Game: "Find the Differences"

Think of the DiG task as playing the classic "Spot the Difference" puzzle you might find in a children's magazine.

The Setup: The AI is shown two very similar images side-by-side.
The Challenge: It has to find every difference (a changed color, a missing object, a new item) and draw a box around where that difference is.
The Twist: The AI doesn't know how many differences there are. It has to hunt for them all on its own.

2. The Training Ground: A 3D Video Game Studio

You can't train a detective by showing them thousands of real photos and asking humans to draw boxes on every single difference (that would take forever and cost a fortune).

Instead, the authors built a 3D rendering pipeline (like a video game engine).

The Metaphor: Imagine a robot director in a virtual studio. It sets up a scene with toy blocks. Then, it secretly swaps a red block for a blue one, or removes a small sphere.
The Benefit: Because the robot knows exactly what it changed, it automatically knows the perfect answer (the "ground truth"). This allows them to generate millions of practice puzzles instantly without any human help.

3. The Learning Curve: From "Baby Steps" to "Marathon"

If you threw a beginner into a puzzle with 10 hidden differences, they would get frustrated and give up. The AI faces the same problem. At first, it can't find any differences, so it gets no "points" (rewards) and stops learning.

To fix this, the authors use Curriculum Learning, which is like a video game with levels:

Level 1 (Easy): The AI only has to find one difference. It gets lots of points and learns the basic skill of "looking closely."
Level 2 (Medium): Now there are two differences. The AI has to juggle finding both.
Level 3 (Hard): The AI faces scenes with multiple differences and doesn't even know how many to look for. It has to be a true detective.

By starting easy and getting harder, the AI builds confidence and skill without getting overwhelmed.

4. The Reward System: The Coach's Whistle

The AI learns using a method called Reinforcement Learning. Think of a coach giving feedback:

Format Reward: "Did you write your answer in the right format?" (Yes/No).
Accuracy Reward: "Did you find the right objects?" (F1 Score).
Precision Reward: "Did you draw the box exactly around the object, or was it too big?" (IoU).

The AI tries to maximize these points. Over time, it learns that to get the highest score, it must be both thorough (find everything) and precise (draw the boxes perfectly).

Why Does This Matter?

Once the AI masters this "Spot the Difference" game, something magical happens: The skills transfer.

Just like a person who practices spotting small details in a game becomes better at spotting errors in a legal contract or finding a specific ingredient in a recipe, the AI becomes better at:

Answering tricky questions: "Is the cat's tail longer than the dog's?"
Finding objects: "Point to the red cup on the table."
Avoiding hallucinations: It stops making up things that aren't there because it's trained to look for what actually exists.

The Bottom Line

DiG is a clever way to turn AI training into a giant, automated game of "Spot the Difference." By starting with simple puzzles and gradually making them harder, the AI learns to pay attention to the tiny details it used to ignore. This makes it a sharper, more reliable observer of the visual world.

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved remarkable success in holistic scene understanding and high-level semantic alignment. However, they suffer from significant limitations in fine-grained visual perception and precise spatial reasoning.

The Gap: Current models often overlook subtle visual cues such as small object changes, color variations, or missing elements. They struggle to associate global semantics with precise spatial locations.
Existing Limitations:
- Supervised Fine-Tuning (SFT): Traditional grounding datasets (e.g., RefCOCO) rely on costly manual annotation, limiting scalability and diversity. Fine-tuning on these often leads to overspecialized models that generalize poorly.
- Proxy Tasks: Existing proxy tasks (e.g., visual jigsaw puzzles, hallucination detection) provide auxiliary supervision but are not explicitly designed to strengthen fine-grained difference localization.
- Reinforcement Learning (RL) Challenges: Applying RL to visual perception is difficult due to reward sparsity. At initialization, models rarely localize differences correctly, causing rewards to collapse near zero and preventing stable optimization.

2. Methodology: The DiG Framework

The authors propose DiG (Differential Grounding), a novel proxy task framework that trains MLLMs to identify and localize all visual discrepancies between paired images without prior knowledge of the number of differences.

A. Automated 3D Data Generation Pipeline

To enable scalable training without manual annotation, the authors developed a fully automated pipeline using the Blender 3D rendering engine:

Procedural Generation: A base 3D scene is generated with objects defined by attributes (shape, color, size, material, position).
Controlled Discrepancies: A stochastic process modifies the base configuration to create a "counterpart" image. Modifications include object addition, removal, or attribute changes (e.g., color, shape).
Perfect Ground Truth: Because modifications are defined programmatically in 3D space, the 2D bounding boxes for differences are automatically extracted from the renderer, ensuring 100% accurate annotations.
Scalability: The pipeline can generate infinite high-quality pairs with controllable difficulty (number of differences, subtlety, location).

B. Reinforcement Learning with GRPO

The model is optimized using Group Relative Policy Optimization (GRPO) with a structured reward function comprising three components:

Format Reward ( $r_{format}$ ): A binary signal ensuring the output is a parsable list of bounding boxes (syntactic validity).
Accuracy Reward ( $r_{acc}$ ): A composite metric combining:
- F1 Score: Measures detection completeness (Precision/Recall) using Hungarian matching between predicted and ground-truth boxes.
- IoU (Intersection over Union): Measures spatial alignment precision.
Overall Objective: A weighted combination ( $r = (1-\alpha)r_{acc} + \alpha r_{format}$ ) that prioritizes visual reasoning while maintaining structural consistency.

C. Curriculum Learning Strategy

To address the sparsity of difference signals (where early models get near-zero rewards), the authors employ a Curriculum Learning approach:

Stage 1 (Single Difference): The model trains on pairs with exactly one visual discrepancy. This provides dense, interpretable feedback to establish basic localization skills.
Stage 2 (Double Difference): Complexity increases to two differences, fostering compositional reasoning.
Stage 3 (Mixed Difference): The model is fine-tuned on pairs with up to four differences, where the count of differences is hidden. This forces the model to infer the number of discrepancies, enhancing robustness and generalization.

3. Key Contributions

DiG Framework: Introduced a novel proxy task that forces MLLMs to move beyond coarse semantic alignment to develop fine-grained perceptual reasoning by localizing all differences between image pairs.
Scalable Data Pipeline: Developed an automated 3D rendering pipeline that generates high-quality, diverse, and perfectly annotated training data, eliminating the bottleneck of manual labeling.
Curriculum Optimization: Proposed a curriculum-based training strategy that mitigates reward sparsity, enabling stable RL optimization for fine-grained perception tasks.
Empirical Validation: Demonstrated that DiG significantly improves performance across diverse benchmarks and transfers effectively to downstream tasks.

4. Experimental Results

The authors evaluated DiG on Qwen3-VL (4B and 8B variants) across three categories of benchmarks:

Visual Perception Benchmarks:
- DiG consistently improved performance on benchmarks like HalBench (reducing hallucinations), HRBench-8K, VSR (Visual Spatial Reasoning), and CV-Bench.
- Example: The 8B model saw a +3.4 improvement on HalBench and +2.1 on V*.
General Multimodal Benchmarks:
- Improvements were observed in holistic reasoning and understanding on MMBench, MM-Vet, MMStar, and ScienceQA.
- Example: The 4B model gained +3.5 on MMStar and +2.3 on ScienceQA, outperforming larger proprietary models in some metrics.
Visual Grounding Benchmarks:
- Significant gains were achieved on RefCOCO, RefCOCO+, and RefCOCOg, which test the ability to locate objects based on referring expressions.
- The 8B model achieved average improvements of 2–3 points across all splits, demonstrating enhanced spatial grounding.

Ablation Studies:

Curriculum: Progressive training (Single $\to$ Double $\to$ Mixed) yielded the best results, confirming that starting simple stabilizes learning.
Rewards: Combining both IoU (spatial) and F1 (detection) rewards was essential; relying on only one led to suboptimal performance.

5. Significance and Conclusion

Bridging Perception and Reasoning: DiG provides a scalable, principled paradigm for bridging high-level reasoning with low-level perceptual sensitivity in MLLMs.
Scalability: The automated data generation pipeline allows for the creation of massive, diverse datasets that are impossible to curate manually, making fine-grained perception training accessible.
Generalization: The skills learned through differential grounding transfer effectively to standard downstream tasks, suggesting that the ability to detect subtle differences is a fundamental capability for robust multimodal intelligence.
Future Direction: The work highlights that differential grounding is a promising direction for developing more perceptually aligned and reasoning-capable multimodal models, moving beyond simple image-text matching to precise spatial understanding.