💻 computer science

H-GRPO: Permutation-Invariant Reinforcement Learning for Grounded Visual Reasoning

This paper introduces H-GRPO, a framework that enhances the performance and interpretability of Vision-Language Models by employing Permutation-Invariant Reinforcement Learning to decompose complex queries into atomic sub-questions grounded in specific visual evidence, thereby reducing hallucinations and fostering human-like deductive reasoning.

Original authors: Eric Peh, Debaditya Roy, Basura Fernando

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Eric Peh, Debaditya Roy, Basura Fernando

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but slightly mischievous, art student named "Vision-Language Model" (VLM). When you show this student a picture and ask a question like, "Is the chef cooking?" they often get the right answer. But here's the catch: they might be guessing based on the fact that chefs usually wear white hats, even if the person in the photo is actually a baker wearing a white hat. They are taking a "shortcut" and might even make up details (hallucinations) to justify their guess.

The paper you shared introduces a new training method called H-GRPO to fix this. Think of it as a new way of grading the student's homework that forces them to show their work, step-by-step, with proof.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Black Box" Guess

Currently, these AI models are like "black boxes." You put an image in, and an answer pops out. We don't know how they got there. They might be right, but for the wrong reasons. It's like a student who gets the right math answer but wrote down the wrong formula; they got lucky, but they didn't actually learn.

2. The Solution: The "Detective's Notebook"

The authors propose a new framework where the AI isn't allowed to just give a final answer. Instead, it must act like a detective filling out a structured notebook.

For every question, the AI must break the problem down into tiny steps. For each step, it has to write down three things:

The Question: "What am I looking at right now?" (e.g., "Is that a muffin tray?")
The Answer: "Yes, it is."
The Evidence: A specific box drawn around the muffin tray in the photo to prove it.

This turns the AI's thinking process from a mysterious guess into a transparent chain of facts: I see a muffin tray (evidence here) -> I see a white jacket (evidence here) -> Therefore, this is a chef.

3. The Challenge: Different Paths to the Same Truth

Here is where it gets tricky. Imagine two detectives solving the same crime.

Detective A looks at the shoe first, then the hat, then the gun.
Detective B looks at the gun first, then the hat, then the shoe.

Both detectives found the same clues and reached the same conclusion. If you were a strict teacher, you might say, "Detective A, you looked in the wrong order! You get zero points." That would be unfair.

The paper's innovation, H-GRPO, is like a smart teacher who understands that the order doesn't matter, as long as the clues are there. It uses a mathematical tool (called "Hungarian Matching") to pair up the student's clues with the correct clues, regardless of the order they were written down. It checks: "Did you find the shoe? Yes. Did you find the hat? Yes. Great job, even if you did it in a different order."

4. The Reward System: "Show Me the Proof"

In the old days, the AI only got a "Good Job!" if the final answer was right. Now, with H-GRPO, the AI gets a reward only if:

It followed the notebook format.
It found the right final answer.
Crucially: Every single step in its notebook is backed up by a specific part of the image.

If the AI tries to make up a fact without pointing to a spot in the photo, it gets a low score. This forces the AI to stop guessing and start actually "seeing" the image.

5. The Results: Better at "Seeing," Not Just "Knowing"

The authors tested this on various puzzles involving pictures.

For tasks requiring spatial reasoning (like figuring out where objects are or how they relate to each other), the new method worked wonders. The AI became much better at not getting tricked by shortcuts.
For tasks requiring deep book knowledge (like complex science questions), it helped, but the AI still needed to know the facts first. The method ensures the AI uses the picture correctly, but it can't teach the AI facts it doesn't already know.
Interpretability: Because the AI has to write down its steps and point to the evidence, humans can actually read its "thought process" and verify if it makes sense. It's no longer a black box; it's a transparent one.

Summary

In short, H-GRPO is a training technique that teaches AI models to be honest detectives. Instead of letting them guess the answer and hope for the best, it forces them to:

Break the problem into small pieces.
Point to the exact spot in the photo that proves each piece.
Get rewarded for finding the right clues, even if they find them in a different order than expected.

This makes the AI's reasoning more reliable, less prone to making things up, and much easier for humans to understand and trust.

Technical Summary: H-GRPO for Grounded Visual Reasoning

1. Problem Statement

Vision-Language Models (VLMs) currently face a critical disconnect between high benchmark accuracy and faithful visual reasoning. While models often achieve strong performance, they frequently rely on superficial correlations, ignore relevant visual evidence, or hallucinate intermediate facts. This issue is exacerbated when VLMs are post-trained using Reinforcement Learning (RL); optimizing solely for final-answer correctness can amplify "shortcut" reasoning rather than encouraging evidence-grounded deduction.

Existing RL-based methods (e.g., R1-VL, Vision-R1) primarily supervise textual reasoning or final outcomes without grounding intermediate steps in localized visual evidence. Conversely, grounding methods (e.g., Visual-RFT, ViGoRL) treat localization as auxiliary rather than integral to the reasoning process. Furthermore, current datasets lack large-scale annotations that jointly capture reasoning structure, intermediate answers, and specific spatial evidence. A key challenge in training such models is that valid reasoning paths are not unique; different models may solve the same visual question using different intermediate steps or orders. Standard sequence-level comparison penalizes correct reasoning simply because it deviates from a fixed reference order.

2. Methodology

2.1 Grounded Reasoning Decomposition

The proposed framework forces the model to decompose a global visual query into a sequence of atomic, verifiable sub-tasks. Instead of a monolithic inference, the model generates a structured output $y$ consisting of:

Intermediate Triplets ( $\tau_i$ ): A sequence $\{ \tau_1, \dots, \tau_m \}$ ${τ_{1}, \dots, τ_{m}}$ where each triplet contains:
- $q_i$ : A sub-question.
- $a_i$ : An intermediate answer.
- $b_i$ : A spatial evidence bounding box ( $[x_{min}, y_{min}, x_{max}, y_{max}]$ ) localizing the visual support for the answer.
Reasoning Path ( $r$ ): A natural language synthesis of the triplets.
Final Answer ( $a_{final}$ ): The concise answer to the original question.

2.2 Data Synthesis (GVRS)

To address the lack of training data, the authors constructed the Grounded Visual Reasoning Synthesis (GVRS) dataset.

Pipeline: A high-quality reference set of 40 examples was drafted by GPT-4o and refined via human-in-the-loop verification. Spatial precision was enhanced using SAM3 for box refinement.
Scale: Using an optimized system prompt selected via reward scoring, 10,000 training samples were generated from four sources: Visual7W, Visual-CoT, A-OKVQA, and ERQA.
Validation: A subset was manually validated by experts for logical consistency, semantic correctness, and spatial accuracy.

2.3 H-GRPO: Hungarian Group Relative Policy Optimization

The core innovation is H-GRPO, a permutation-invariant RL framework designed to handle non-unique reasoning paths.

Permutation-Invariant Matching: Instead of comparing generated and reference reasoning chains in a fixed order, H-GRPO formulates the evaluation as a bipartite matching problem between the set of predicted triplets ( $\hat{D}$ ) and reference triplets ( $D^*$ ).
Similarity Metric: A similarity matrix $S$ is constructed where each entry $S_{ij}$ combines:
- Existence and compatibility of bounding boxes.
- Sentence-BERT cosine similarity for sub-questions ( $sim_q$ ) and sub-answers ( $sim_a$ ).
- Intersection over Union (IoU) for spatial overlap.
Hungarian Algorithm: The optimal one-to-one assignment is found using the Hungarian algorithm to maximize the total similarity score ( $S_{HS}$ ).
Reward Function: The total reward $R$ is a composite of:
1. Format Reward ( $R_{format}$ ): Ensures the output follows the triplet structure (sub-questions, answers, reasoning path, final answer).
2. Hungarian Reasoning Reward ( $R_{HS}$ ): A dense process-level reward derived from the matching score, thresholded to suppress weak reasoning.
3. Final Answer Reward ( $R_{answer}$ ): Activated only if the reasoning is sufficiently grounded (i.e., $R_{HS}$ exceeds a threshold).
The formula is: $R = \alpha R_{format} + \beta R_{answer} \cdot R_{HS}$ .

This formulation ensures the model receives strong rewards only when the final answer is correct and derived from semantically aligned, visually grounded intermediate steps, regardless of the order in which those steps are generated.

3. Key Contributions

Grounded Reasoning Decomposition Protocol: A novel representation of reasoning steps as triplets (sub-question, sub-answer, spatial evidence), forcing explicit externalization of visual evidence.
Scalable Data Synthesis Pipeline: A method to generate step-wise grounded reasoning traces without extensive manual annotation, creating the GVRS dataset.
H-GRPO Framework: A permutation-invariant RL algorithm using Hungarian matching to align predicted and reference reasoning steps, enabling flexible reasoning trajectories while enforcing evidence consistency.
Unified Evaluation Protocol: A comprehensive metric suite measuring answer correctness, grounding fidelity, step-wise alignment, and visual evidence dependence.

4. Experimental Results

The method was evaluated on open-source VLM backbones (Qwen2.5-VL-3B and SmolVLM-2.2B) across in-domain and out-of-distribution (OOD) benchmarks.

In-Domain Performance: H-GRPO achieved the best results on A-OKVQA and Visual7W for the SmolVLM-2.2B backbone (73.4% and 77.2%, respectively), outperforming standard SFT and vanilla GRPO. For Qwen2.5-VL-3B, H-GRPO matched or slightly exceeded SFT/GRPO performance, particularly on Visual7W.
Out-of-Distribution Generalization: H-GRPO demonstrated superior generalization on spatial and geometric reasoning tasks. It achieved the highest scores on RealWorldQA, RoboSpatial (70.2%), and MMStar. The improvement on MMMU was moderate, suggesting the method is most effective for visually grounded tasks rather than pure knowledge-intensive reasoning.
Interpretability: Evaluated via LLM-as-a-Judge, H-GRPO reasoning traces scored highest across all dimensions (Relevance, Coherence, Consistency, Clarity, Interpretability), significantly outperforming R1-VL, ViGoRL, and Reason-RFT.
Qualitative Analysis: Examples showed the model successfully grounding category-level, spatial, and action reasoning in specific image regions (e.g., identifying a chef via a white jacket and food counter), reducing the risk of unsupported hallucinations.

5. Significance and Claims

The paper claims that H-GRPO addresses the trade-off between reasoning accuracy and grounding fidelity. By enforcing that every intermediate logical step be anchored to a specific visual region, the framework produces reasoning paths that are:

Verifiable: Humans and reward functions can inspect the visual evidence for each step.
Faithful: The final answer emerges as a logical consequence of verified visual facts rather than statistical guessing.
Flexible: The permutation-invariant reward allows for diverse, valid reasoning orders, avoiding the penalization of correct logic that simply follows a different sequence than the reference.

The authors note that while H-GRPO significantly improves performance on localization-heavy and spatial reasoning tasks, it does not fully compensate for gaps in domain knowledge required for abstract or scientific reasoning (e.g., MMMU). They conclude that future work must extend grounded supervision to symbolic, chart-based, and long-context multimodal reasoning to fully realize the potential of evidence-based visual deduction.