💬 NLP

VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation

The paper proposes EVisRAG, an evidence-guided framework that mitigates visual hallucinations in multi-image reasoning by explicitly collecting question-relevant visual evidence and employing a novel RS-GRPO algorithm for improved joint optimization of localization and reasoning, achieving significant performance gains on visual question answering benchmarks.

Original authors: Yubo Sun, Chunyi Peng, Yukun Yan, Shi Yu, Zhenghao Liu, Sen Mei, Chi Chen, Maosong Sun

Published 2026-07-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yubo Sun, Chunyi Peng, Yukun Yan, Shi Yu, Zhenghao Liu, Sen Mei, Chi Chen, Maosong Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but instead of a single clue, you are handed a messy stack of three different newspapers, photos, and maps. You have a super-smart detective in your pocket—a computer program called a Vision-Language Model (VLM)—who can read text and look at pictures. This detective is great at answering questions, but sometimes, when faced with a huge pile of images, they get confused. They might start making up details they didn't see, like claiming a photo shows a cat when it's actually a dog, or mixing up facts from two different pages. This is called a "visual hallucination."

To help these detectives, scientists have developed a system called Visual Retrieval-Augmented Generation (VRAG). Think of this as giving the detective a library card. When you ask a question, the system instantly pulls out the most relevant pages from the library and hands them to the detective. The idea is that if the detective has the right evidence right in front of them, they won't need to guess or make things up. However, even with the right pages, the detective often struggles to find the exact sentence or picture they need among the noise, or they might get distracted by irrelevant details and still make up an answer. The big question is: How do we teach the detective to be a true investigator who carefully scans every page, writes down only the facts they actually see, and then solves the mystery without guessing?

This is exactly what the paper "VisRAG2.0" (specifically the EVisRAG framework) sets out to solve. The authors, a team of researchers from universities in China, propose a new way to train these AI detectives to stop hallucinating and start reasoning with real evidence. They found that previous methods were like giving the detective a stack of papers and saying, "Just figure it out!" which often led to confusion. Instead, EVisRAG forces the model to act like a meticulous detective who follows a strict three-step process: Observe, Record, and Reason.

First, the model looks at the retrieved images and explicitly states what it sees, page by page. Second, it creates a "evidence log," writing down specific facts from each image and, crucially, noting when an image has no useful information. Finally, it uses only that written log to solve the problem. To make sure the model actually learns this behavior, the researchers invented a new training method called RS-GRPO (Reward-Scoped Group Relative Policy Optimization). You can think of this as a very strict coach who doesn't just give a grade for the final answer. Instead, the coach gives a separate score for "Did you look at the right picture?" and another score for "Did you write down the fact correctly?" This prevents the model from getting a good grade just for guessing the right answer by luck, ensuring it actually learns to find the evidence first.

The results of this new approach are quite impressive. When tested on various visual question-answering benchmarks (like reading charts, documents, and slides), the EVisRAG model consistently outperformed the standard "backbone" models by an average of about 19% in accuracy. More importantly, it significantly reduced the number of times the model made up facts. In one specific test, a standard model might look at a chart and confidently say the wrong country has a larger population because it hallucinated a number. EVisRAG, however, would look at the chart, write down the correct numbers in its evidence log, and then correctly deduce the answer.

The paper also argues against the idea that simply making the model "think longer" or adding more complex agents to the system is the best solution. Instead, they show that a structured, evidence-guided approach combined with their specific training method (RS-GRPO) is more effective. They demonstrated that by scoping the rewards—giving credit only for the right actions at the right time—the model becomes much more stable and reliable. While the paper doesn't claim this solves every possible problem in AI, the experiments suggest that this method is a major step forward in making visual AI systems more trustworthy, accurate, and less prone to making things up when looking at multiple images.

Technical Summary: VisRAG2.0 (EVisRAG)

Problem Statement

Visual Retrieval-Augmented Generation (VRAG) aims to equip Vision-Language Models (VLMs) with external visual evidence to answer questions that exceed parametric knowledge. However, existing VRAG approaches face significant challenges in multi-image settings:

Visual Hallucinations: VLMs often fail to accurately identify question-relevant evidence across multiple retrieved images, leading to incorrect answers or the fabrication of non-existent visual content.
Lack of Explicit Evidence Collection: Current methods often transplant text-based RAG practices into the visual domain without an explicit cross-image evidence collection process. They rely on external agents or tools, increasing architectural complexity and computational cost.
Training Instability: Jointly optimizing visual perception and reasoning using mixed rewards often leads to unstable training. The lack of clear credit assignment causes interference between signals, where gradients from heterogeneous objectives compete across irrelevant tokens.

Methodology: EVisRAG

The authors propose EVisRAG (Evidence-guided Vision Retrieval-augmented Generation), a framework designed to enable VLMs to perform precise visual perception and reasoning across multiple images.

1. Evidence-Guided Reasoning Process

EVisRAG structures the generation process into four distinct stages, moving beyond standard Chain-of-Thought (CoT) by explicitly separating observation, evidence recording, and reasoning:

Retrieval: Retrieves the top- $k$ document page snapshots relevant to the query.
Visual Perception (Observation): The model sequentially observes the retrieved images, generating a coarse, page-aware observation ( $y_o$ ) to establish an understanding of visual content.
Evidence Recording: Conditioned on the query and observations, the model generates per-image evidence sequences ( $y_e$ ). Crucially, it explicitly records "no relevant information" for images lacking evidence, preventing the model from forcing irrelevant details into the reasoning.
Answer Reasoning: The model performs "detective-style" reasoning ( $y_r$ ) over the aggregated evidence ( $y_p = \{y_o, y_e\}$ ), formulating hypotheses, cross-checking contradictions, and organizing a coherent trajectory before generating the final answer ( $y_a$ ).

2. Reward-Scoped Group Relative Policy Optimization (RS-GRPO)

To train EVisRAG effectively, the authors introduce RS-GRPO, an extension of Group Relative Policy Optimization (GRPO) that addresses the credit assignment problem in multi-stage reasoning.

Reward Scopes: The output sequence is segmented into four scopes: Observation ( $T_o$ ), Evidence Recording ( $T_e$ ), Reasoning ( $T_r$ ), and Answering ( $T_a$ ).
Fine-Grained Rewards: Three specific rewards are applied:
- Format Reward ( $R_{format}$ ): Enforces adherence to the structured workflow (observe $\to$ record $\to$ reason $\to$ answer).
- Perception Reward ( $R_{perception}$ ): Evaluates whether question-relevant regions are correctly localized and summarized in the evidence recording phase.
- Derivation Reward ( $R_{derivation}$ ): Evaluates whether the final answer is correctly derived from the recorded evidence.
Scoped Optimization: Unlike traditional methods that broadcast rewards across the entire sequence, RS-GRPO restricts each reward to its effective token scope (e.g., $R_{perception}$ only supervises $T_o$ and $T_e$ ). This reduces cross-stage interference, sharpens optimization signals, and stabilizes training.

Key Contributions

EVisRAG Framework: A novel architecture that explicitly separates visual evidence collection from reasoning, enabling VLMs to act like "detectives" by gathering and verifying evidence image-by-image before concluding.
RS-GRPO Algorithm: A training strategy that introduces scope-specific rewards to align reward signals with specific token spans. This mitigates the interference between perception and reasoning objectives, leading to more stable and effective joint optimization.
Comprehensive Evaluation: Extensive experiments across five diverse VQA benchmarks (ChartQA, InfoVQA, DocVQA, SlideVQA, ViDoSeek) demonstrating the efficacy of the approach.

Experimental Results

Performance Gains: EVisRAG (using a 7B parameter backbone) consistently outperforms the baseline Qwen2.5-VL-7B by an average of 19% in accuracy and 27% in F1 score. It also surpasses larger 32B models and other state-of-the-art VLRMs and VRAG methods.
Hallucination Reduction: The framework substantially reduces visual hallucinations. In scenarios with incorrect retrieval, EVisRAG shows a controlled increase in abstention (refusing to answer) rather than generating incorrect answers, indicating improved faithfulness to the retrieved content.
Ablation Studies:
- Replacing generic CoT with Evidence-Guided Reasoning yields consistent gains, proving the value of structured evidence localization.
- RS-GRPO outperforms DAPO and StepGRPO, confirming that scoped credit assignment is critical for performance.
Attention Analysis: EVisRAG achieves the highest visual evidence attention ratio (concentration on human-annotated evidence boxes) among compared methods, correlating strongly with higher reasoning accuracy.
Efficiency: Despite the multi-stage generation, EVisRAG maintains competitive inference latency (approx. 100s) and output length, outperforming baselines like R1-Router which incur higher computational costs for similar or lower accuracy.

Significance

The paper claims that explicit evidence collection and scoped reward design are effective mechanisms for improving visual grounding and reasoning reliability in multi-image settings. By decoupling perception from reasoning and applying targeted supervision, EVisRAG addresses the root causes of visual hallucinations in VRAG. The work demonstrates that a 7B model equipped with this framework can exceed the performance of significantly larger models, offering a practical and scalable path toward more reliable, interpretable, and hallucination-resistant visual retrieval-augmented generation systems.