💬 NLP

Chain of Evidence: Pixel-Level Visual Attribution for Iterative Retrieval-Augmented Generation

The paper introduces Chain of Evidence (CoE), a retriever-agnostic framework that leverages Vision-Language Models to enable pixel-level visual attribution in Iterative Retrieval-Augmented Generation, thereby overcoming the limitations of text-only parsing by directly reasoning over document screenshots to provide precise bounding box citations and preserve essential visual layout information.

Original authors: Peiyang Liu, Ziqiang Cui, Xi Wang, Di Liang, Wei Ye

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Peiyang Liu, Ziqiang Cui, Xi Wang, Di Liang, Wei Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a complex mystery. In the past, you had to read through hundreds of pages of typed reports to find the clues. If the report said, "The suspect was in Paris," you had to trust it. But what if the report was actually a messy flyer with a map, a photo of a train ticket, and a handwritten note? If you just typed out the words from that flyer, you'd lose the map and the photo, and you'd have no idea where on the page the clue was hiding.

This is the problem the authors of this paper are solving with a new system called Chain of Evidence (CoE).

The Problem: The "Blurry Photocopy"

Current AI systems that answer questions (called Retrieval-Augmented Generation, or RAG) usually work like this:

You ask a question.
The AI finds some documents.
It strips away all the pictures, charts, and fancy layouts, turning everything into plain text (like a blurry photocopy).
It gives you an answer and says, "I found this in Document A."

The Catch: If Document A is a slide deck with a graph or a PDF with a complex table, turning it into plain text destroys the logic. It's like trying to understand a recipe by only reading the list of ingredients but ignoring the pictures of how to mix them. Furthermore, when the AI says "Document A," it doesn't tell you which part of that document holds the answer. You have to manually scroll through 50 pages to find the specific sentence. This is the "Verification Bottleneck."

The Solution: The "Pixel-Level Detective"

The authors propose Chain of Evidence (CoE), which changes the game by treating documents as images (screenshots) rather than just text.

Think of CoE as a detective who doesn't just read the report but looks at the original, unedited photo of the document.

No More Stripping: It doesn't convert the document to text first. It looks at the screenshot, keeping all the charts, arrows, and layout intact.
The "Magic Highlighter": Instead of just saying "It's in Document 3," CoE draws a precise box (a bounding box) around the exact spot on the image where the answer lives. It points directly to the specific number in a chart or the specific line in a paragraph.
The Chain: For complex questions that require multiple steps (e.g., "Who directed the movie, and where did they go to school?"), CoE builds a visual chain. It shows you: "Step 1: Look at this box on Document A to find the director. Step 2: Look at this box on Document B to find the school."

How They Tested It

To prove this works, the team built two new "training grounds" (datasets):

Wiki-CoE: A massive collection of Wikipedia pages turned into screenshots. They took questions that require jumping between multiple pages and taught the AI to find the exact visual spot for the answer.
SlideVQA: A collection of presentation slides with messy layouts, arrows, and complex diagrams. This is the "hard mode" test because text-based systems usually fail here.

The Results: Why It Matters

The paper shows that CoE is a game-changer, especially for documents that aren't just plain text:

It's Smarter at "Where": On complex slides, standard AI systems that rely on text often get lost. CoE, by looking at the visual layout, correctly identified the evidence much more often.
It's Trustworthy: Because CoE draws a box around the evidence, you can instantly verify the answer. You don't have to guess; you can see the proof right there on the screen.
It Doesn't Need a Specific Search Engine: CoE works with any search tool that finds documents. It just takes the top 5 results (as images) and figures out the logic.

The Bottom Line

The authors argue that for the AI to be truly trustworthy, especially with complex documents like reports, slides, and charts, it needs to "see" the document, not just "read" a text version of it. Chain of Evidence turns the AI into a visual detective that can point its finger directly at the truth, making the reasoning process transparent and easy for humans to check.

In short: Instead of giving you a vague reference to a book, CoE hands you the book, opens it to the exact page, and draws a circle around the sentence you need.

Technical Summary: Chain of Evidence (CoE) for Iterative Retrieval-Augmented Generation

1. Problem Statement

Iterative Retrieval-Augmented Generation (iRAG) has emerged as a paradigm for answering complex multi-hop questions by progressively retrieving and reasoning over external documents. However, current iRAG systems face three critical bottlenecks when operating on real-world, visually rich documents (e.g., PDFs, slides, web reports):

The Verification Bottleneck: Existing systems provide coarse-grained, text-level citations (e.g., "[Source: Doc-1]"). In multi-hop scenarios, this forces users to manually scan lengthy documents to locate specific supporting sentences, creating a high cognitive load.
Information Loss in Text Conversion: Converting visually rich documents into linear text via OCR or parsing destroys spatial logic and layout cues (e.g., chart trends, diagram flows, slide hierarchies). For many documents, the evidence exists in the visual relationship between elements, not just the text, making text-based citations fundamentally insufficient.
Opaque Reasoning Chains: Users lack visibility into the trajectory of decisions. Current methods do not visualize the "chain of evidence"—how an intermediate piece of evidence guides the selection of the next document in a multi-hop trajectory.

2. Methodology: Chain of Evidence (CoE)

To address these limitations, the authors propose Chain of Evidence (CoE), a retriever-agnostic visual attribution framework that operates directly on document screenshots rather than parsed text.

Core Architecture

Visual-First Paradigm: CoE bypasses brittle text parsing pipelines. It takes a set of candidate document screenshots (from an upstream retriever) and processes them as images ( $d^{vis}_i \in \mathbb{R}^{H \times W \times 3}$ ).
Retriever-Agnostic Design: The framework assumes a generic upstream retriever returning a top- $k$ candidate set. CoE focuses on selecting, ordering, and grounding evidence within these candidates without introducing retriever-specific parameters.
Output Format: Instead of text citations, CoE outputs a structured chain of evidence consisting of:
- Selected candidate image identifiers.
- Precise bounding boxes $[(x_1, y_1), (x_2, y_2)]$ pinpointing evidence regions (text paragraphs, table cells, diagrams).
- Natural language sub-queries or reasoning thoughts for each hop.
Generation Process: The model performs a single autoregressive pass to generate the complete evidence chain, mapping the query and candidate screenshots to an answer and a sequence of $(document, \text{bounding boxes}, \text{reasoning})$ tuples.

Training Strategy

The authors employ a two-phase curriculum learning strategy using the Qwen3-VL-8B-Instruct model:

Phase I (Single-hop): Focuses on establishing visual grounding capabilities by training the model to locate evidence in a single image.
Phase II (Multi-hop): Introduces multi-hop evidence chain generation over top-5 candidate screenshots, warm-starting from the Phase I checkpoint.

Data Augmentation: To ensure robustness, the training incorporates spatial augmentation (cropping, translation), resolution variation (handling different visual densities), and evidence permutation (decoupling presentation order from logical reasoning order).

3. Datasets and Evaluation

To rigorously evaluate CoE across varying levels of visual complexity, the authors introduce a dual-benchmark strategy:

Wiki-CoE: A large-scale dataset derived from 2WikiMultiHopQA, containing 70,418 multi-hop questions. It features high-fidelity screenshots of Wikipedia pages with bounding box annotations for structured layouts (tables, infoboxes).
SlideVQA: A challenging dataset of presentation slides featuring complex diagrams, free-form layouts, and charts where spatial arrangement is semantically crucial.

Evaluation Metrics:

Answer Accuracy (EM): Exact Match for the generated answer.
Evidence Localization Accuracy (Loc-Acc): A joint metric requiring the correct candidate image selection and a bounding box overlap (IoU $\ge$ 0.3) with the ground truth.
Reasoning Chain Accuracy (Chain-Acc): Verifies if the ordered sequence of selected documents matches the gold reasoning path.

4. Key Results

Experiments demonstrate that CoE significantly outperforms text-based baselines and zero-shot Vision-Language Models (VLMs), particularly in scenarios requiring visual layout understanding.

Performance on Wiki-CoE: CoE-8B achieves 82.3% EM, 94.4% Chain-Acc, and 80.4% Loc-Acc. While general VLMs (e.g., GPT-5) achieve competitive answer accuracy, their attribution quality is significantly weaker (e.g., GPT-5 achieves only 31.7% Loc-Acc).
Performance on SlideVQA: The advantage of the visual-first approach is most pronounced here. Text-based baselines fail due to layout information loss. CoE-8B achieves 58.8% EM, 87.5% Chain-Acc, and 61.0% Loc-Acc. In contrast, zero-shot Qwen3-VL-235B drops to 6.8% Loc-Acc, and text-based OCR baselines struggle significantly with diagram-heavy slides.
Impact of Visual Complexity: On slides with flowcharts and data charts, CoE outperforms OCR-based methods by margins of up to 28.1% in accuracy, proving that visual attribution is a necessity, not just an enhancement, for non-linear information structures.
Ablation Studies:
- Curriculum Learning: Removing the single-hop pre-training phase causes significant drops in Loc-Acc (e.g., -7.3 points on Wiki-CoE), confirming that isolated grounding is a prerequisite for multi-hop reasoning.
- Resolution: High-resolution input (1536×1536) is critical; reducing resolution to 512×512 causes a catastrophic drop in performance, especially for fine-grained text in infoboxes.
- Modality: A text-only baseline (using OCR) exhibits a performance gap of ~26 points on Wiki-CoE and ~22 points on SlideVQA compared to the visual model, validating that visual structure is the primary carrier of logical information in these contexts.

5. Significance and Claims

The paper claims that Chain of Evidence (CoE) represents a paradigm shift in iRAG by transitioning from brittle text parsing to robust visual reasoning.

Visual Grounding as a Reasoning Necessity: The authors argue that for complex documents, visual modality is not merely an interpretability feature but a fundamental requirement for reasoning. Spatial logic often supersedes linear text, and discarding it leads to hallucinations and unverifiable answers.
Solving the Verification Bottleneck: By providing pixel-level bounding boxes and visualizing the complete reasoning chain, CoE offers an immediate visual verification mechanism. This allows users to audit why an answer was generated, addressing the opacity of current iRAG systems.
Retriever-Agnostic Interpretability: CoE establishes a solution for pixel-level interpretable iRAG that is compatible with any upstream retriever (lexical, dense, hybrid, or visual), decoupling the reasoning and attribution capabilities from the retrieval mechanism.
Trustworthy AI: The work provides a blueprint for the next generation of verifiable AI systems, particularly for high-stakes domains where source attribution and reasoning transparency are essential.

The authors conclude that their work challenges the prevailing text-centric view of information retrieval, demonstrating that preserving visual semantics is critical for trustworthy and interpretable multi-hop reasoning.