💬 NLP

Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels

This paper demonstrates that replacing coordinate-based bounding box outputs with a text-based "quote-and-retrieve" interface significantly reduces attribution hallucinations and improves evidence recall in visual document understanding, while also enabling effective training without costly region-level labels.

Original authors: Zhuchenyang Liu, Yao Zhang, Yu Xiao

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhuchenyang Liu, Yao Zhang, Yu Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are looking at a giant, messy stack of documents—receipts, charts, and handwritten notes. In the world of artificial intelligence, these are called "visual documents." For a computer to be a good detective, it can't just guess the answer; it has to point exactly where it found the clue. This is called "attribution."

For a long time, the standard way to make a computer point to a clue was to ask it to draw a box around the text and give the box's GPS coordinates (like "x=50, y=100"). Think of this like asking a child to point to a specific word on a page by shouting out numbers. It's a clumsy way to play "I Spy." The paper you are about to read investigates why this "coordinate game" is failing so badly. It suggests that the problem isn't that the computer is too dumb to find the clue, but that the game rules (the coordinates) are too confusing. The researchers propose a new rule: instead of shouting numbers, the computer should just read the clue out loud, word-for-word. Then, a smart helper (a retrieval system) finds the exact spot on the page where those words live.

The Great "Pointing" Glitch

The researchers started with a frustrating observation. When they asked powerful AI models to answer questions about long documents and prove their work, the models were great at getting the answer right but terrible at showing where they found it. Even when the answer was perfect, the AI would often point to the wrong paragraph or draw a box in the middle of a blank page. The paper calls this "Attribution Hallucination." It's like a student who gets the math problem right but writes the solution on the back of the teacher's desk instead of in their notebook.

The big question was: Is the AI actually bad at finding the location, or is it just bad at speaking the language of "coordinates"?

The "Read Aloud" Experiment

To test this, the team ran a massive experiment using six different AI models. They kept everything the same—the documents, the questions, and the models—but they changed the rules of the game.

Game A (The Old Way): The AI had to output a set of numbers (coordinates) to draw a box around the evidence.
Game B (The New Way): The AI was told to simply "read the evidence out loud" by quoting the exact text from the document. A separate computer program then took that quote and found the matching piece of the document automatically.

The results were like a magic trick. When the AI was allowed to "read aloud" instead of "shout numbers," its ability to find the right spot skyrocketed.

The Old Way: The AI found the right evidence less than 8% of the time.
The New Way: The AI found the right evidence between 26% and 47% of the time.

That's a huge jump! The rate of "hallucinations" (pointing to the wrong place) was cut roughly in half. The most surprising part? The quality of the actual answers didn't change much. The models were just as smart; they were just much better at showing their work when they didn't have to struggle with coordinates.

Why Coordinates Are the Problem

The paper argues that the failure wasn't a lack of intelligence. It was an "interface artifact." Think of it like asking a chef to describe a recipe. If you force them to describe the ingredients using only a secret code of numbers, they might get the dish right but fail to list the ingredients correctly. But if you let them just say "two cups of flour," the list becomes accurate.

The researchers found that the AI models actually knew where the information was; they just described it in words (like "on page 2, under the table") when forced to use coordinates. By switching to the "quote" method, they let the AI use its natural strength: language.

Teaching the AI to Be a Better Detective

The researchers didn't stop at just changing the rules; they wanted to teach the AI to get even better at this new game without needing a human to grade every single box. They created a special training method called GRPO (Group Relative Policy Optimization).

Imagine a game where the AI tries to answer a question, and a "Judge" (another AI) looks at the answer and the evidence. The Judge doesn't need a human to tell them where the evidence is. Instead, the Judge just checks: "Is the answer right? Do the quoted words actually support the answer?" If the AI quotes the right words, it gets a reward. If it makes things up, it gets a zero.

Using this method, they trained a smaller AI model (an 8-billion parameter backbone) to get much better at finding evidence.

Before training: The model had a "Strict Attribution Accuracy" of 22.4%.
After training: It jumped to 33.8%.

This is a big deal because it means we can teach AI to be more trustworthy and verifiable without needing expensive human labels to draw boxes on thousands of pages.

The Takeaway

This paper suggests a practical path forward for making AI more reliable. It shows that the "coordinate interface" (drawing boxes with numbers) is a weak link that causes AI to hallucinate. By switching to a "language interface" (quoting text and letting a system find the location), we can dramatically improve how well AI points to its evidence.

The findings suggest that the problem isn't that AI can't "see" the document; it's that the way we ask it to point is broken. By letting the AI speak in quotes and letting a smart system do the pointing, we get a much more trustworthy detective. The paper suggests this is a viable, cost-effective way to build better AI for law, medicine, and finance, where proving your source is just as important as getting the answer right.

Technical Summary: Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels

Problem Statement
Reliable visual document understanding (VDU) requires models not only to answer questions but to attribute those answers to specific evidence regions within a document. Current benchmarks and systems, such as CiteVQA, enforce a coordinate interface, where models must output bounding box coordinates ( $x_1, y_1, x_2, y_2$ ) to mark evidence. The authors observe that under this interface, vision-language models (VLMs) frequently suffer from Attribution Hallucination: they generate correct answers but fail to identify the supporting regions, often outputting coordinates that do not overlap with the ground truth. The paper investigates whether this failure stems from a fundamental lack of grounding capability in open models or if it is an artifact of the coordinate output format itself. Furthermore, it addresses the practical challenge that training models to localize evidence typically requires expensive, region-level annotations for long documents, which are difficult to collect at scale.

Methodology
The authors propose a comparative study and a new training paradigm that bypasses the coordinate interface and region-level supervision.

Language Interface vs. Coordinate Interface:
- Coordinate Condition: Models follow the standard CiteVQA protocol, generating bounding box coordinates autoregressively.
- Language Condition: Models output verbatim text quotes of the evidence instead of coordinates. A separate pipeline then resolves these quotes to document regions:
  - A layout parser (MinerU) segments the document into semantic blocks (paragraphs, tables, figures, captions).
  - Each block is rendered as an image crop and embedded using a multimodal encoder.
  - The model's generated quotes are embedded and matched to the best-fitting semantic blocks via a one-to-one assignment (solved via the Hungarian algorithm).
- This setup allows the authors to evaluate the same backbone models under both interfaces while keeping the input and scoring logic constant.
Region-Label-Free Reinforcement Learning (GRPO):
- To improve attribution without costly region labels, the authors use the language interface as a training scaffold.
- Reward Construction: A vision-language judge evaluates the model's output based on the question, the gold answer, and the retrieved image crops of the cited regions (not the raw text quotes). The judge scores:
  - $a$ : Answer correctness.
  - $erel$: Evidence relevance (does the crop support the answer?).
  - $ecov$: Evidence coverage (does the set of crops cover all necessary information?).
- The reward function is defined as $r = \frac{a}{5} \cdot \frac{erel + ecov}{10}$ . This multiplicative structure ensures evidence scores only contribute if the answer is correct, directly targeting attribution hallucination.
- Optimization: The model is trained using Group Relative Policy Optimization (GRPO) with asymmetric clipping (DAPO) to stabilize training. The policy learns to generate better quotes to maximize the judge's reward, effectively learning to point to the right regions without ever seeing region coordinates during training.

Key Contributions

Interface Analysis: A controlled study demonstrating that the poor performance of open VLMs in evidence attribution is largely an interface artifact rather than a capability gap. Switching from coordinates to a quote-and-retrieve language interface significantly recovers attribution ability.
Language Evidence Interface: A novel method that adapts quote-based citation (common in text RAG) to visual documents. It resolves natural language quotes to pixel regions via retrieval, bypassing the need for models to generate precise coordinates.
Region-Label-Free Training: A reinforcement learning recipe that improves strict attributed accuracy using only question-answer pairs and a judge that evaluates retrieved crops. This eliminates the need for expensive, manual region-level annotations.

Results
Evaluated on a verified subset of CiteVQA (719 bilingual questions) across six open VLMs (ranging from 8B to 31B parameters):

Attribution Recovery: Under the coordinate interface, evidence recall (IoU $\ge$ 0.5) was at most 8.1%, with hallucination rates (correct answers with wrong evidence) ranging from 82% to 97%. Switching to the language interface raised recall to 25.9%–46.9% and reduced hallucination rates to 38.6%–65.4%.
Training Gains: Applying the GRPO training recipe to an 8B backbone increased strict attributed accuracy (SAA) from 22.4% to 33.8% and further reduced the hallucination rate to 28.4%.
Answer Quality: Answer correctness remained largely stable across interfaces, indicating that the improvement is specific to attribution, not general reasoning.
Ablation: The recovery was attributed to the information encoded in the model's generated language (the quotes) rather than the retrieval machinery alone. The language interface allowed models to express location information that the coordinate format failed to capture.

Significance and Claims
The paper argues that the "Attribution Hallucination" observed in open VLMs is not an insurmountable lack of grounding capability but a mismatch between the evidence models can express in language and the constraints of the coordinate interface. By adopting a quote-and-retrieve approach, the authors demonstrate a practical path to high-quality evidence attribution in visual documents. Crucially, this path does not require:

A coordinate output interface.
Costly, region-level supervision data.

The findings suggest that open models already possess the ability to identify evidence locations when allowed to express them in natural language, and that reinforcement learning with judge-based rewards on retrieved crops can effectively refine this ability. The authors note limitations, such as the dependency on a layout parser (which sets an upper bound on performance) and the current scope being limited to single-document attribution.