Visual, OCR-Text, and Hybrid Evidence Retrieval for Document Visual Question Answering: A Controlled Failure Analysis
This study demonstrates that while visual-based page retrieval significantly improves evidence selection and downstream answer quality in Document Visual Question Answering compared to lexical methods, substantial performance gaps persist due to answer generation limitations even when the correct page is provided, and hybrid fusion offers no additional benefit over strong visual retrievers alone.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine trying to answer a question about a complex document, like a medical report or a financial ledger, but you cannot read the words yourself. Instead, you must rely on a computer system to first find the correct page in a massive stack of papers, and then read that page to give you the answer. This is the challenge of document visual question answering. The system faces two distinct hurdles: it must locate the right piece of evidence among thousands of pages, and then it must correctly interpret the text, charts, and tables on that page to form an answer. If the system picks the wrong page, it will fail no matter how smart it is. If it picks the right page but misreads a number or misses a crucial detail, it will also fail. Understanding where the system breaks down is essential for building tools that can truly assist with complex paperwork.
A team of researchers set out to investigate exactly where these failures happen. They built a controlled experiment to separate the task of finding the page from the task of answering the question. They tested different ways for the computer to "see" the documents. One method relied on converting the text on the page into a list of words, essentially treating the document as a simple text file. Another method allowed the computer to look at the actual image of the page, recognizing the layout, the position of tables, and the visual structure without needing to read every single word first. They also tried combining these two approaches, hoping that mixing text-based searching with image-based searching would create a perfect system that never missed a clue.
The researchers ran thousands of tests using a large collection of document questions. They found that the visual approach, where the computer looks at the page image directly, was significantly better at finding the correct page than the text-based methods. When the system used the visual method, it found the right page more than half the time. In contrast, the text-based methods found the right page only about one-third of the time. This difference mattered greatly for the final answer. When the system was given the page found by the visual method, the quality of the answers improved noticeably. However, the researchers discovered that simply finding the right page was not enough to guarantee a correct answer. Even when they forced the system to use the perfect page—the one that humans knew contained the answer—the computer still struggled to generate the right response. In these cases, the system failed to answer correctly more than half the time, often due to difficulties with numbers, formatting, or interpreting complex charts.
The team also tested whether combining the text and visual methods would fix the problem. They created a hybrid system that took the results from both the text search and the image search and tried to merge them into a single, better list. Surprisingly, this combination did not improve the results. Because the visual method was already so much stronger than the text method, the text search often dragged the combined result down, causing the system to miss pages it would have found on its own. The researchers concluded that while looking at the page image is a superior way to find evidence, the biggest remaining hurdle is not finding the page, but understanding it. The computer still needs to get much better at reading the content it finds, especially when that content involves numbers, tables, or specific visual layouts. The study shows that improving the search tool alone will not solve the problem; the system must also become a much more careful and accurate reader.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.