VGA-KIE: Reliable Layout-Aware Key Information Extraction from Visually Rich Documents via Evidence-Grounded Verification and Repair
This paper introduces VGA-KIE, a modular post-hoc reliability framework that enhances Key Information Extraction from visually rich documents by coupling evidence-grounded state definition, typed failure diagnosis, and failure-conditioned repair with re-verification, achieving significant improvements in Field-F1 scores, evidence accuracy, and repair success rates across multiple benchmarks.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every day, we are surrounded by documents that look simple to a human but are confusing to a computer. A receipt, a contract, or a medical form is not just a list of words; it is a visual puzzle where the meaning of a number depends entirely on where it sits on the page. Is that "100" a price, a quantity, or a date? To answer that, a machine must understand the layout, the spacing, and the relationship between different parts of the page. This challenge, known as key information extraction, is the backbone of modern automation, allowing businesses to process invoices, banks to read checks, and hospitals to digitize patient records without human intervention. For years, researchers have taught computers to recognize text and guess where information might be, but a persistent problem remains: the computer often gets the right answer for the wrong reason, or it confidently pulls a number from the wrong spot on the page.
A new approach called VGA-KIE, developed by researchers at Henan University of Animal Husbandry and Economy, tackles this problem by changing how a computer decides if an answer is trustworthy. Instead of just asking a model to guess a value, this system forces the computer to prove where it found that value and then double-check its own work. The researchers built a process that acts like a rigorous editor. First, the system gathers a list of possible answers for a specific field, such as the total price on a receipt. Then, it demands that each answer be tied to a specific, visible piece of the document, like a line of text or a table cell. If the computer cannot point to the exact spot where the information came from, it rejects the answer. If the answer passes this test, the system checks if it makes sense against other rules, such as whether the total matches the sum of the items. If a rule is broken, the system does not just give up; it tries a specific fix based on the type of error, finds new evidence, and checks the answer again.
The core of this method is a loop of verification and repair that happens after the initial guess is made. Imagine a computer reading a receipt and suggesting a total of $50. The VGA-KIE system first asks, "Show me the $50 on the page." If the computer cannot link that number to a specific line of text, the system marks it as unsupported. If it can find the text, the system then asks, "Does this number fit the rules?" If the receipt lists three items that add up to $45, but the suggested total is $50, the system flags a conflict. Instead of accepting the wrong number, the system looks for a different candidate, perhaps a line it missed earlier, or it re-examines the text to see if a digit was misread. This process of finding evidence, checking rules, and fixing errors happens in a single, continuous flow. The researchers found that this method significantly improves accuracy. When they tested it on standard datasets containing forms, receipts, and invoices, the system correctly identified the right information in nearly 98 percent of cases, a notable jump from the 93 percent accuracy of the underlying model without this extra checking layer.
What makes this work particularly robust is that it does not just improve the final score; it makes the computer's thinking transparent. The system keeps a record of every step it took, including which parts of the document it looked at, why it rejected a wrong answer, and how it fixed a mistake. In their tests, the system was able to locate the correct evidence for over 92 percent of the fields it accepted. More importantly, it successfully repaired about 70 percent of the answers that were initially wrong, turning errors into correct data. However, the researchers were careful to note that this system is not a magic fix-all. It works best when the initial guesses come from a reliable source, and it is designed to work with specific types of documents. The system also learned to be cautious; if it could not find enough evidence to be sure, it would simply refuse to give an answer rather than guessing and risking an error. This "refusal" happened in only a tiny fraction of cases, but it was a critical feature that prevented the system from confidently stating incorrect information.
The study also revealed that the most important part of this process was the ability to link an answer to a specific location on the page. When the researchers removed this linking step, the system's performance dropped significantly, showing that knowing where an answer came from is just as important as knowing what the answer is. The system also proved that checking for different types of errors—like a number being in the wrong place or a math sum being incorrect—helps the computer choose the right fix. By treating the extraction of information as a cycle of guessing, proving, and correcting, the researchers created a method that is not only more accurate but also more reliable for real-world use. The results suggest that for documents where mistakes can be costly, such as in finance or law, adding this layer of evidence-based verification is a powerful way to ensure that computers are not just reading words, but truly understanding the documents they process.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.