From Pixels to Pairs: A Comprehensive Benchmark of LLM-Driven Key–Value Extraction in Noisy Document Settings
This paper introduces a comprehensive benchmarking framework that evaluates the robustness of large language models in extracting key-value pairs from noisy OCR-generated documents, revealing that clean-text performance is an unreliable proxy for real-world reliability and challenging assumptions about consistent model rankings and the monotonic benefits of few-shot prompting.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world, vast amounts of information are trapped inside paper documents: invoices, medical records, and government forms. To make this data useful for computers, we must first turn the images of these pages into text, a process called optical character recognition. Once the image becomes text, a second layer of intelligence, known as a large language model, reads that text to find specific pieces of information, such as a total amount due or a customer's name. This two-step process is the backbone of many automated systems, yet it relies on a fragile chain. If the first step produces a messy or inaccurate version of the text, the second step must work with that noise, often leading to errors that are hard to trace. For years, researchers have tested these intelligent models using perfect, clean text, assuming that if a model works well on a perfect page, it will work well on a real, scanned one. This assumption has left a gap in our understanding of how these systems actually behave when faced with the blurry, smudged, or distorted reality of the physical world.
A new study by Zahra Anvari at Westcliff University steps into this gap to see what really happens when these digital tools meet imperfect reality. The researcher set up a massive, controlled experiment to test five different types of intelligent language models. These models were asked to pull specific key-value pairs—like matching the label "Date" to the number "12/05/2023"—from three different types of documents: complex forms, simple receipts, and dense invoices. To simulate the real world, the study did not just feed the models perfect text. Instead, it ran the original document images through three different optical character recognition engines, which are the software tools that convert images to text. These engines are known to make mistakes, such as swapping a zero for an 'O', breaking a word in half, or scrambling the order of lines. The study then compared how the models performed when given the perfect, original text versus the messy, machine-generated text.
The results revealed that the common belief about these models was largely incorrect. The study found that a model's ability to perform well on clean, perfect text is not a reliable predictor of how it will handle noisy, scanned text. In fact, the ranking of which model was the "best" changed completely depending on whether the text was clean or noisy. A model that was the top performer on perfect documents often fell behind its competitors when the text was corrupted by optical character recognition errors. This suggests that the skills required to read a perfect page are different from the skills needed to decipher a messy one. Furthermore, the study showed that these models are particularly good at finding the right category of information, such as knowing where the "Total" field is, but they struggle significantly when the actual numbers or letters in that field are slightly wrong. A model might correctly identify that a document contains a total, but if the optical character recognition engine misread a digit, the model often fails to recover the correct value, or worse, it confidently invents a plausible-looking but wrong number to fill the gap.
Another surprising discovery concerned the use of examples to help the models learn. It is a common practice to show a computer model a few examples of the task before asking it to perform the task itself, a technique known as few-shot prompting. The study tested whether giving the models one, two, or three examples would consistently improve their accuracy. The findings showed that this is not a guaranteed fix. Adding more examples did not always lead to better results; sometimes, giving the model more examples actually made its performance worse. The success of this method depended heavily on the specific model being used, the type of document, and even the specific examples chosen to show the model. In some cases, a model performed best with no examples at all, while in others, a single example was enough, and more became a hindrance. This instability means that simply adding more training examples is not a magic solution for making these systems more robust against real-world errors.
The study also highlighted how the structure of the document itself plays a critical role. When the researchers tested the models on forms where the relationship between a question and an answer depends on their physical position on the page, the performance dropped sharply once the text was flattened into a simple list. Without the visual layout to guide them, the models struggled to connect the right pieces of information. This was especially true for dense documents like receipts with many line items, where a small error in reading the order of the text caused a cascade of mistakes, misaligning prices with the wrong items. The research demonstrated that while these intelligent models are powerful, they are not immune to the degradation of the input they receive. They cannot magically reconstruct information that was lost or corrupted before they ever saw it.
Ultimately, this work serves as a rigorous reality check for the field of document automation. It proves that testing these systems only on perfect data gives a false sense of security. The study provides a detailed map of where these systems fail, showing that the path from a scanned image to a usable digital record is fraught with specific, predictable pitfalls. By separating the errors made by the text-reading software from the errors made by the intelligent model, the research offers a clearer way to evaluate and improve these tools. The findings suggest that to build truly reliable systems, we cannot rely on the models alone to fix upstream mistakes. Instead, we must treat the entire process as a connected chain, where the quality of the final result depends on the quality of every link, from the initial scan to the final extraction of data.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.