DocAnnot -- Accelerating the Creation of Key Information Extraction Datasets with GenAI-Powered Auto-annotation
DocAnnot is a GenAI-powered framework that accelerates Key Information Extraction dataset creation by combining Large Vision Language Models, OCR, and a novel Spatially Informed Contextual Matching algorithm to generate high-quality auto-annotations that significantly reduce manual effort while enabling effective downstream model training.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you're trying to teach a robot to read a messy, handwritten receipt or a complex insurance form. You want the robot to find the "Total," the "Date," and the "Store Name" and write them down in a neat list. This is called Key Information Extraction (KIE). For a long time, the only way to teach a robot this trick was to have a human sit down and manually point out every single word and number on thousands of documents. It's like hiring a team of people to copy-paste data from a million receipts by hand—slow, expensive, and boring.
Recently, a new kind of super-smart computer brain, called a Large Vision Language Model (LVLM), has arrived. Think of it as a robot that can not only read text but also "see" the picture of the document, understanding how words are arranged on the page. However, these robots are still a bit like over-eager interns: they are great at understanding the story but sometimes get confused about exactly which number belongs to which label, or they might make up facts that aren't there (a phenomenon known as "hallucinating"). The big question scientists are asking is: Can we use these smart robots to do the boring work of labeling data for us, so we can train other robots faster and cheaper, without needing a human to check every single line?
This is exactly what the paper DocAnnot tackles. The researchers built a new system that acts like a super-efficient assembly line for creating training data. Instead of a human drawing boxes around words, their system uses a three-step dance to do it automatically. First, a "Vision Robot" (the LVLM) looks at the document and guesses what the important labels and values are, like finding the "Total" and the number next to it. Second, a classic "Text Scanner" (OCR) reads every single word on the page and draws a precise box around where each word sits.
Here is where the magic happens: the third step, a new algorithm called SICM (Spatially Informed Contextual Matching). Imagine the Vision Robot says, "The total is 100 dollars," but the Text Scanner sees the number "100" appear three times on the page. How does the system know which "100" is the right one? The SICM algorithm acts like a detective using a ruler. It looks at the "Key Text" (the word "Total") that the Vision Robot found, measures the distance to all the candidate "100"s, and picks the one that is physically closest to the word "Total." It combines the robot's "brain" (understanding the meaning) with a "ruler" (understanding the layout) to make the right choice.
The team tested this system on two famous sets of receipt data, called CORD and SROIE. The results were promising but not perfect. When the system tried to label the receipts on its own, it got an accuracy score (called an F1-score) of about 0.679 on the CORD dataset and 0.846 on the SROIE dataset. To put that in perspective, a human expert would score closer to 0.9 or higher, so the robot is good, but not yet perfect.
However, the real surprise came when they used these robot-made labels to train a new, smaller robot to do the job. They asked: "If we train a model only on data that DocAnnot made, can it still learn?" The answer was a cautious "yes." A model trained entirely on the auto-annotated data achieved a score of 0.6765 on the CORD test set. While this is lower than a model trained on perfect human data (which scored 0.9141), it is a respectable performance that suggests you don't need humans to label every single document to get a working system.
The paper also suggests that the "Spatially Informed" part of their system is crucial. When they ran the system without the "ruler" (the SICM algorithm), the accuracy on the CORD dataset dropped significantly, from 0.679 down to 0.571. This proves that simply having a smart robot isn't enough; you need the spatial logic to know which word belongs to which label.
Finally, the researchers tried a "hybrid" approach. They mixed the robot's work with just a tiny bit of human help—specifically, they added 10%, 20%, or 30% of human-labeled data to the robot's data. This small boost pushed the performance up, with the F1-score climbing to 0.7241 with just 30% human help. This suggests that the best future might not be "all robots" or "all humans," but a team where the robot does the heavy lifting and the human just steps in to fix the tricky bits.
In short, DocAnnot doesn't claim to have solved the problem of perfect automation. Instead, it offers a practical, cost-saving tool that can generate "good enough" data to train models, or serve as a powerful assistant that cuts down the time humans spend on boring paperwork. It's a step toward making document processing faster and cheaper, even if we still need a human to double-check the work.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.