Handwritten German Administrative Forms (HGAF): A Dataset for Handwriting Recognition with Fine-Tuned and Zero-Shot Baselines

In the quiet corners of offices and government buildings across Germany, a vast amount of paperwork still exists in its original, physical form. Despite the digital age, nearly forty percent of companies there still handle half of their administrative tasks on paper, and many others rely on it even more heavily. This creates a persistent bottleneck: handwritten notes on forms must be converted into digital data to be useful. While computers have become remarkably good at reading printed text, they struggle significantly with human handwriting, especially when that writing is scattered across a page in small, isolated boxes rather than flowing in continuous sentences. The challenge is not just recognizing the letters, but understanding which word belongs to which specific field, such as a name, a date, or a signature. For organizations dealing with sensitive personal information, the solution cannot involve sending these documents to a cloud server; the data must stay within the company's own secure walls. This constraint forces a difficult choice between building specialized, custom software or using powerful, general-purpose artificial intelligence that has never seen the specific form before.

A team of researchers at the Karlsruhe University of Applied Sciences set out to solve this puzzle by creating a new benchmark for this exact scenario. They gathered 291 real-world administrative forms, filled out by 95 different people, to create a dataset they named Handwritten German Administrative Forms. To ensure the data could be shared publicly without violating privacy, the contents were synthetic, meaning the names and addresses were fake, but the handwriting was genuine and varied. The researchers designed three distinct versions of the same form, labeled A, B, and C. Layout A was spacious and easy to fill out, Layout B was a compact table that forced people to write smaller, and Layout C had no boxes at all, leaving the writer to decide where to place their answers. Crucially, every writer filled out all three layouts with the exact same information, allowing the team to isolate how the shape and structure of the paper itself affected the computer's ability to read the text.

The study compared two different approaches to digitizing these forms. The first was a "specialist" pipeline, a traditional method where a computer program first cuts out each individual box of handwriting, cleans it up, and then feeds it to a model that has been specifically trained on thousands of similar examples. The second approach used "vision-language models," a type of artificial intelligence that can look at an entire image and understand it like a human does, without needing to be retrained for every new task. The researchers tested these systems on the three layouts to see which method was more accurate and which could handle changes in the form's design without breaking.

The results revealed a surprising shift in the balance of power. On the simple, spacious Layout A, the general artificial intelligence models actually outperformed the highly trained specialist. Two of the vision-language models made fewer errors than the specialist pipeline, even though the specialist had been specifically fine-tuned on that exact layout. The specialist model struggled most with the long, free-text comment fields, where the handwriting was less constrained, while the general models handled these sections with greater ease. When the researchers moved to the more difficult layouts, B and C, the gap widened. The specialist model, which relied on knowing exactly where the boxes were, began to fail significantly when the boxes were small or missing entirely. It could not adapt to the new visual structures, whereas the general models remained robust, continuing to read the text accurately despite the changes in layout.

One of the most critical findings concerned the reliability of the output. While the general models were more accurate on average, the researchers found that no system evaluated could completely replace a human reviewer. The best-performing on-premise setup managed to transcribe a form without a single error in only about 7.7 percent of cases. This means that for the vast majority of forms, a human would still need to check the work. However, the study highlighted a major advantage for the general models in terms of flexibility. To adapt the specialist pipeline to a new form design, engineers would need to retrain the model and rewrite the code that cuts out the text boxes. In contrast, the general models could be adapted to a new layout simply by changing the instructions given to them, without any retraining. This suggests that for organizations dealing with many different types of forms, using a flexible, general-purpose model might be more practical than building a custom system for each one, provided they accept that a human will still need to verify the final results.

The study also touched on the limitations of current technology. While the general models performed well, they sometimes produced strange errors, such as repeating words or failing to recognize when a writer had crossed something out. The specialist model, meanwhile, was brittle; if the computer could not perfectly locate the text box because the writer had written slightly outside the lines, the entire field would be misread. The researchers concluded that while artificial intelligence has made great strides, the dream of fully automatic, error-free digitization of handwritten forms remains out of reach for now. The most effective path forward appears to be a hybrid approach: using powerful, general models to do the heavy lifting of reading the text, while relying on human oversight to catch the inevitable mistakes, all while keeping the sensitive data securely within the organization's own infrastructure.