From Handwriting to Structured Data: Benchmarking AI Digitisation of Handwritten Forms

1. Problem Statement

The digitization of paper-based medical archives, particularly handwritten clinical records, remains a critical bottleneck in healthcare infrastructure, especially in low- and middle-income countries (LMICs). Traditional Optical Character Recognition (OCR) and modular computer vision pipelines struggle with:

• Unconstrained Handwriting: Inconsistent stroke widths, overlapping characters, and non-standard symbols.
• Structural Anomalies: Text written outside designated fields, vertical writing, and squeezed information due to space constraints.
• Semantic Variability: Heavy use of medical shorthand (e.g., "NAD" for "No abnormalities detected"), ambiguous abbreviations, and custom encoding conventions (e.g., using "1/2/3" for user status).
• High Stakes: Errors in digitization can lead to data fragmentation, patient safety risks, and delayed clinical feedback.

The authors aim to evaluate whether frontier Multimodal Large Language Models (MLLMs) can overcome these challenges to automate the extraction of structured data from complex, real-world medical forms.

2. Methodology

Dataset

The study utilized data from the UBOMI BUHLE Project, a Pregnancy Exposure Registry in South Africa monitoring maternal medication exposure.

• Real Data: 49 anonymized Maternity Case Records (MCRs) containing handwritten responses, checkboxes, dates, and free text.
• Synthetic Data: 4 forms with simulated data used for initial model screening (to avoid POPIA compliance issues during early testing).
• Field Types: The forms were categorized into three types:
  1. Discrete Selection Fields: Checkboxes and predefined lists.
  2. Formatted Fields: Dates and numerical values requiring specific structures (e.g., DD/MM/YYYY).
  3. Free-Text Fields: Unconstrained clinical notes.

Model Selection

The authors benchmarked 17 models, including:

• Frontier Proprietary Models: Google Gemini (2.5, 3.1 series), OpenAI GPT (4.1, 5 series), Anthropic Claude (Haiku, Sonnet, Opus), and Grok.
• Open-Source Models: Qwen2-VL, Qwen2.5-VL, and Dots.ocr.

Pipeline Architecture

• End-to-End Processing: Unlike traditional OCR, the pipeline uses MLLMs to treat document understanding as a unified sequence-to-sequence problem. Images are encoded into visual tokens and processed directly by the language decoder.
• Prompt Engineering: Two prompt configurations were tested:
  • Optimized Prompt: Included 16 domain-specific normalization rules and a discretized JSON template (e.g., forcing specific options like ["NVD", "C/S"] for delivery type).
  • Ablated (Base) Prompt: Removed discretization and relied on general instructions.
• Evaluation Metrics:
  • Accuracy: Strict string match (global and section-level).
  • Classification: Precision, Recall, and F1-scores (Macro and Weighted) for discrete fields.
  • Free Text: Word Error Rate (WER) and Character Error Rate (CER).
  • Hallucination Rate: Proportion of fabricated outputs not present in the ground truth.

3. Key Contributions

1. Real-World Benchmark: Established a rigorous benchmark using highly challenging, real-world medical forms from South Africa, featuring messy handwriting, structural anomalies, and domain-specific shorthand.
2. Prompt Optimization Analysis: Demonstrated that prompt engineering (specifically schema discretization and normalization rules) significantly impacts performance, particularly for discrete classification tasks.
3. Model Comparison: Provided a comprehensive comparison of frontier MLLMs against open-source alternatives in a high-stakes clinical context.
4. Error Characterization: Identified specific failure modes, such as hallucination in checkbox interpretation and struggles with non-standard medical abbreviations in free text.

4. Results

Overall Performance

• Frontier vs. Open Source: Frontier models (Gemini, GPT, Claude) significantly outperformed open-source models (Qwen, Dots.ocr). Small open-source models (<5B parameters) achieved accuracies as low as 24–34%, while top frontier models reached ~85% overall accuracy.
• Top Performers:
  • Gemini 3.1 Pro: Best overall performance, particularly in discrete selection fields (Weighted F1 = 0.91) and free-text extraction (WER = 0.50, CER = 0.31).
  • GPT 5.4: Excelled in noisy date extraction (100% median accuracy in the Counselling section) and had the lowest hallucination rate (6%).
  • Claude Sonnet 4.6: Best performance on formatted fields (dates and numerical values) with 77% median accuracy (excluding nulls).

Specific Findings

• Discrete Fields: Macro-level metrics were low across all models due to class imbalance (rare subclasses), but Weighted F1 scores were high (>0.90) for top models, indicating strong performance on frequent classes.
• Formatted Fields: Performance on dates and numbers did not strongly correlate with model size or reasoning capability; smaller models performed comparably to larger ones in these tasks.
• Free Text: This remained the most difficult category. High WERs indicated models often omitted words or phrases, particularly when dealing with medical abbreviations (e.g., "P83", "APH").
• Hallucinations: While rates were relatively low (6–14%), they remain a critical concern for medical data where a single fabrication can be dangerous.

Impact of Prompt Engineering

• Discretization: Using a discretized template (forcing specific output options) improved Macro F1 scores by ~50% for discrete fields.
• Trade-offs: While optimization improved macro metrics, it had a minimal impact on weighted metrics (only ~2–5% improvement) because the null class dominates the dataset.
• Ablation: Removing discretization caused smaller models (like GPT-5 Mini) to perform better in some cases because they struggled to interpret the complex schema, reverting to raw visual recognition.

5. Significance and Conclusion

The study concludes that Multimodal Large Language Models have reached a maturity level where they offer a compelling pathway for fully automated digitization of complex handwritten workflows.

• Feasibility: The ~85% accuracy achieved by top models suggests that AI can replace manual data entry for large-scale medical archives, particularly in resource-constrained settings where manual digitization is costly and slow.
• Limitations: Free-text extraction and handling of highly irregular handwriting remain challenging. The "hallucination" risk, though low, requires human-in-the-loop verification for critical medical data.
• Future Outlook: The authors suggest that with further dataset expansion and continued model iteration, these systems can unlock vast amounts of previously inaccessible medical data, enabling better epidemiological research and improved healthcare delivery.

Key Takeaway: While no model is perfect, the combination of frontier MLLMs (specifically Gemini 3.1 Pro and GPT 5.4) with optimized prompting strategies provides a viable, scalable solution for digitizing complex, handwritten medical records.