Judge a Book by its Cover: Investigating Multi-Modal LLMs for Multi-Page Handwritten Document Transcription

1. Problem Statement

Handwritten Text Recognition (HTR) remains a significant challenge compared to Optical Character Recognition (OCR) for printed text. While state-of-the-art HTR models exist, they typically require fine-tuning on large, labeled datasets, which is impractical for real-world archival digitization due to the cost and time required for manual transcription.

Existing zero-shot solutions (e.g., Google Vision, Amazon Textract, Transkribus) and Multi-Modal Large Language Models (MLLMs) face specific limitations in multi-page document scenarios:

• Context Loss: Most tools process documents page-by-page, discarding shared context such as handwriting style, semantic continuity, and formatting quirks that exist across pages.
• Mode Limitation: MLLMs are often used either as pure end-to-end transcribers (prone to hallucination and high cost due to image tokenization) or as text-only post-processors (lacking visual context to correct OCR errors).
• Efficiency: Processing every page of a long document with high-resolution images via MLLMs is computationally expensive and token-heavy.

The paper addresses the gap in empirical research regarding zero-shot, multi-page handwritten document transcription, specifically investigating how to leverage MLLMs to correct OCR errors while minimizing cost and prompt complexity.

2. Methodology

The authors propose a suite of methods combining OCR engines and MLLMs, categorized into "Page-by-Page" (PBP) and "All-at-Once" (AAO) processing. The core contribution is the introduction of two novel prompting strategies: OCR+PAGE1 and OCR+PAGEN.

Core Hypothesis

Multi-page documents share significant context (handwriting consistency, semantic content, image artifacts). Therefore, providing an MLLM with the full OCR text of a document alongside a single representative page image should allow the model to learn the mapping between the visual style and the text, enabling it to correct errors on unseen pages without needing to process every image.

Proposed Strategies

1. OCR+PAGE1:
   • Input: The OCR transcription of all pages + the image of the first page.
   • Mechanism: The MLLM uses the first page image to understand the handwriting style and formatting, then applies this understanding to correct the OCR text for the entire document.
2. OCR+PAGEN:
   • Input: The OCR transcription of all pages + the image of a selected page (N).
   • Mechanism: A lightweight upstream LLM analyzes the OCR text to determine which page is most "informative" (e.g., contains the most text, fewest errors, or best representation of the writer's style) and selects that page image for the downstream MLLM. This avoids the pitfalls of the first page being a title page with little text.

Baselines Evaluated

The study compares these strategies against:

• IMAGES (PBP): MLLM end-to-end transcription per page.
• IMAGES-ALL-AT-ONCE (AAO): MLLM processes all page images and text simultaneously.
• OCR (PBP): Raw OCR output (post-processed by a text-only LLM).
• OCR+IMAGES (PBP): MLLM processes the image and OCR text for each page individually.

3. Key Contributions

1. New Benchmark & Dataset:

   • Malvern-Hills: A new dataset of 161 images (70 multi-page documents) derived from 19th/20th-century charity archives. It features archaic language, multiple handwriting styles, and challenging conditions.
   • Synthetic Multi-Page Benchmarks: The authors constructed multi-page versions of existing single-page datasets: IAM (English), Bentham (18th-century English), and CASIA-5 (Chinese).

2. Novel Prompting Strategies:

   • Introduction of OCR+PAGE1 and OCR+PAGEN, which demonstrate that a single image can effectively "teach" an MLLM the context needed to correct a full document's OCR output.

3. Comprehensive Zero-Shot Evaluation:

   • The first extensive evaluation of MLLM prompting strategies for multi-page HTR, comparing end-to-end transcription, OCR-only, and hybrid approaches across different models (GPT-4o, Gemini-2.5-Pro, Gemma-3-27B) and languages.

4. Experimental Results

The experiments were conducted on IAM, Malvern-Hills, and Bentham datasets using Azure Vision as the primary OCR engine and various MLLMs.

• Performance vs. Cost:

  • OCR+PAGE1 and OCR+PAGEN consistently achieved the best Pareto frontier (optimal trade-off between accuracy and cost) across all datasets.
  • On the IAM dataset, OCR+PAGE1 achieved a Character Error Rate (CER) of 1.36% (Gemma-3) and 0.85% (GPT-4o), outperforming raw OCR and often matching or beating full-image methods while using significantly fewer tokens.
  • On the challenging Malvern-Hills dataset, OCR+PAGE1 (Gemini-2.5-Pro) achieved 5.83% CER, outperforming raw OCR (7.47%) and competing closely with the much more expensive OCR+IMAGES method (6.42% CER) at a fraction of the cost.
  • On CASIA-5 (Chinese), OCR+PAGEN achieved 8.94% CER, nearly matching the full-image method (8.39%) at lower cost.

• Semantic Accuracy:

  • Beyond CER, semantic evaluation (classifying errors as minor vs. major/hallucinations) showed that OCR+PAGE1/PAGEN reduced major errors by over 20% compared to the next best method (OCR+IMAGES). This indicates the methods are better at avoiding hallucinations and preserving proper nouns.

• Scaling to Long Documents:

  • Experiments on documents with 5 to 13 pages showed that OCR+PAGE1 remained the top or joint-top performer, even when accessing <20% of the document images.
  • The method proved robust even in ablation studies where handwriting styles varied across pages (IAM-5-Random), though performance gains were slightly reduced, confirming the importance of shared context.

• Open-Source Baselines:

  • Open-source models like TROCR and DOCOWL2 performed poorly in zero-shot settings (CER >30% or >90%), highlighting the superiority of commercial MLLMs for this specific task.

5. Significance and Implications

• Efficiency in Digitization: The paper demonstrates that for multi-page documents, it is unnecessary to feed every page image to an MLLM. A single "cover" page (or a strategically selected page) combined with the full text stream is sufficient to achieve state-of-the-art zero-shot transcription. This drastically reduces token costs and inference time.
• Contextual Correction: The work validates that MLLMs can effectively perform in-context learning to map visual handwriting styles to text, correcting OCR errors that text-only post-processors miss.
• Practical Deployment: The proposed strategies offer a viable, cost-effective solution for archiving historical documents, legal records, and other multi-page handwritten materials without the need for expensive, task-specific fine-tuning.
• Future Directions: The authors suggest that prompt caching could further optimize costs for long documents and that using cheaper MLLMs as the OCR engine itself (replacing traditional OCR) is a promising avenue for future research.

In conclusion, the paper establishes that "judging a book by its cover" (using a single page image to guide the transcription of the whole) is a highly effective strategy for multi-modal document transcription, balancing high accuracy with economic feasibility.