A Survey of OCR Evaluation Methods and Metrics and the Invisibility of Historical Documents

This paper critiques current OCR evaluation methods for their overreliance on modern, Western datasets, arguing that this bias renders historical and marginalized documents—particularly Black newspapers—structurally invisible and leads to representational harm by failing to capture their unique typographic and material challenges.

The Gist

Imagine you have a library of ancient, fragile newspapers written in a language that changes slightly every few decades. Now, imagine you hire a super-smart robot to read them aloud and type them out for the world to see.

This paper is about how we test that robot, and why our current tests are failing to catch a very specific, dangerous problem: the robot is great at reading modern, clean documents, but it completely misunderstands and often destroys the meaning of historical Black newspapers.

Here is a breakdown of the paper using simple analogies:

1. The "Perfect Score" Trap

Right now, when we test these reading robots (called OCR systems), we give them a math test. The test asks: "Did you type every single letter correctly?"

If the robot types "The cat sat" instead of "The cat sat," it gets a high score. But imagine the newspaper was written with three columns of text side-by-side, like a puzzle. If the robot reads the first column, then jumps to the third, then the second, it might still type every letter correctly. However, the story makes no sense anymore.

The Analogy: It's like a student who memorizes a recipe perfectly but serves the ingredients in the wrong order. They get an "A" for spelling the words "flour," "eggs," and "sugar," but they serve you a bowl of raw flour because they didn't understand the structure of the recipe.

2. The "Training Diet" Problem

The paper argues that these robots are trained on a very specific diet. They have eaten millions of modern business forms, scientific papers, and clean digital PDFs. They have never been fed a plate of 19th-century Black newspapers.

The Analogy: Imagine you only ever taught a dog to fetch tennis balls. Then, you throw it a heavy, muddy boot and ask it to fetch. The dog isn't "stupid"; it just has no idea what a boot is because it was never part of its training. When the dog tries to fetch the boot, it might chew it up or drop it. Similarly, these AI models try to "fetch" the text from old newspapers, but because they've never seen this "boot" before, they hallucinate (make things up) or scramble the layout.

3. The "Ghost" in the Machine

When these robots try to read old Black newspapers, they face unique challenges:

• The Paper is Rotting: The ink is faded, the paper is stained, and the scans are blurry (like looking through a dirty window).
• The Fonts are Weird: They use old-fashioned, fancy letters (Gothic or Blackletter) that look like alien symbols to modern AI.
• The Layout is Complex: The text is crammed into 7 columns, with ads and headlines weaving in and out.

The Result: The robot doesn't just make a typo. It often erases the history.

• It might read a poem and a news report as one giant, confusing paragraph.
• It might "hallucinate" (invent) words that sound like they fit the time period but are actually fake.
• It might completely ignore the political message of the layout because it's trying to force the text into a straight line.

4. The "Invisibility"

The most important point of the paper is that this isn't just a technical glitch; it's a form of invisibility.

Because the current tests only care about "Did you spell the word 'freedom' correctly?", the system gets a green light. It is declared "State-of-the-Art." But in reality, it has failed to capture the soul of the document.

The Analogy: Imagine a museum curator who is hired to restore a painting. They are tested on how well they can match the colors of the paint. They do a perfect job matching the red and blue. But in the process, they paint over the artist's signature and the background scenery because they didn't think those parts mattered for the "color test." The painting looks colorful, but the story is gone.

5. Why This Matters

The authors say we need to change the rules of the game. We can't just ask, "Is the text accurate?" We need to ask:

• "Did you keep the columns in the right order?"
• "Did you respect the way the editor used space to make a political point?"
• "Did you preserve the history, or did you just clean it up until it looked like a modern blog post?"

The Conclusion:
To truly understand history, especially the history of marginalized communities like the Black Press, we can't just use tools built for modern, corporate documents. We need to feed the robots the right "food" (historical data) and give them a new "test" that values structure and culture just as much as spelling. If we don't, we risk turning our history into a distorted, unreadable mess, even if the robot thinks it did a perfect job.

Technical Summary

1. Problem Statement

The paper identifies a critical disconnect between the rapid advancement of Optical Character Recognition (OCR) and Document Understanding (DU) systems and the evaluation metrics used to assess them.

• The Bias in Evaluation: Current state-of-the-art (SOTA) systems, including Vision-Language Models (VLMs) and Transformers, are evaluated primarily on modern, Western, institutional documents (e.g., scientific papers, corporate forms, clean PDFs).
• The "Invisibility" Gap: These evaluation frameworks fail to capture failure modes specific to historical and marginalized archives, particularly Black historical newspapers.
• Technical Limitations: Standard metrics like Character Error Rate (CER) and Word Error Rate (WER) rely on Levenshtein edit distance. These metrics treat a misread character and a scrambled column layout as equivalent errors. Consequently, a model can achieve a "perfect" CER score while completely destroying the semantic structure (e.g., merging distinct columns, hallucinating text, or ignoring complex Victorian typography) essential to historical documents.
• Consequence: This leads to structural invisibility, where systems trained on modern datasets fail to read historical texts accurately, and evaluation benchmarks do not penalize these structural failures, rendering culturally significant archives computationally inaccessible or distorted.

2. Methodology

The authors employed a Systematic Literature Review following the PRISMA 2020 framework to analyze OCR evaluation practices from 2006 to 2025.

• Data Collection:
  • Searched six databases (ACM, IEEE, arXiv, ACL, Springer, Google Scholar).
  • Initial retrieval: 80 papers; final selection: 78 unique records after deduplication and screening.
  • Inclusion Criteria: Papers presenting quantitative OCR evaluation, analyzing benchmarks, or overviewing evaluation processes.
• Analysis Dimensions:
  • Training Data: Sources, scale, transparency, and inclusion of historical/community archives.
  • Benchmarks: Document types, language coverage, historical inclusion, and degradation testing.
  • Metrics: Types of metrics reported (e.g., CER/WER vs. structural/layout metrics).
• Case Study:
  • To validate findings, the authors tested three distinct OCR architectures on a page from The Weekly Advocate (1837), an early African American newspaper.
  • Models Tested: Tesseract v5 (traditional pipeline), Surya (layout-centric), and olmOCR 2 (SOTA VLM).
  • Analysis: Qualitative failure analysis focusing on geometric collapse, decoding instability, and hallucination, rather than just ground-truth character matching.

3. Key Contributions

1. Systematic Analysis of Evaluation Gaps: The paper provides a comprehensive mapping of how current OCR systems are trained and evaluated, revealing a near-total absence of Black historical newspapers and community-produced archives in training data and benchmarks.
2. Argument for Structural Metrics: The authors argue that character-level accuracy is insufficient for historical documents. They demonstrate that current metrics fail to detect structural failures (column collapse, reading order errors, typographic hallucinations) common in historical layouts.
3. Case Study on Representational Harm: Through the analysis of The Weekly Advocate, the paper illustrates how "successful" models (based on CER) can produce epistemological errors—erasing the rhetorical and political meaning encoded in the original multi-column layouts and typography.
4. Framework for Cultural Evaluation: The paper proposes that evaluation gaps are not merely technical but result from organizational (meso) and institutional (macro) behaviors, including funding decisions and dataset curation practices that prioritize modern, corporate data over marginalized histories.

4. Key Results

• Training Data Homogeneity:
  • Most major models (e.g., LayoutLMv3, DocFormer, DiT) rely heavily on the IIT-CDIP dataset (42 million pages of mid-20th-century US business/legal documents) or synthetic data.
  • Only a few models (e.g., olmOCR 2) include historical data, and even then, it lacks specific curation for community archives like the Black Press.
  • Table 1 Findings: 6 of 13 major systems rely on IIT-CDIP; most report "No Historical Coverage" or require fine-tuning for historical text.
• Benchmark Limitations:
  • Major benchmarks (OCRBench, OmniDocBench, DocVQA) focus on modern layouts, scientific notation, or "born-digital" PDFs.
  • Table 3 Findings: None of the major benchmarks explicitly target underrepresented community archives. KITAB-Bench is an exception for Arabic heritage, but Black press remains largely unrepresented.
• Failure Modes in Case Study:
  • Tesseract v5: Suffered from Geometric Collapse, misinterpreting vertical column rules as text and defaulting to a left-to-right "Z-pattern," merging distinct editorial content.
  • Surya: Experienced Decoding Instability, where high-contrast 19th-century fonts triggered token collapse and generative repetition ("garbage tokens").
  • olmOCR 2 (VLM): Exhibited Corrective Hallucination, overwriting visual evidence with high-probability tokens based on its training, effectively fabricating text ("knowledge bleed") and replacing entities with incorrect but plausible-sounding text.
• Metric Pluralism Deficit: The review found that very few systems report structural metrics (layout preservation, column ordering) alongside CER/WER.

5. Significance and Implications

• Epistemological Impact: The paper argues that OCR evaluation is not neutral; it shapes "what systems are built to see." By ignoring historical layouts, AI systems erase the editorial logic and political identity embedded in Black newspapers.
• Beyond Technical Fixes: The challenge is identified not as a lack of technical capability (specialized systems can work on historical text), but as a resource allocation and values problem. The industry prioritizes modern, scalable datasets over culturally significant, degraded archives.
• Call to Action:
  • Redefine Benchmarks: Evaluation must move beyond string-edit distance to include layout-aware and fidelity-oriented metrics.
  • Inclusive Curation: Black digital archives should be treated as core test datasets, not edge cases.
  • Critical Engineering: Researchers must recognize that "cleaning" noise (e.g., removing marginalia) often destroys historical meaning. Future systems must balance accuracy with the preservation of archival integrity.

In conclusion, the paper asserts that without a fundamental shift in how OCR systems are trained and evaluated—specifically by centering marginalized historical archives—the field risks perpetuating a form of digital erasure, where the histories of Black America remain computationally legible only in a distorted, structurally broken form.