From OCR to Analysis: Tracking Correction Provenance in Digital Humanities Pipelines

This paper proposes a provenance-aware framework for tracking OCR correction lineage in digital humanities pipelines, demonstrating that recording edit details at the span level significantly improves the reproducibility and interpretability of downstream NLP tasks by revealing how textual transformations impact scholarly analysis.

The Gist

Imagine you are a historian trying to read a dusty, damaged letter from 1850. You can't read the handwriting, so you use a robot scanner (called OCR) to turn the image into digital text.

But here's the problem: The robot is bad at reading old, smudged handwriting. It might read the name "Madison" as "Madifon" or "Madi son."

To fix this, researchers usually run the text through a "clean-up" process. They use rules, AI, or human editors to fix the mistakes. The problem is that once they fix it, they often throw away the "before" picture. They overwrite "Madifon" with "Madison" and pretend the mistake never happened.

This paper argues that throwing away the history of the mistake is dangerous. If you don't know why the robot changed "Madifon" to "Madison," you can't be sure if it's actually the right name or just a lucky guess.

Here is the paper's solution, explained with simple analogies:

1. The "Edit History" for Text (Provenance)

Think of a Google Doc. When you type, you can see the "Version History." You can see who changed what, when, and why.

• Current DH (Digital Humanities) practice: It's like taking a Google Doc, fixing the typos, and then deleting the version history. You only see the final, clean text.
• This paper's idea: Keep the version history forever. Every time a word is changed, we attach a tiny digital "tag" that says:
  • What was the original word? ("Madifon")
  • What did we change it to? ("Madison")
  • Who did it? (A computer rule, an AI model, or a human?)
  • How sure are we? (Confidence score: 74%)
  • Did a human check it? (Yes/No)

This is called Provenance. It's like a "food label" for text, telling you exactly where the ingredients came from and how they were processed.

2. The "Filter" Analogy

The researchers tested this idea on a small collection of old historical texts. They created three different versions of the same text to see how it affected a computer program trying to find names (like "George Washington" or "Paris").

• Version A (Raw OCR): The messy, robot-read text. (Full of errors, but honest about what the robot saw).
• Version B (Fully Corrected): The text where every possible fix was applied automatically. (Looks perfect, but might have hidden mistakes).
• Version C (Provenance-Filtered): The "Smart" version. Here, the researchers said: "Only apply the fixes that the computer is very sure about, or that a human has checked."

The Result:

• Version B found the most names, but it also found a lot of fake names (ghosts) because the computer made wild guesses.
• Version C found almost as many real names as Version B, but it drastically reduced the fake ones.

It's like a security guard at a concert.

• Version B lets everyone in, even people who look suspicious, because they might have a ticket. (High coverage, high risk).
• Version C only lets in people with a clear ID or a VIP pass. (Slightly fewer people, but much safer).

3. Why This Matters for History

The paper found that small changes in the text can completely change the story a computer tells.

• If the computer thinks "Madi son" is two people instead of "Madison" (one person), the entire analysis of who was important in that era changes.
• By keeping the "Edit History" (Provenance), researchers can say: "Hey, this computer thinks this is a famous general, but it's only 60% sure, and no human checked it. Let's be careful before we write a history book about him."

The Big Takeaway

In the past, Digital Humanities researchers treated the "cleaned" text as the Truth.
This paper says: The "cleaned" text is just one opinion.

By treating the correction process as a first-class citizen (giving it its own spotlight), researchers can:

1. Audit their work: See exactly where the computer might be lying.
2. Manage uncertainty: Decide how much risk they are willing to take (e.g., "I only want to analyze text where a human has checked the names").
3. Reproduce results: If another researcher wants to check their work, they can see the exact "recipe" of edits used to get the final result.

In short: Don't just give us the final, polished essay. Show us the scratch notes, the red pen marks, and the sticky notes so we know exactly how the story was built. That is the power of Provenance.

Technical Summary

Here is a detailed technical summary of the paper "From OCR to Analysis: Tracking Correction Provenance in Digital Humanities Pipelines" by Haoze Guo and Ziqi Wei.

1. Problem Statement

In Digital Humanities (DH), Optical Character Recognition (OCR) is the foundational step for converting scanned historical documents into analyzable text. However, historical OCR output is often noisy due to degraded materials, non-standard fonts, and complex layouts. Researchers typically apply correction workflows (rule-based, neural network-based, or manual) to improve text quality before performing Natural Language Processing (NLP) tasks like Named Entity Recognition (NER).

The Core Issue: Current workflows often overwrite intermediate decisions. When a correction is applied, the original OCR output and the specific nature of the change (e.g., who made the edit, the confidence score, or whether it was a normalization vs. an error repair) are lost. This "black box" approach obscures the lineage of textual transformations, making it impossible to:

• Audit why downstream NLP results changed.
• Distinguish between restoring fidelity to the source and altering historically meaningful variations (normalization).
• Identify which specific edits introduced instability or uncertainty into the final analysis.

2. Methodology

A. Provenance-Aware Correction Schema

The authors propose a span-level provenance schema designed to be tool-agnostic and interoperable with existing NLP pipelines. Unlike character-level edit traces or token-only records, this schema operates at the "span" level (tokens, phrases, or split/merge operations).

Key Schema Components:

• Base Anchoring: All edits are anchored to the raw OCR text using character offsets (Unicode codepoints) to prevent cascading drift.
• Metadata Fields: Each correction record includes:
  • Document/Page ID and Span Offsets.
  • Original vs. Corrected Text.
  • Edit Type: Substitution, Split, Merge.
  • Correction Source: Rule-based, Model-assisted, or Human.
  • Confidence Score: Derived from model scores, rule certainty, or reviewer status.
  • Review Status: Approved/Unreviewed.
  • Layout Zone: Body, Header, Footnote, etc.
• Serialization: Records can be serialized as JSONL, CSV/Parquet, or stand-off annotations, allowing multiple correction layers to be applied without overwriting the original text.

B. Pilot Study Design

The authors conducted a pilot study on a corpus of historical texts to test the utility of this schema.

• Text Variants: Three versions of the text were generated from the same raw OCR source:
  1. Raw OCR: No corrections.
  2. Fully Corrected: All available corrections applied.
  3. Provenance-Filtered: Only corrections meeting a specific "trust policy" (e.g., confidence $\ge$ 0.70 or human-approved) were applied.
• Downstream Task: A fixed NER pipeline (Transformer-based model fine-tuned on CoNLL-2003) was run on all variants.
• Metrics:
  • Entity Volatility: Entities that appear/disappear or change surface form/boundary across variants.
  • Jaccard Similarity: Overlap of unique entity sets.
  • Attribution: A heuristic linking volatile entities to nearby correction events (using span overlap and a $\pm$50 character window) to determine if specific edits caused instability.
• Secondary Task: Entity Linking (EL) was tested to see if surface form changes affected linking to knowledge bases (e.g., Wikidata).

3. Key Contributions

1. Span-Level Provenance Schema: A formal representation for OCR corrections that records edit lineage, source, confidence, and approval status, enabling traceability from NLP outputs back to specific editorial decisions.
2. Empirical Comparison: A pilot study demonstrating that different correction pathways (Raw vs. Full vs. Filtered) yield significantly different NER outputs, proving that "correction" is not a neutral step but a variable that alters analytical results.
3. Diagnostic Framework: A method for using provenance signals (confidence, edit type, layout zone) to identify "unstable" outputs, prioritize human review, and distinguish between error repair and historical normalization.

4. Key Results

Quantitative Findings

• Impact of Correction: Moving from Raw OCR to Fully Corrected text increased entity mentions (1,184 $\to$ 1,342) and unique entities (512 $\to$ 566), but also introduced high volatility (176 volatile entities).
• Value of Filtering: The Provenance-Filtered variant retained most coverage gains (1,287 mentions) while significantly reducing volatility (121 entities).
• Correlation with Unreviewed Edits: A substantial portion of volatile entities in the "Fully Corrected" set were linked to low-confidence or unreviewed edits.
• Threshold Sensitivity: As the confidence threshold for filtering increased (e.g., from 0.50 to 0.85), entity volatility decreased monotonically, but coverage also declined. This establishes a clear coverage-stability tradeoff that researchers can explicitly manage.

Qualitative & Diagnostic Findings

• Sources of Instability:
  • Boundary Edits (Split/Merge): Despite being infrequent, these had the highest "volatility lift" (3.3x), as entity extraction is highly sensitive to segmentation.
  • Layout Zones: Non-body zones (headers, footnotes) were instability hotspots due to OCR challenges in those regions.
  • Normalization Shifts: Many changes were not error repairs but "normalizations" (e.g., modernizing spellings), which altered the historical record.
• Attribution Success: The heuristic successfully linked volatile entities to specific correction events, allowing analysts to answer why an entity changed (e.g., "This entity changed because a low-confidence model split a token").
• Entity Linking: Provenance filtering reduced the rate of link instability caused by low-confidence corrections, preserving the benefits of correction while minimizing erroneous linking.

5. Significance and Conclusion

The paper argues that provenance should be treated as a first-class analytical layer in Digital Humanities NLP pipelines, rather than mere implementation metadata.

• Reproducibility & Auditability: By preserving the lineage of corrections, researchers can reproduce specific analytical states and audit the impact of editorial decisions.
• Uncertainty-Aware Interpretation: The framework allows scholars to explicitly handle uncertainty. Instead of presenting a single "corrected" text as ground truth, researchers can present a range of interpretations based on different trust policies (e.g., "conservative" vs. "recall-oriented").
• Source Criticism: It enables the distinction between repairing OCR errors (restoring fidelity) and normalizing text (altering historical variation), a critical distinction for historical scholarship.

Future Work: The authors suggest extending this framework to multilingual collections, more complex page layouts, and other downstream tasks like topic modeling or sentiment analysis, while working toward standardized reporting criteria for DH-NLP pipelines.