💬 NLP

When Simpler Is Better: Evaluating Translation Pipelines for Medieval Latin Manuscripts

This paper introduces the Interpres-Parallel-Corpus dataset and demonstrates that for translating medieval Latin manuscripts, a simple pipeline combining specialized OCR with a Vision Language Model outperforms more complex multi-component approaches by avoiding error propagation and prompt saturation.

Original authors: Nguyen Kim Hai Bui, Md. Easin Arafat, Tamás Gábor Orosz, Mufti Mahmud

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nguyen Kim Hai Bui, Md. Easin Arafat, Tamás Gábor Orosz, Mufti Mahmud

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a box of ancient, dusty letters written in a forgotten language on crumbling parchment. Your goal is to turn these images of old paper into a clear, modern English story. This paper is like a detective report on the best way to solve that puzzle.

The researchers found that when dealing with these tricky old documents, the "bigger is better" rule doesn't work, and adding more tools to your toolbox often makes the job harder, not easier.

Here is the breakdown of their findings using simple analogies:

1. The "Specialist vs. The Giant" (The Specialization Gap)

Usually, in the world of AI, we think the biggest, most powerful robot (a massive model with billions of brain cells) will win any challenge. The researchers tested this by asking giant, general-purpose AI robots to read medieval Latin handwriting.

The Result: The giants failed. They got confused by the weird squiggles, shorthand, and ink stains on the paper.
The Surprise: A much smaller, specialized robot (trained only on medieval handwriting) did the job perfectly.
The Analogy: Imagine trying to fix a 19th-century pocket watch. You could hire a brilliant, famous engineer who knows everything about modern cars, planes, and smartphones (the Giant). Or, you could hire a quiet, local watchmaker who has spent 50 years fixing only pocket watches (the Specialist). The paper shows that for this specific job, the local watchmaker is 4.3 times better, even though the famous engineer is "bigger."

2. The "Too Many Cooks" Problem (The Complexity Paradox)

The researchers tried building a "super-pipeline" to translate these documents. They thought: If reading is hard, let's add a spell-checker. If spell-checking isn't enough, let's add a dictionary lookup. If that fails, let's add a history expert.

They built four different versions of this team:

The Solo Artist: Just the Specialist Robot reading the image.
The Editor: The Robot + a Spell-Checker.
The Librarian: The Robot + a Dictionary Lookup.
The Super-Team: The Robot + Spell-Checker + Dictionary Lookup.

The Result: The Solo Artist actually produced the best translations.
The Analogy: Imagine you are trying to translate a secret code.
- The Solo Artist just reads the code and translates it.
- The Super-Team passes the code to a spell-checker, who accidentally changes a word. Then they pass it to a dictionary, which gets confused by the changed word and suggests a wrong definition. Finally, the translator gets a garbled mess and tries to translate that mess.
- The paper found that adding the spell-checker and dictionary created "Prompt Saturation" (the translator got so overwhelmed by extra information it just started copying the dictionary instead of translating) and "Brittleness Propagation" (small errors made by the spell-checker got amplified and ruined the final story).

3. The New Map (The IPC Dataset)

To prove all this, the team couldn't just guess; they needed a map. They created the Interpres Parallel Corpus (IPC).

What it is: A collection of 1,383 specific lines from old manuscripts. For every single line, they have:
1. The picture of the old paper.
2. The exact Latin text written on it.
3. A perfect English translation made by human experts.
Why it matters: Before this, researchers had pictures, or they had translations, but they rarely had all three linked together. This is like having a "Gold Standard" test to grade every AI system fairly.

4. The Two Main Mistakes

The paper explains why the complex teams failed using two specific terms:

Prompt Saturation: Imagine trying to give directions to a driver while shouting 50 different street names at once. The driver gets confused and just repeats the last name they heard. The AI did this: it got so much extra info from the dictionary that it stopped translating and just echoed the dictionary back.
Brittleness Propagation: Imagine a game of "Telephone." If the first person whispers a word slightly wrong, the next person hears it wrong, and by the end, the message is nonsense. The "Spell-Checker" in the complex pipeline made small mistakes, which the "Translator" then tried to fix, making the final result worse than if the Spell-Checker had never been there.

The Bottom Line

If you want to translate ancient, messy manuscripts:

Don't use the biggest, most expensive AI. Use a smaller, specialized one trained specifically on old handwriting.
Keep it simple. Don't add extra steps like spell-checking or dictionary lookups. Let the specialized robot read the image and translate it directly. Adding more steps just introduces more chances for the system to get confused and make mistakes.

The paper concludes that for this specific, difficult task, simpler is truly better.

Technical Summary: Evaluating Translation Pipelines for Medieval Latin Manuscripts

Problem Statement

Medieval Latin manuscripts serve as foundational primary sources for European history, philosophy, and law, yet the vast majority remain untranscribed and untranslated due to a "paleographic barrier." This domain presents a unique stress test for Natural Language Processing (NLP) and Vision Language Models (VLMs) because it combines low-resource transliteration, archaic vocabulary, scribal shorthand, non-standardized ligatures, and noisy input signals from parchment degradation.

Current approaches typically rely on a multi-stage pipeline: Optical Character Recognition (OCR), post-OCR correction, and semantic translation. However, there is a lack of systematic evaluation regarding how these components should be integrated into an end-to-end (E2E) pipeline. Key questions remain unresolved:

Does model specialization outperform massive parameter scale in historical paleography?
Do complex, multi-component pipelines (e.g., those using Retrieval-Augmented Generation or post-OCR correction) consistently outperform simpler baselines on noisy historical data?

Existing benchmarks often fail to capture the full complexity of the task. Some rely on clean, pre-transcribed text (bypassing the OCR challenge), while others lack aligned image-to-translation triplets necessary for granular error analysis.

Methodology

Dataset Construction: The Interpres-Parallel-Corpus (IPC)

To address the lack of aligned data, the authors introduced the Interpres-Parallel-Corpus (IPC), the first tripartite dataset for medieval Latin comprising 1,383 aligned image lines, character-accurate transcriptions, and expert English translations.

Sources: Manuscript line images were sourced from the HTRomance project, focusing on four literary works: Ovid's Metamorphoses, Catullus's Poems, Cicero's Tusculan Disputations, and Quintilian's Institutio Oratoria.
Alignment: These were aligned with ground-truth transcriptions and authoritative translations from the Perseus Digital Library and Project Gutenberg using Gemini 3 Pro for semantic matching, constrained by sliding windows to prevent cross-chapter drift.
Fair Arena Protocol: To ensure statistical integrity across commercial VLMs (which may refuse to process noisy text due to safety guardrails), the authors restricted aggregate metrics to a "Fair Arena" subset of 1,003 samples successfully processed by all pipeline configurations. This subset was verified to maintain a representative difficulty profile regarding character error rates (CER) and linguistic complexity.

Experimental Framework

The study evaluated five pipeline topologies, progressively augmenting the input to a GPT-4o translation backend:

P0 (Baseline): Manuscript image only.
P1 (OCR): Image + raw OCR text.
P2 (+Correction): Image + raw OCR + ByT5-corrected text.
P3 (+RAG): Image + raw OCR + reranked retrieval context from historical dictionaries and parallel texts.
P4 (Full): Image + raw OCR + correction + RAG.

The evaluation compared various OCR engines, including traditional models (Tesseract, PaddleOCR), massive general-purpose VLMs (DeepSeek-OCR, Mistral-OCR-3), and domain-specialized models (TrOCR-Medieval-Base, TRIDIS).

Key Contributions

The Interpres-Parallel-Corpus (IPC): A novel benchmark of 1,383 aligned image-transcription-translation triplets, enabling the first systematic evaluation of the full image-to-translation pipeline for medieval Latin.
Mechanistic Explanation of the Specialization Gap: The paper demonstrates that domain-specific OCR models significantly outperform massive general-purpose VLMs, despite having orders of magnitude fewer parameters. This advantage is attributed to "learned paleographic priors" acquired through training on specialized corpora (e.g., CATMuS Medieval, TRIDIS) rather than sheer scale.
Identification of the Complexity Paradox: The study reveals that adding complexity (correction modules or RAG) does not reliably improve translation quality. In fact, the simplest pipeline (Specialized OCR $\to$ VLM) outperforms all multi-component variants.
Formal Failure Taxonomy: The authors provide the first formal taxonomy for historical translation pipeline failures, characterizing two distinct mechanisms:
- Prompt Saturation: High-density retrieval context distracts the generator, causing "copy-back" behavior where the model echoes source tokens rather than translating.
- Brittleness Propagation: Character-level repetition loops introduced by correction layers interfere with downstream semantic reasoning, degrading translation quality.

Results

The Specialization Gap

Benchmarking on the CATMuS Latin dataset revealed that specialized models achieve a Character Error Rate (CER) up to 4.3× lower than the largest dedicated OCR-focused VLMs (e.g., DeepSeek-OCR).

TrOCR-Medieval-Base achieved a CER of 14.6%, significantly outperforming models with over 3 billion parameters.
Massive general-purpose VLMs frequently failed to interpret scribal shorthand and ligatures, confirming that for high-noise historical domains, specialized exposure is more valuable than model scale.

The Complexity Paradox

In the E2E pipeline evaluation (Fair Arena subset, N=1,003):

P1 (OCR only) achieved the highest mean ChrF score of 26.00.
Adding correction (P2: 25.83), RAG (P3: 25.38), or both (P4: 25.58) yielded no statistically reliable gain over the simple baseline.
While P4 remained competitive on semantic metrics, the aggregate complexity overhead outweighed the benefits.
Noise Sensitivity: All pipeline topologies exhibited nearly identical degradation slopes against OCR noise (approx. -5 ChrF per CER point), indicating that pipeline depth does not mitigate the impact of upstream noise.

Failure Analysis

RAG (P3): Often triggered "echo failures" where the model bypassed semantic translation to directly output retrieved dictionary definitions or source text, particularly when retrieval density was high.
Correction (P2/P4): The ByT5 correction layer introduced repetition artifacts in 0.60% of samples, causing an average penalty of 7.15 ChrF points. These artifacts propagated to the VLM, corrupting the final translation.

Significance and Claims

The paper claims that for medieval manuscript translation, model specialization and pipeline simplicity are paramount.

Specialization over Scale: Current generic VLMs are not yet capable of replacing specialized historical OCR systems. The "specialization gap" is driven by learned priors from paleographic corpora, which general models cannot acquire through scaling alone.
Simplicity over Complexity: Complex multi-component pipelines may suffer from error propagation and prompt saturation that degrade aggregate translation quality. The authors argue that the OCR integration problem is one of pipeline architecture rather than component replacement.
Practical Guidance: The findings offer practical guidance for deploying translation systems in low-resource historical settings, suggesting that a specialized OCR model feeding directly into a VLM is the most robust configuration.

The authors conclude that future progress in historical translation is fundamentally tied to improvements in vision models (OCR) rather than the addition of downstream correction or retrieval layers. They commit to open-sourcing the IPC to foster transparent and reproducible benchmarking in this domain.