💻 computer science

TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation

This paper presents a systematic ablation study demonstrating how specific fine-tuning strategies, including layer freezing and contrast normalization, impact TrOCR's performance on medieval manuscripts, revealing that freezing specific decoder layers and omitting preprocessing can achieve competitive accuracy while providing cross-dataset validation and error analysis.

Original authors: Sachin Sharma, Michele Flammini, Federico Simonetta

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sachin Sharma, Michele Flammini, Federico Simonetta

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that learned to read modern handwriting from a massive library of today's notes, letters, and forms. Now, you want to teach this robot to read a 700-year-old Italian manuscript written on old, stained parchment with fading ink and strange, curly letters. This is the challenge the paper tackles.

The researchers are asking: How do we tweak this robot's brain so it can read these ancient texts without needing to retrain it from scratch?

Here is a breakdown of their findings using simple analogies:

1. The Robot and the Old Book

The robot they used is called TrOCR. It's like a brilliant student who aced a modern handwriting exam. But medieval manuscripts are a different world entirely. The ink is faded, the paper is stained, and the letters look nothing like what the robot learned in school.

The researchers tried three main ways to help the robot adapt:

Cleaning the image (Preprocessing): Like using a photo editor to brighten a dark, blurry picture.
Simulating damage (Data Augmentation): Like training the robot by showing it pictures that are intentionally smudged, rotated, or faded, so it learns to handle real-world messiness.
Freezing parts of the brain (Layer Freezing): Imagine the robot has two main brains: one that sees the shapes (the Encoder) and one that understands the words (the Decoder). "Freezing" means locking certain parts of the brain so they can't change, forcing the robot to only learn from the new parts.

2. The "Cleaning" Experiment: Do We Need to Polish the Glass?

The Question: Does the robot need the image to be perfectly brightened and contrasted (using a tool called CLAHE) before it starts reading?
The Finding: No, not really.
The Analogy: It's like trying to read a sign in the fog. You might think you need a powerful flashlight to clear the fog first. But the researchers found that if the robot is smart enough, it can figure out the letters even if the picture is a bit dim or messy. In fact, turning on the "brightener" didn't actually help the robot read better in their tests. The robot learned to ignore the noise on its own.

3. The "Brain Freezing" Experiment: What Can We Lock?

This was the most important part of the study. They tried locking different parts of the robot's brain to save computing power and time.

The "Eyes" (Encoder): This part learns what the letters look like.
- Result: You cannot lock the "eyes" very much. If you freeze too many layers of the visual brain, the robot forgets how to recognize the weird, ancient shapes. It's like putting a blindfold on a student trying to learn a new alphabet; they just can't adapt.
The "Language Center" (Decoder): This part learns how to put the letters together into words.
- Result: You can lock a good chunk of this part. The robot's language center is flexible enough that even if you freeze half of it, it still does a great job. It's like telling a student, "You don't need to relearn how to spell; just focus on recognizing these new letters."

The Golden Rule: They found a sweet spot where they could freeze the first few "eye" layers and the first half of the "language" layers. This saved a lot of computing power without hurting the robot's reading accuracy.

4. The "Stress Test": Does It Work on Other Books?

To make sure their rules weren't just a fluke for this one specific book, they tested the same rules on a different dataset (READ-16), which is like giving the robot a different book with different handwriting styles.

The Surprise: What worked for the first book didn't always work perfectly for the second.
The Lesson: You can't just copy-paste your settings from one ancient book to another. The "freezing" strategy needs to be tested on each new collection. However, the general rule held up: Don't freeze the eyes too much; you can freeze the language center more.

5. The "X-Ray Vision" (Diagnosing Mistakes)

Finally, the researchers used special tools (Grad-CAM and Attention Maps) to look inside the robot's brain while it was reading. They wanted to see where the robot was looking when it made a mistake.

The Finding: When the robot got a word wrong, its "gaze" was often scattered and confused, like a student guessing wildly. When it got it right, its gaze was sharp and focused.
The Catch: Sometimes the "gaze" tool would go blank (like a broken flashlight) even when the robot was confused. So, the researchers learned to use two different tools together to get a complete picture of why the robot failed.

Summary for the Everyday Reader

If you want to teach a modern AI to read ancient, messy handwriting:

Don't obsess over cleaning the images first. The AI is smart enough to handle the mess if trained correctly.
Be careful with the "Visual" part of the AI. Let it learn the new shapes freely; don't lock it down.
You can lock down the "Language" part. This saves time and money without hurting performance.
Test your settings on every new book. What works for one manuscript might not work for another.

The paper proves that with the right tweaks, a modern AI can become a very effective translator for history, even without needing a specialized "medieval" education from the start.

Problem Statement
Fine-tuning transformer-based Handwritten Text Recognition (HTR) models, specifically TrOCR, for medieval manuscripts presents significant challenges. These models are pre-trained on modern text and must adapt to a visual domain characterized by shifting letterforms, pervasive abbreviations, degraded supports (faded ink, bleed-through), and irregular layouts. While recent studies confirm that transformer models can be adapted to historical documents, reported gains are often difficult to interpret because preprocessing, data augmentation, and optimization are frequently adjusted simultaneously. Furthermore, it remains unclear how far a single hyperparameter configuration transfers across different scripts, languages, and time periods. Additionally, interpretability in this domain is underexplored; while attention visualizations exist, gradient-based attributions like Grad-CAM are rarely used as quantitative diagnostics for HTR failure modes.

Methodology
The authors conducted a systematic ablation study to isolate the impact of three controllable fine-tuning choices on recognition accuracy:

Preprocessing: Specifically, the use of Contrast Limited Adaptive Histogram Equalization (CLAHE) for contrast normalization.
Data Augmentation: A pipeline combining geometric transformations (rotations, elastic deformations), photometric adjustments (blur, brightness, contrast), and simulated manuscript degradation (speckle noise, shadows, staining).
Parameter-Efficient Fine-Tuning: Layer freezing strategies for the encoder and decoder.

The experiments were performed on two datasets:

I-CT 91 "Cortonese": A 13th-century Italian manuscript containing 2,517 annotated text lines, serving as the primary low-resource test case.
READ-16: A public benchmark used for cross-dataset validation to assess robustness and transferability.

The study utilized a fixed TrOCR-base initialization and a consistent training protocol (AdamW, One-Cycle schedule, 50 epochs) across 13 configurations. These configurations varied the presence of CLAHE and augmentation, and swept the number of frozen layers ( $K \in \{0, 3, 6, 9, 12\}$ ) in the encoder and/or decoder. A domain-specialized baseline (Kraken with the CATMuS Medieval model) was also fine-tuned for comparison.

To diagnose error patterns, the authors employed Grad-CAM gradient attributions and decoder cross-attention maps, analyzing normalized entropy and Gini coefficients to understand model confidence and spatial focus during token prediction.

Key Results

Preprocessing and Augmentation: On the Cortonese dataset, neither CLAHE nor the full augmentation pipeline produced a statistically reliable gain in Character Error Rate (CER) compared to the baseline. Removing CLAHE yielded a CER of 7.84%, comparable to the specialized Kraken-CATMuS baseline (7.9%). However, on the READ-16 dataset, augmentation proved clearly beneficial, suggesting dataset-dependent regularization effects.
Layer Freezing Asymmetry: A marked asymmetry exists between encoder and decoder freezing.
- Encoder: Freezing beyond the first few layers is consistently detrimental. Freezing 6 or more encoder layers significantly increased CER on both datasets.
- Decoder: The decoder is more tolerant; freezing up to 6 layers (50%) did not significantly harm accuracy on Cortonese.
- Combined Strategies: The configuration freezing 3 encoder layers and 6 decoder layers ( $enc\_3\_dec\_6$ ) was non-significant compared to full fine-tuning on Cortonese but resulted in significantly worse performance on READ-16.
Cross-Dataset Validation: Findings regarding encoder freezing thresholds and combined freezing strategies did not transfer universally. While decoder freezing thresholds showed some robustness, combined strategies require dataset-specific re-validation.
Diagnostic Analysis: Grad-CAM and attention maps revealed distinct behaviors. High-error tokens often correlated with high-entropy (diffuse) attention maps. Conversely, correct predictions could still contain tokens with high loss, where Grad-CAM remained localized while attention was diffuse. The study also identified a failure mode where Grad-CAM maps collapsed to zero (due to ReLU or vanishing gradients), necessitating a hybrid diagnostic approach that falls back to attention-based measures in such cases.

Significance and Claims
The paper does not claim to outperform all existing HTR systems. Instead, its significance lies in providing controlled ablations and actionable fine-tuning guidance for practitioners working with small medieval HTR datasets. The authors offer four practical recommendations:

Treat CLAHE as optional rather than a default requirement under modern optimization schedules.
Validate data augmentation strategies per dataset rather than assuming universal gains.
Avoid aggressive encoder freezing, as the visual stack requires adaptation to manuscript-specific features.
Prefer full fine-tuning unless compute constraints justify moderate decoder freezing, noting that combined freezing recipes are not universally transferable.

The study emphasizes that while optimization choices can partially compensate for the lack of specialized pre-training, domain-aware pre-training remains valuable. The work serves as a methodological reference to help future researchers avoid redundant ablation work when adapting TrOCR to similar low-resource historical settings, while cautioning against over-generalizing findings across different scripts and material conditions.