💻 computer science

TrOCR for Medieval HTR: A Systematic Ablation Study with Cross-Dataset Validation

This paper presents a systematic ablation study on adapting TrOCR for medieval handwritten text recognition, demonstrating that specific layer freezing strategies and the removal of contrast normalization can achieve competitive accuracy on small historical datasets while providing cross-dataset validation and interpretability insights.

Original authors: Sachin Sharma, Federico Simonetta, Michele Flammini

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sachin Sharma, Federico Simonetta, Michele Flammini

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot that learned to read modern handwriting from millions of examples of clean, printed notes. Now, you want to teach this robot to read a 700-year-old medieval manuscript. The problem? The old paper is stained, the ink is faded, the letters are squiggly, and the writer used abbreviations we don't use anymore. It's like asking a robot trained to read a crisp newspaper to suddenly decipher a messy, coffee-stained diary written in a secret code.

This paper is a systematic "tuning guide" for that robot (called TrOCR) to see how we can best teach it to read these ancient texts without breaking it.

Here is the breakdown of their experiment using simple analogies:

1. The Setup: The "Tuning Knobs"

The researchers treated the robot like a complex radio with three main knobs they could turn to see what happened to the sound (the reading accuracy):

Knob A: Image Cleaning (CLAHE): Before feeding the image to the robot, they tried "cleaning" it up—making the dark ink darker and the light paper lighter, like adjusting the contrast on an old TV.
Knob B: Data Augmentation (The "Gym"): They tried to make the robot stronger by showing it "trick" versions of the text during training. They digitally added fake stains, rotated the lines slightly, or blurred them, simulating the messy conditions of real medieval books.
Knob C: Freezing Layers (The "Brain Freeze"): The robot has two main brain parts: an Encoder (the eyes that see the shapes) and a Decoder (the brain that turns shapes into words). "Freezing" means locking parts of the brain so they can't learn anything new, forcing the robot to rely on what it already knows. They tested locking different amounts of the "eyes" and the "brain."

2. The Experiments: Two Different Libraries

They tested these knobs on two very different "libraries" of text:

Library 1 (Cortonese): A specific 13th-century Italian manuscript with very specific, messy handwriting.
Library 2 (READ-16): A public benchmark with different scripts and languages, used to see if their findings work elsewhere or just on the first book.

3. The Surprising Results

The "Cleaning" Knob (Preprocessing):

The Finding: You don't actually need to clean up the images first.
The Analogy: It's like trying to teach someone to recognize a face in a foggy mirror. The researchers found that if you just let the robot train on the raw, foggy image, it learned to see through the fog just fine on its own. Adding extra "cleaning" didn't make it read better; in fact, sometimes it was just extra work.

The "Gym" Knob (Augmentation):

The Finding: It depends on the book.
The Analogy: For the first library (Cortonese), giving the robot "trick" training examples didn't help much. But for the second library (READ-16), the "gym" training made the robot significantly smarter. It's like how a runner might need to train in the rain to prepare for a rainy marathon, but if the race is always sunny, the rain training might be useless.

The "Brain Freeze" Knob (Layer Freezing):

The Finding: This was the most important discovery. The robot's "eyes" (Encoder) and "brain" (Decoder) react very differently to being frozen.
- The Eyes (Encoder): If you freeze even a little bit of the "eyes," the robot gets confused. It needs to re-learn how to see the specific shapes of medieval letters. Freezing too much of the eyes makes the reading accuracy crash.
- The Brain (Decoder): The "brain" is more flexible. You can freeze a good chunk of it (up to half) without hurting the reading much. This is great news because freezing parts of the brain saves computer power and time.
- The Catch: If you try to freeze both the eyes and the brain at the same time, the results get messy. What works for one library might fail for another.

4. The "X-Ray" Vision (Diagnosis)

To understand why the robot made mistakes, the researchers used two special tools:

Grad-CAM: Like a heat map showing exactly which part of the ink stroke the robot was looking at.
Attention Maps: Like a spotlight showing which words the robot was thinking about.

They found that sometimes the robot got the word right but wasn't "confident" about it (the spotlight was scattered). Other times, the robot was confident but wrong. Interestingly, sometimes the "heat map" tool would go blank (a glitch), but the "spotlight" tool still worked. This taught them that you need to use both tools to diagnose errors, not just one.

The Bottom Line

If you are trying to teach a modern AI to read old, messy manuscripts:

Don't obsess over cleaning the images first; the AI can handle the mess.
Be careful with the "Eyes": Let the AI re-learn how to see the old handwriting; don't lock those parts down.
You can lock the "Brain": It's safe to freeze the middle part of the AI's brain to save time, but test it first to make sure it works for your specific book.
One size does not fit all: A strategy that works for one ancient book might fail for another, so always test your settings on your specific data.

The paper concludes that while we can't make the AI perfect, we can make it much more efficient and accurate by knowing exactly which knobs to turn and which to leave alone.

Technical Summary: TrOCR for Medieval HTR

Problem Statement
Fine-tuning transformer-based Handwritten Text Recognition (HTR) models, specifically TrOCR, for medieval manuscripts presents significant challenges. These models are pre-trained on modern text and must adapt to a visual domain characterized by shifting letterforms, pervasive abbreviations, degraded supports (faded ink, bleed-through), and irregular layouts. While recent studies confirm that transformer models can be adapted to historical documents, reported performance gains are often difficult to interpret because preprocessing, data augmentation, and optimization strategies are frequently adjusted simultaneously. Furthermore, it remains unclear how far a single hyperparameter configuration transfers across different scripts, languages, and time periods, particularly when historical collections offer only limited supervision (typically a few thousand labeled lines).

Methodology
The authors conducted a systematic ablation study to isolate the impact of three controllable fine-tuning choices on recognition accuracy:

Preprocessing: Specifically, the application of Contrast Limited Adaptive Histogram Equalization (CLAHE) for contrast normalization.
Data Augmentation: A pipeline combining geometric transformations (rotations, elastic deformations), photometric adjustments (blur, brightness, contrast), and simulated manuscript degradation (speckle noise, shadows, staining).
Parameter-Efficient Fine-Tuning: Layer freezing strategies for the encoder and decoder.

The experiments utilized a 13th-century Italian manuscript (I-CT 91 "Cortonese") containing 2,517 annotated text lines. To ensure robustness and cross-dataset validation, the same experimental grid was replicated on the public READ-16 benchmark. The study employed a fixed full fine-tuning baseline (TrOCR-base) using an AdamW optimizer with a One-Cycle learning rate schedule for 50 epochs.

The ablation design involved:

Comparing four variants under full fine-tuning: (i) CLAHE + augmentation (baseline), (ii) no CLAHE, (iii) no augmentation, and (iv) no CLAHE + no augmentation.
Sweeping the number of frozen layers ( $K \in \{0, 3, 6, 9, 12\}$ ) in either the encoder or decoder while keeping the other module fully trainable.
Testing a combined configuration ( $enc\_3\_dec\_6$ ) to verify if individually non-significant freezing thresholds remain non-significant when applied jointly.
Statistical significance was assessed using Wilcoxon signed-rank tests with Holm correction on line-level Character Error Rate (CER).
A domain-specialized baseline (Kraken-CATMuS) was used for context, and interpretability was analyzed using Grad-CAM and decoder cross-attention maps.

Key Results

Preprocessing and Augmentation: On the Cortonese dataset, neither CLAHE nor the full augmentation pipeline produced a statistically reliable gain in accuracy compared to the baseline. Removing CLAHE yielded a CER of 7.84%, comparable to the domain-specialized Kraken baseline. However, on the READ-16 dataset, augmentation became clearly beneficial, suggesting dataset-dependent regularization effects.
Layer Freezing Asymmetry: A marked asymmetry was observed between the encoder and decoder.
- Encoder: Freezing beyond the first few layers is consistently detrimental. On Cortonese, freezing 6 or more encoder layers caused significant performance degradation.
- Decoder: The decoder is more tolerant of freezing. Freezing up to 6 decoder layers on Cortonese did not significantly harm accuracy compared to full fine-tuning.
Cross-Dataset Transfer: While the encoder-decoder asymmetry persisted on READ-16, the specific freezing thresholds were not universally transferable. The combined configuration ( $enc\_3\_dec\_6$ ), which was non-significant on Cortonese, performed significantly worse than full fine-tuning on READ-16. This indicates that combined freezing strategies require dataset-specific re-validation.
Interpretability: Grad-CAM and cross-attention maps served as diagnostic tools. High token loss correlated with high entropy (diffuse) attention maps, while correct predictions could still contain tokens with elevated loss. The study noted that Grad-CAM can occasionally collapse to zero maps (a failure mode), in which case attention-based diagnostics serve as a fallback.

Contributions and Significance
The paper provides practical, evidence-based recommendations for practitioners fine-tuning TrOCR on low-resource historical datasets:

Preprocessing: CLAHE should be treated as optional rather than a default requirement, as strong optimization can reduce reliance on aggressive image preprocessing.
Augmentation: Augmentation strategies should be validated per dataset rather than assumed to provide universal gains.
Freezing Strategy: Aggressive encoder freezing should be avoided. Moderate decoder freezing can reduce training costs without measurable loss on specific corpora, but combined freezing recipes must be re-validated for each new dataset.
Diagnostic Tools: Grad-CAM and cross-attention maps are valuable for interpreting error patterns and failure modes, particularly when used in conjunction with token-level loss signals.

The authors modestly claim that this work does not aim to outperform all existing HTR systems but rather to provide a controlled ablation study and actionable guidance. The findings suggest that while optimization choices can partially compensate for the lack of specialized pre-training, domain-aware pre-training remains valuable when data is available. The study emphasizes that conclusions regarding freezing thresholds and preprocessing are specific to TrOCR-like encoder-decoder architectures on low-resource historical line images and should not be overgeneralized without further validation across different scripts and periods.