TRIDIS: A Comprehensive Medieval and Early Modern Corpus for Handwritten Text Recognition and Named Entity Recognition

Technical Summary: TRIDIS Corpus and Evaluation Strategy

Problem Statement
Handwritten Text Recognition (HTR) for historical manuscripts faces significant hurdles regarding data consistency and evaluation rigor. Existing datasets often suffer from inconsistent annotation standards (e.g., varying approaches to abbreviation expansion and punctuation), limited scope (focusing on single scripts or regions), and, most critically, the lack of challenging test splits. Conventional random splits frequently result in substantial overlap between training and test data, yielding overly optimistic performance metrics that obscure a model's true ability to generalize to out-of-domain (OOD) conditions. This limitation hinders the robust assessment of HTR models when applied to new, unseen historical documents.

Methodology
The paper introduces TRIDIS (Tria Digita Scribunt), a unified corpus aggregating multiple open-licensed sub-collections of medieval and early modern manuscripts (12th–18th centuries). The methodology encompasses three core components:

1. Corpus Construction and Harmonization:

   • Data Aggregation: TRIDIS combines legacy collections (e.g., Alcar-HOME, Himanis, e-NDP, Bullinger, CODEA) covering Latin, Old French, Middle High German, and Old Spanish.
   • Transcription Standards: The corpus adheres to a semi-diplomatic transcription paradigm. This involves the silent expansion of abbreviations, normalization of allographic variations (e.g., long-s to short-s), modernization of punctuation, and capitalization of named entities. While this facilitates modern NLP pipeline usage, the paper acknowledges it introduces an interpretative layer distinct from raw paleographic reality.
   • Data Model: The resource is structured under a standardized schema and distributed in Apache Parquet format. Each record includes the line image (RGB), UTF-8 transcription, metadata (manuscript ID, language, century, script family), and Named Entity Recognition (NER) annotations where available.

2. Outlier-Based Partitioning Strategy:

   • To address the issue of in-domain bias, the authors propose a novel splitting method. Instead of random partitioning, they compute joint visual-textual embeddings for each line by concatenating outputs from a vision encoder and a text encoder.
   • A corpus centroid is defined as the element-wise median of all embeddings.
   • The test set is constructed by selecting the top 5% of lines with the largest Euclidean distances from this centroid. These "outliers" are stratified to ensure diversity in their "atypicality."
   • This strategy isolates lines characterized by unusual scripts, rare vocabulary, physical degradation, or atypical layouts, creating a more demanding evaluation scenario.

3. Baseline Experiments:

   • Two models were fine-tuned on 90% of the data (with 5% validation and 5% outlier testing): TrOCR (Large) and MiniCPM-Llama3-V 2.5.
   • Performance was measured using Character Error Rate (CER), Word Error Rate (WER), and BERTScore under both random and outlier-based test conditions.

Key Contributions

• TRIDIS Corpus: A comprehensive, multilingual resource containing nearly 200,000 lines and ~2 million tokens, explicitly documenting the provenance, editorial logic, and transcription rules of its components. It supports both HTR and downstream tasks like NER.
• Evaluation Framework: The introduction of an outlier-driven partitioning strategy that moves beyond random splits to simulate the unpredictability of new historical documents.
• Taxonomy of Challenges: A detailed classification of the specific difficulties found in the outlier test set, including:
  • Names: Lines dominated by proper names with heavy abbreviation.
  • Long Lines: Unusually long lines or those with variable geometry.
  • Physical Defects: Issues like faded ink, stains, or transparency.
  • Underrepresented Scripts: Rare script families (e.g., Praegothica, Humanistica) with limited training examples.
  • Complex Contexts: 16th-century notarial hands and marginalia.

Results
Baseline experiments demonstrate a significant performance gap between in-domain (random) and out-of-domain (outlier) evaluations:

• TrOCR: CER increased from 9.1% (random) to 11.3% (outlier); WER increased from 21.3% to 24.9%.
• MiniCPM-Llama3-V 2.5: CER increased from 10.2% to 12.6%; WER increased from 24.2% to 28.0%.
• Semantic similarity (BERTScore) also declined in the outlier setting (e.g., TrOCR dropped from 0.94 to 0.91).

These results confirm that while state-of-the-art models perform well on standard splits, their robustness degrades noticeably when faced with the specific, atypical characteristics of the outlier partition.

Significance and Claims
The paper positions TRIDIS not merely as a data aggregation but as a tool for robust and fair evaluation. The authors claim that the outlier-based strategy provides a more realistic benchmark for heritage-document HTR, revealing performance limits that remain hidden under conventional random evaluation settings.

The significance lies in:

1. Realistic Benchmarking: It offers a framework for heritage institutions to evaluate models against the "domain drift" inherent in newly digitized archival materials.
2. Identifying Gaps: The performance drop highlights specific areas requiring targeted augmentation, normalization strategies, or specialized modeling (e.g., for underrepresented scripts or complex abbreviations).
3. Future-Proofing: By explicitly encoding editorial heterogeneity and providing rich metadata, the corpus supports cross-linguistic analysis, style-shift investigations, and the development of multimodal representations for deeper document understanding.

The authors conclude that TRIDIS serves as both a foundational resource for cross-domain transfer in heritage document processing and an invitation to rethink evaluation standards to better reflect the challenges of historical text recognition.