BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval

Technical Summary: BullingerDB

Problem Statement
The field of historical document analysis lacks large-scale, challenging benchmarks that simultaneously address Handwritten Text Recognition (HTR) and Writer Retrieval (WR) with rich temporal metadata. Existing datasets are often limited in scale (e.g., IAM, CVL), lack temporal variation, or do not provide the necessary layout and transcription ground truth for modern deep learning models. Specifically, the correspondence of Heinrich Bullinger (1504–1575), a central figure of the Reformation, represents a massive, multilingual (primarily Latin and Early New High German) corpus with significant stylistic variation over six decades. However, this resource previously lacked transcriptions suitable for training and evaluating modern HTR systems, and its potential for studying long-term handwriting evolution remained underutilized.

Methodology
The authors present BullingerDB, a dataset processing pipeline and benchmark evaluation framework:

• Dataset Construction: The corpus comprises 20,898 pages and 499,222 text lines from 796 distinct writers, spanning 1523–1575.

  • Preprocessing: The pipeline involves layout analysis using Transkribus to segment lines and remove non-text artifacts (marginalia, footnotes).
  • Transcription Refinement: Initial semi-diplomatic transcriptions were refined using a text-to-image alignment method (Peer et al.) to correct hyphenation and character-level inconsistencies, achieving a Character Error Rate (CER) of 6.5% against manual annotations.
  • Filtering: Pages without handwriting or those with insufficient text content (e.g., envelopes with single signatures) were filtered out.
  • Splits:
    • HTR: A split based on writer frequency (frequent vs. non-frequent) to evaluate performance on known styles versus unseen styles.
    • WR: A writer-disjoint train/test split (20:80) designed to minimize temporal distribution differences between sets, enabling leave-one-out validation.

• Evaluation Protocols:

  • HTR: Four state-of-the-art architectures were evaluated: HTRFlor (CNN-BiGRU), PyLaia (CNN-BiLSTM), Retsinas et al. (CNN with shortcut branches), and TrOCR (Transformer encoder-decoder). Metrics included CER and Word Error Rate (WER).
  • WR: The study evaluated CNN-based aggregation (ResNet56), Vision Transformers (ViT), and self-supervised models (AttMask). Features were extracted from SIFT keypoints and aggregated via Sum-Pooling, VLAD, mVLAD, and NetVLAD.
  • Metrics: In addition to standard mAP and Top-n accuracy, the authors introduced temporal nDCG (Normalized Discounted Cumulative Gain). This metric assesses retrieval quality based on chronological coherence, rewarding results where retrieved documents are temporally close to the query.

Key Contributions

1. BullingerDB Dataset: The release of a large-scale, publicly available dataset (via Zenodo) containing 20,898 pages, line segmentations, transcriptions, and metadata (writer identity, date) for 796 writers. It is significantly larger than existing benchmarks like NorhandV3 and IAM.
2. Temporal WR Metric: The introduction of the temporal nDCG metric to evaluate the time-awareness of writer retrieval systems, addressing the unique challenge of stylistic evolution over time.
3. Baseline Results: Comprehensive benchmark results for both HTR and WR tasks, establishing performance baselines for future research on multilingual historical documents.

Results

• Handwritten Text Recognition:

  • TrOCR achieved the best performance with a CER of 9.1% and WER of 30.6% on the total dataset.
  • Performance degraded significantly for non-frequent writers (unseen styles) and shorter text lines (e.g., headers, addresses), where contextual information is limited.
  • Despite the large parameter count of TrOCR (334M), smaller models like PyLaia (4.8M) achieved comparable CERs, though TrOCR excelled in WER.

• Writer Retrieval:

  • AttMask (a self-supervised ViT-based model) outperformed all other architectures, achieving a mAP of 78.3% and a temporal nDCG of 90.3 (using VLAD encoding).
  • Unsupervised (Cl-S) training generally yielded better results than supervised training, likely due to the noise introduced by temporal variation in the labels.
  • While Top-1 accuracy was high (up to 95.6%), the mAP scores were lower than those reported on other datasets (e.g., Historical-WI), indicating that long-term stylistic variation negatively impacts the overall ranking quality.
  • High temporal nDCG scores confirmed that current methods can achieve temporally coherent retrieval, even if the absolute ranking precision is challenged by style evolution.

Significance
The paper positions BullingerDB as a new, challenging benchmark for the historical document analysis community. Its significance lies in its scale, multilingual nature, and the inclusion of precise temporal metadata, which allows for the study of handwriting evolution over a writer's lifetime. The authors note that while current models can handle the dataset, the long-term stylistic variation remains a significant hurdle for robust writer retrieval, particularly for supervised training. The dataset aims to facilitate future research into large-scale HTR training from scratch, writer adaptation, and the disentanglement of writer-specific traits from temporal changes in handwriting.