Digitizing Nepal's Written Heritage: A Comprehensive HTR Pipeline for Old Nepali Manuscripts

1. Problem Statement

The paper addresses the critical challenge of Handwritten Text Recognition (HTR) for Old Nepali, a historically significant but low-resource language.

• Context: The study focuses on manuscripts from the 18th and 19th centuries (primarily from the Nepal German Manuscript Preservation Project), written in the Devanagari script.
• Challenges:
  • Data Scarcity: Only 155 manuscript images (approx. 3,100 text lines) are available for training, making standard deep learning approaches difficult.
  • Script Complexity: Old Nepali features intricate conjunct characters, inconsistent orthography (scribes spelled words differently), and a lack of standardized word spacing (scriptio continua).
  • Document Degradation: Historical documents suffer from ink fading, paper damage, and diverse handwriting styles.
  • Domain Gap: Existing OCR models (like Tesseract or standard TrOCR) are trained on Latin or modern printed scripts and fail on historical, handwritten Devanagari.

2. Methodology

The authors propose a three-stage training pipeline combined with data-centric strategies to bridge the gap between high-resource pretraining and low-resource target data.

A. Three-Stage Training Pipeline

To overcome the lack of labeled handwritten data, the model undergoes a progressive adaptation process:

1. Stage 1 (Synthetic Pretraining): The model is trained on 100,000 synthetic images generated from historical Nepali textbooks using various Devanagari fonts and noise/distortion effects. This teaches the model general Devanagari patterns.
2. Stage 2 (Transfer Learning): The model is fine-tuned on 5,139 lines of printed Devanagari text from the Heidelberg open research data (heiDATA). This bridges the gap between synthetic data and real-world scanned documents.
3. Stage 3 (Final Fine-tuning): The model is finally fine-tuned on the 3,100 lines of actual Old Nepali handwritten manuscripts.

B. Model Architecture

• Encoder: Primarily uses TrOCR (Transformer-based OCR) with a ViT (Vision Transformer) backbone (trocr-base-handwritten and trocr-large-handwritten). A Swin Transformer was also tested but performed worse due to data limitations.
• Decoder: Explores BERT-style and GPT-2 language models. The best-performing configuration pairs a BERT decoder with a Byte-Level BPE tokenizer.
• Tokenization: Compares character-level vs. byte-level tokenization. Byte-level BPE was found to be optimal for handling the complex conjuncts of Devanagari.

C. Data-Centric Enhancements

Given the small dataset, the authors prioritized data quality and diversity over architectural complexity:

• Label Normalization: Unified inconsistent Unicode representations (e.g., different codes for the chandrabindu) to reduce training noise.
• Data Augmentation: Applied 20 types of realistic augmentations (geometric distortions, Gaussian blur, salt-and-pepper noise, elastic warping) to simulate scanning artifacts and handwriting variations. The 8× augmentation intensity yielded the best results.
• Preprocessing: Used Kraken for line-level segmentation. Notably, binarization was found to hurt performance, so grayscale images were retained.

D. Decoding Strategies

The study systematically evaluated deterministic (greedy, beam search, contrastive search) and stochastic (temperature, top-k, top-p) decoding methods. Results showed that decoding strategies had minimal impact on the final Character Error Rate (CER) compared to data quality and model architecture.

3. Key Contributions

1. First End-to-End Pipeline: The first comprehensive HTR system specifically designed for Old Nepali manuscripts.
2. Three-Stage Adaptation: A novel workflow demonstrating how synthetic data and printed text can effectively pretrain models for low-resource historical scripts.
3. Data-Centric Insights: Demonstrated that label normalization and data augmentation contributed more to performance gains than architectural changes (e.g., switching from BERT to GPT-2 decoders).
4. Open Source Release: Released training code, model configurations, and evaluation scripts to foster further research (hosted on GitHub).
5. Web Application: Developed a functional web tool (nepocr.streamlit.app) allowing researchers to upload and transcribe Old Nepali manuscripts.

4. Results

• Performance: The best model achieved a Character Error Rate (CER) of 4.9% (weighted CER of 4.8%).
• Improvement: This represents a 49% reduction in error compared to a fine-tuned baseline TrOCR model (9.6% CER) and significantly outperforms Google Cloud Vision OCR, which failed to recognize conjuncts and diacritics.
• Ablation Studies:
  • Label Cleaning: Reduced CER from 8.9% to 8.4%.
  • Data Augmentation: Reduced CER from 8.4% to 5.6% (at 8× intensity).
  • Model Size: Upgrading from trocr-base to trocr-large further reduced CER to 4.9%.
• Error Analysis:
  • Errors are structured, not random (e.g., ya confused with pa due to visual similarity).
  • Long Sequence Degradation: Performance drops significantly for lines exceeding 120 characters due to a lack of long-sequence training examples.
  • Token Confusion: A small subset of 10 characters accounts for over 55% of all errors.

5. Significance

• Cultural Preservation: Provides a scalable solution for digitizing Nepal's vast manuscript heritage, which is currently at risk of degradation and inaccessibility.
• Low-Resource HTR Framework: Establishes a robust methodology for other low-resource historical scripts, proving that data quality and augmentation strategies are often more critical than model size in data-scarce environments.
• Transfer Learning Validation: Successfully demonstrates that pretraining on synthetic and printed data can effectively transfer to complex, handwritten historical scripts.
• Future Directions: The paper highlights the need for expanded datasets (specifically long sequences) and the potential for post-correction methods leveraging the identified structured error patterns.

In conclusion, this work sets a new state-of-the-art baseline for Old Nepali HTR, shifting the focus from complex model architectures to rigorous data preparation and domain adaptation strategies.