ViHistScriptBench: Benchmarking Vietnamese Historical Script and Document–Type Classification in Low–Resource Settings

Technical Summary: ViHistScriptBench

Problem Statement
Vietnamese historical archives contain a complex mixture of scripts, including chữ Hán (classical Chinese), chữ Nôm (a logographic script adapting Chinese characters for Vietnamese), early quốc ngữ (Latin-based), and modern quốc ngữ. While digitization is crucial for cultural heritage preservation, Optical Character Recognition (OCR) and document analysis for these materials face significant hurdles. Existing systems are predominantly trained on modern Latin scripts and struggle with the specific challenges of historical Vietnamese documents: complex calligraphy, irregular glyph shapes, vertical layouts, marginal glosses, and various forms of degradation. Furthermore, resources for Han–Nôm OCR are sparse compared to character-level datasets like NomNaOCR, which focus on segmentation rather than page-level classification. There is a lack of standardized benchmarks to evaluate script identification and document-type classification in low-resource settings.

Methodology
The paper introduces ViHistScriptBench, a lightweight benchmark designed to facilitate reproducible research on Vietnamese historical document understanding.

• Dataset Construction: The corpus consists of 500 digitized pages sourced from public archives, including the National Library of Vietnam, the Nom Foundation, and Columbia University Libraries. The dataset covers diverse genres (epic poetry, chronicles, folk narratives) and centuries. Images were downsampled to 300 dpi and stored in PNG format.
• Annotation Scheme: Each page is labeled with two attributes:
  1. Script Class: Hán, Nôm, Mixed (containing both), or Early Quốc Ngữ.
  2. Document Type: Woodblock Print, Handwritten Manuscript, or Modern Print.
     Annotations were performed by graduate students with palaeography backgrounds and verified by a senior linguist.
• Benchmark Tasks: The benchmark defines three primary classification tasks:
  1. Script Identification: Predicting the script class of a full page.
  2. Document-Type Classification: Classifying the material and layout type.
  3. Mixed-Script Detection: A binary task to identify pages containing a mixture of Hán and Nôm.
• Baseline Models: The authors evaluated three categories of models adapted for image classification:
  • Classical CNNs: ResNet-50 and EfficientNet-B0, fine-tuned from ImageNet pre-training.
  • Vision Transformers (ViT): A base ViT model fine-tuned on the dataset.
  • Multimodal Large Language Models (MLLMs): The CLIP image encoder, tested with both supervised fine-tuning and zero-shot prompting.
• Preprocessing: Images were resized to 224×224. Specific preprocessing (contrast stretching, median filtering) was applied to woodblock prints to reduce noise, while color information was retained for manuscripts to preserve brush strokes.

Key Contributions

1. Curated Corpus: A small but diverse collection of 500 annotated Han–Nôm pages with script and document-type labels.
2. Defined Tasks: A formalization of page-level script identification, script mixing detection, and document-type recognition specific to Vietnamese historical contexts.
3. Baseline Evaluations: Comprehensive performance metrics for classical CNNs, Transformers, and MLLMs on these tasks, including accuracy, macro-F1 scores, and confusion matrices.
4. Error Analysis: Identification of specific failure modes, such as confusion between Hán and Nôm due to visual similarity, misinterpretation of vertical layouts, and errors caused by diacritic degradation.
5. Public Release: The data and code are released to support fair comparison and future research.

Experimental Results
The authors report results on a held-out test set (20% of the data). Key findings include:

• Performance: The proposed "Our Pipeline" (combining preprocessing, deep features, and a lightweight classifier) achieved the highest performance, with 90.1% accuracy on Script Identification, 93.4% on Document-Type Classification, and 89.3% on Mixed-Script Detection.
• Model Comparison:
  • ViT-Base performed well among standard architectures (88.5% Script ID accuracy), suggesting an advantage in capturing long-range dependencies and column layouts.
  • EfficientNet-B0 offered a strong balance of accuracy and parameter efficiency (86.7% Script ID accuracy).
  • CLIP showed lower performance in zero-shot settings (68.0% Script ID accuracy) compared to supervised fine-tuning, though it demonstrated some generalization capability.
• Error Patterns: The confusion matrix revealed that the most frequent errors occur between Hán and Nôm classes due to visual similarity. Mixed pages are often misclassified when the minority script appears only in marginal spaces. Early Quốc Ngữ pages are sometimes confused due to degraded printing and diacritic similarities.

Significance and Claims
The paper positions ViHistScriptBench as a first step toward systematic benchmarking for Vietnamese historical script recognition. The authors maintain a modest tone, acknowledging that the dataset is small (500 pages) and reflects the practical reality of data scarcity in cultural heritage. They claim that even modest results provide useful baselines for a field previously lacking standardized evaluation.

The significance of the work lies in:

• Addressing the Gap: Providing the first dedicated benchmark for page-level classification of Han–Nôm documents, distinct from character-level segmentation tasks.
• Highlighting Specific Challenges: Exposing unique error categories tied to diacritics, vertical layouts, and material degradation that are specific to this domain.
• Foundation for Future Work: Establishing a framework for future research into self-supervised pretraining, semi-supervised learning, and community-driven annotation initiatives (similar to Transkribus) to expand the dataset and improve model robustness.

The authors do not claim to have solved the OCR problem entirely but rather to have provided the necessary infrastructure (data, tasks, baselines) to accelerate progress in preserving and unlocking Vietnam's textual heritage.