A Fine-Tuned BERT Classifier for Personal-Letter Titles in Late-Ming and Early-Qing Collected Works

Technical Summary: Lepton – A Fine-Tuned BERT Classifier for Personal-Letter Titles in Late-Ming and Early-Qing Collected Works

Problem Statement
The primary challenge addressed is the automated identification of personal letters (shu 書) within the tables of contents of wenji (collected works) from the late Ming and early Qing dynasties. While the wenji structure provides a curated list of an author's output, distinguishing letters from other genres is difficult for computational methods. Specifically, letters are often confused with prefaces (xu 序), particularly farewell prefaces (songxu 送序).

Traditional rule-based approaches (regular expressions) fail due to the linguistic variability of Classical Chinese titles:

• Terminal Markers: Only 9.2% of letter titles end with the character shu (書); the majority use other markers or none at all. A regex searching for titles ending in shu misses 91% of letters.
• Opening Verbs: While letters often begin with transmission verbs (e.g., yu 與, da 答), these same verbs frequently appear at the start of preface titles, leading to high false-positive rates for regex-based filters.

Methodology
The paper introduces Lepton (Letter Prediction), a binary classifier designed to distinguish between personal letter titles and preface titles.

• Model Architecture: Lepton is a fine-tuned instance of bert-base-chinese (approximately 102M parameters). The model utilizes a BertForSequenceClassification head over the [CLS] token, with all parameters updated during training.
• Backbone Selection: The authors selected a modern-Chinese pre-trained backbone rather than domain-adapted Classical Chinese models (e.g., AnchiBERT, GuwenBERT) available at the time. This decision was based on the observation that the discriminating features (opening verbs and terminal markers) are well-represented in modern pre-training data, allowing the model to learn title-level signals without requiring continued pre-training on historical texts.
• Corpus Construction: The training dataset consists of 5,438 hand-labeled titles from the collected works of 33 literati (mid-Ming to early-Qing).
  • Positive Class: 3,206 letter titles.
  • Negative Class: 2,232 preface titles.
  • Metadata: Each entry includes CBDB (China Biographical Database) metadata such as writer ID, recipient ID, and relationship codes.
• Experimental Setup: The dataset was stratified into an 80/10/10 train/dev/test split. The model was evaluated against several baselines:
  1. Majority class baseline.
  2. Regex baselines (ending in shu, starting with transmission verbs, or a disjunction of both).
  3. TF-IDF character n-grams (ranges 2-2 to 2-4) combined with logistic regression.

Key Results
On a held-out test set, Lepton demonstrated superior performance compared to all baselines:

• Regex Baselines: The naive "ends with shu" regex achieved an F1 score of only 0.166. The "starts with verb" regex performed better (F1 = 0.915) but suffered from lower recall (0.851) compared to Lepton.
• TF-IDF Baselines: These models achieved strong results (F1 ≈ 0.935–0.937) with high recall (0.99) but lower precision (0.89), indicating they captured the opening-verb signal but struggled with some false positives.
• Lepton Performance: The deployed model achieved an F1 score of 0.984 on the matched test subset. Notably, it attained perfect precision (1.000) for the letter class, meaning it produced zero false positives on the test set, with a recall of 0.968.

Error Analysis
The six errors made by Lepton on the test set were all instances of very short, elliptical titles (e.g., recipient names or continuation markers) where the standard opening-verb signal was absent. The model correctly defaulted to classifying these as prefaces based on the lack of the learned "letter" signal, highlighting a limitation in handling out-of-template inputs that rely on editorial convention rather than surface-level genre markers.

Significance and Claims
The paper positions Lepton as a practical tool for digital humanities rather than a theoretical breakthrough in language modeling. Its significance is framed around three main contributions:

1. Scale and Application: The model has been deployed at the China Biographical Database (CBDB) to identify approximately 55,000 letters across mid-Ming to early-Qing wenji, populating the "Ming Letter Platform" to support social-network research.
2. Transfer Learning: It provides a worked example of how a modern-Chinese pre-trained model can effectively transfer to a title-level pre-modern classification task without the need for continued pre-training on historical corpora.
3. Interpretability: The results show that interpretable n-gram baselines sit "within striking distance" of neural models on signal-rich short-text tasks. The convergence of the linear baselines' top-weighted features with the philological analysis (opening verbs and terminal markers) supports the call for machine learning to expose features rather than just scores.

The authors explicitly note the model's boundaries: it is trained on 1550–1700 material, classifies only against the preface genre (not poetry or memorials), and does not extract recipients, dates, or distinguish between private and official correspondence.