💬 NLP

Named Entity Recognition of Historical Texts via Large Language Model

This study demonstrates that large language models, utilizing zero-shot and few-shot prompting strategies, offer a viable and efficient alternative for Named Entity Recognition in historical texts where annotated training data is scarce, achieving reasonably strong performance despite falling short of fully supervised models.

Original authors: Shibingfeng Zhang, Giovanni Colavizza

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shibingfeng Zhang, Giovanni Colavizza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of old, dusty newspapers and handwritten letters from the 18th and 19th centuries. You want to quickly find every mention of a specific person, city, or organization within them. This is the job of Named Entity Recognition (NER): teaching a computer to spot and label names like "Napoleon," "Paris," or "The Times" in a block of text.

The problem is that these old documents are a nightmare for computers. They are written in strange, archaic languages, have inconsistent spelling (like "shew" instead of "show"), and are full of typos from the scanning process. To teach a computer to handle this, you usually need a human expert to read thousands of pages and manually label every name. This is expensive, slow, and often impossible because there just aren't enough experts or money to do it.

The New Approach: The "Show, Don't Tell" Strategy

This paper tests a new way to solve this problem using Large Language Models (LLMs). Think of an LLM as a super-smart, well-read robot that has read almost everything on the internet. Instead of forcing the robot to go through a grueling training course (which requires those expensive labeled examples), the researchers tried a different approach: Prompting.

They treated the robot like a new employee who hasn't been trained on the specific job yet. They gave the robot a "cheat sheet" (a prompt) and asked it to do the work. They tested two main strategies:

Zero-Shot (The "Blind" Test): They simply told the robot, "Here is a text. Find the people and places." No examples were given.
Few-Shot (The "Show Me" Test): They gave the robot a tiny sample of the work first. They said, "Here is a sentence: 'Napoleon went to Paris.' In this sentence, 'Napoleon' is a Person and 'Paris' is a Location. Now, look at this new sentence and do the same."

The Experiment: The "One Example" Surprise

The researchers tested this on a collection of historical texts in five different languages (German, French, English, Swedish, and Finnish). They tried giving the robot different numbers of examples: 1, 3, or 5. They also tried different ways to pick those examples, such as picking them randomly or picking ones that looked very similar to the text the robot was about to read.

Here is what they found, using some simple analogies:

One Example is the Sweet Spot: Surprisingly, giving the robot just one example worked better than giving it three or five. It's like trying to explain a game to a child: one clear example is often enough. If you give them five examples, they might get confused by the extra noise or the length of the instructions. The robot performed best with a single, clear demonstration.
How You Pick the Example Doesn't Matter Much: The researchers wondered if picking an example that was exactly like the target text (like finding a twin) would help more than picking a random one. They found that it didn't really matter. Whether the example was a perfect match or just a random sample, the robot learned just as well from a single example. It's as if the robot is so smart that it can figure out the rules of the game from almost any single example, regardless of how similar it is to the task.
The "Group Vote" Trick: Since AI can sometimes be a bit unpredictable (like flipping a coin), the researchers asked the robot to do the task three times and then took the answer that appeared most often (Majority Voting). This helped slightly, like a committee agreeing on a decision, but the improvement was small.

The Results: Good, But Not Perfect

The robot did a decent job. It was much better at finding names when given even one example compared to when it was given no examples at all. However, it still couldn't beat the "gold standard" systems that had been trained for months by humans on massive amounts of labeled data.

The Exception: There was one dataset (a collection of German texts called "Sonar") where the robot actually did better than the previous best methods.
The Gap: For most other texts, the robot was still behind the experts who had done the heavy lifting of manual training.

The Bottom Line

This paper proves that you don't always need a massive team of human labelers to get a computer to read old history books. You can use a smart AI and just show it one example to get surprisingly good results.

While this "one-shot" method isn't perfect yet and can't replace the most accurate, human-trained systems, it is a powerful, free, and fast tool. It allows researchers to start exploring historical archives immediately, even if they don't have the budget or time to create a massive training dataset. It's like having a helpful assistant who can read the old books with you, even if they need a quick reminder of what to look for before they start.

1. Problem Statement

Context: Named Entity Recognition (NER) is critical for extracting structured information from unstructured historical texts (e.g., newspapers, manuscripts) to aid digital humanities research.
Challenges:

Data Scarcity: High-quality annotated datasets for historical texts are rare due to the high cost and expertise required for manual labeling.
Linguistic Variability: Historical texts contain archaic vocabulary, inconsistent spelling, limited punctuation, and noise from Optical Character Recognition (OCR) digitization.
Limitations of Traditional Methods: Standard supervised machine learning and deep learning approaches require large amounts of domain-specific annotated training data, which is often unavailable for historical corpora.
Goal: To investigate whether Large Language Models (LLMs), specifically using zero-shot and few-shot prompting strategies, can effectively perform NER on historical texts without the need for task-specific fine-tuning or extensive annotated training data.

2. Methodology

The study utilizes the HIPE-2022 dataset, a multilingual corpus (German, French, English, Swedish, Finnish) covering the 18th to 20th centuries, focusing on the coarse NER task (identifying Person, Location, Work, etc.).

Experimental Setup:

Model: DeepSeek-V3-0324 was selected for its strong multilingual performance and cost-efficiency.
Temperature: Set to 0 to ensure determinism, though experiments were repeated three times to account for model stochasticity.
Approaches Evaluated:
1. Zero-Shot Baseline: The LLM is provided with instructions on entity tags and output format but no examples.
2. Few-Shot Prompting: The LLM is provided with $k$ $k$ annotated examples ( $k \in \{1, 3, 5\}$ $k \in {1, 3, 5}$ ) retrieved from the training/dev sets.
  - Retrieval Strategies:
    - Random Selection: Baseline for few-shot.
    - Lexical Overlap: Uses TF-IDF weighted token overlap to find surface-level similar documents.
    - Embedding Similarity: Uses distiluse-base-multilingual-cased-v2 to find semantically similar documents via cosine similarity.
3. Post-Processing: LLM outputs (Python lists of tuples) are converted to the IOB format required by the HIPE-scorer.
4. Ensemble Method: Majority Voting is applied across the three experimental runs to aggregate predictions and reduce variance.

Evaluation Metrics:

HIPE-scorer: Official evaluation tool used.
Settings: Both Strict (exact boundary and label match) and Fuzzy (overlap with correct label) F1 scores (Micro).
Statistical Analysis: Mean and confidence intervals over three runs; Wilcoxon tests with Bonferroni correction for significance.

3. Key Contributions

Empirical Evaluation: A comprehensive assessment of LLMs for historical NER across five languages and multiple genres using a zero/few-shot paradigm.
Strategy Comparison: A comparative analysis of zero-shot vs. few-shot prompting, revealing that single-example (1-shot) prompting is often superior to multi-example (3-shot, 5-shot) approaches.
Retrieval Analysis: Investigation into example selection methods (Random vs. Lexical vs. Embedding), finding that the presence of an example matters more than the method of selection.
Error Analysis: Detailed categorization of error types (e.g., missed entities, type substitution) to understand how prompting mitigates specific failure modes.

4. Key Results

Performance Gain: Few-shot prompting significantly outperforms the zero-shot baseline across all datasets. Even a single random example improves performance substantially.
The "One-Example" Phenomenon: Contrary to intuition, providing more examples (3 or 5) often decreases performance compared to using just one. The authors attribute this to prompt length exceeding the model's optimal context window or diluting the task clarity.
Selection Strategy: There was no statistically significant difference between Random, Lexical Overlap, and Embedding Similarity selection strategies for single-example prompts. The model generalizes well regardless of how the example is chosen.
Majority Voting: Applying majority voting over three runs yields modest improvements, particularly in Fuzzy settings, but gains are often statistically insignificant in Strict settings.
Comparison with SOTA:
- LLM-based prompting generally underperforms fully supervised State-of-the-Art (SOTA) models (which use fine-tuning on annotated data) by a significant margin (e.g., -0.20 F1 on ajmc (de)).
- Exception: The LLM approach slightly outperformed prior SOTA on the Sonar (de) dataset.
Error Reduction: Few-shot prompting is most effective at reducing "Missed Entity" errors (Type 2) and "Entity Type Substitution" errors (Type 3), helping the model align with correct categories.

5. Significance and Implications

Viable Low-Resource Alternative: While not yet a replacement for supervised fine-tuning in high-accuracy scenarios, LLM prompting offers a cost-effective, language-agnostic, and training-free baseline for historical NER where annotated data is non-existent or too expensive to produce.
Efficiency: The finding that 1-shot is optimal suggests that researchers can achieve strong results with minimal prompt engineering and retrieval overhead, avoiding the complexity of large context windows.
Digital Humanities Impact: This approach enables the analysis of multilingual, under-resourced historical corpora without the bottleneck of manual annotation, facilitating broader access to archival materials.
Future Directions: The paper suggests future work in prompt optimization (compression), advanced retrieval using historical metadata, and expanding evaluation to broader corpora to improve generalizability.

Conclusion: The study demonstrates that LLMs are a powerful tool for historical NER in low-resource settings. While they do not yet match the precision of supervised models trained on specific data, their ability to perform reasonably well with zero or one example makes them a practical and scalable solution for the digital humanities.

1. Problem Statement

2. Methodology

3. Key Contributions

4. Key Results

5. Significance and Implications

More like this