Named Entity Recognition for Pre-Modern Serbian Cyrillic: Using Transformer-Based Models and Generative Data Augmentation
This paper demonstrates that fine-tuning transformer-based models, specifically XLM-RoBERTa, with generative data augmentation significantly improves Named Entity Recognition performance and generalization for low-resource pre-modern Serbian Cyrillic texts, achieving an in-domain F1-score of approximately 0.92 and establishing a recommended model trained with 50% synthetic data for broader historical research.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the quiet corners of digital history, a specialized branch of computer science known as natural language processing is working to make the past readable for machines. This field teaches computers to understand human language, not just as a string of words, but as a structured system where specific names—people, places, and groups—carry distinct meaning. For modern languages, this task has become remarkably accurate, allowing software to instantly scan news articles or social media feeds and pull out every mention of a specific city or person. However, the same technology struggles when faced with historical texts. These documents often use spelling and grammar that have vanished from daily speech, and they exist in far smaller quantities than modern writing. When the language is rare and the data is scarce, the computers that excel at today's news often stumble over the words of centuries ago. This creates a barrier for historians and archivists who wish to use digital tools to explore their collections, as the software cannot reliably distinguish a king's name from a common noun in a medieval charter.
A team of researchers set out to bridge this gap by teaching advanced computer models to recognize names in pre-modern Serbian Cyrillic texts. They focused on a specific challenge: how to make these models robust enough to handle documents they have never seen before, which might differ in spelling, style, or time period from the examples they were trained on. To do this, they turned to a technique called generative data augmentation. Instead of waiting for historians to manually label thousands of new documents—a slow and expensive process—the researchers asked a powerful artificial intelligence to write new, synthetic training examples. They guided this AI to mimic the style of medieval charters and legal documents, creating thousands of fake sentences that looked and sounded like the real thing, complete with correct labels for names and places. They then tested two different types of modern language models, one specialized for the region and another trained on many languages, to see which could best learn from this mix of real and synthetic data.
The researchers began with a collection of genuine medieval documents, including royal charters and letters from the fourteenth century. These texts were carefully annotated by human experts who marked every instance of a person's name, a location, or a reference to a specific group of people. This original collection was small, containing just over three thousand sentences, which is a limited amount of material for training a sophisticated computer model. To expand this foundation, the team used a generative AI system to create new sentences that followed the same patterns as the original texts. They produced three different versions of the training data: one where the synthetic sentences made up thirty percent of the total, another where they made up fifty percent, and a third where the synthetic examples doubled the size of the original dataset. This allowed them to test whether feeding the computer more examples, even if some were artificially generated, would help it understand the nuances of the historical language better.
When the team tested these models on the same type of documents they had been trained on, the results were impressive. The model trained with a mix of real and synthetic data, specifically the version that included thirty percent synthetic examples, achieved a high level of accuracy, correctly identifying names and places in the vast majority of cases. This performance was significantly better than an older style of computer model that relied on different mathematical structures. However, the true test came when the researchers asked the models to read completely new historical documents that were not part of their training. These new texts included a will from the year 1500, a manuscript from a church in Vienna written in 1778, and a set of royal charters from the mid-fourteenth century. These documents represented different eras and writing styles, pushing the models to see if they had truly learned the language or simply memorized the training examples.
The results showed that the models trained on synthetic data were much more adaptable than the older systems. The model that used fifty percent synthetic data proved to be the most reliable across all the new documents, maintaining a high level of accuracy even when faced with the oldest and most stylistically different texts. While the model trained with thirty percent synthetic data performed slightly better on the familiar texts, the fifty percent version handled the unfamiliar ones with greater consistency. This suggests that adding a moderate amount of synthetic examples helps the computer build a more flexible understanding of the language, allowing it to recognize patterns even when the spelling or context shifts. The researchers found that the models were particularly good at identifying people's names, which appeared frequently in the training data, but still struggled slightly with place names and group references, which were rarer and more varied.
The study concludes that generating synthetic examples is a powerful tool for teaching computers to read historical texts, but it is not a simple fix-all. The amount of synthetic data matters; too little might not help enough, while too much might introduce noise that confuses the model. The best results came from a balanced approach, where the synthetic data supplemented the real historical records without overwhelming them. By making their trained models and the data they used publicly available, the researchers have provided a new tool for the digital humanities. This work demonstrates that with the right combination of historical expertise and modern artificial intelligence, we can begin to unlock the names and places hidden within centuries of handwritten documents, making the past more accessible to both scholars and the curious public.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.