From Traditional Taggers to LLMs: A Comparative Study of POS Tagging for Medieval Romance Languages

Imagine you are trying to read a library of ancient books written in three different medieval languages: Medieval Occitan, Medieval Catalan, and Medieval French. These aren't just old versions of modern languages; they are like wild, untamed cousins. The spelling changes from page to page (sometimes "fire" is written as fuoc, sometimes foc), the grammar is complex, and there are very few dictionaries or teachers available to help you understand them.

The goal of this paper is to teach computers how to label every word in these books with its "job" (like noun, verb, or adjective). This process is called Part-of-Speech (POS) tagging. It's the essential first step before a computer can truly understand the story, translate it, or analyze its history.

The researchers asked: Can modern "super-smart" AI (Large Language Models or LLMs) do a better job at this than the old, traditional tools we've been using?

Here is the breakdown of their experiment, explained through simple analogies:

1. The Contestants

The study set up a race between two types of "taggers":

• The Old Guard (Traditional Taggers): Think of these as rule-following librarians. They have a strict rulebook based on modern languages. They try to guess a word's job by looking at its shape and comparing it to known patterns. They are fast and cheap, but they get confused easily when the spelling is weird or the word is rare.
• The New Contenders (LLMs): These are like genius polyglots who have read almost everything ever written. They don't just follow rules; they understand context. They can look at a sentence and "feel" what a word is likely to be, even if the spelling is strange.

2. The Training Methods (How they taught the AI)

The researchers didn't just let the AI guess; they tried different ways to teach it:

• Zero-Shot (The "Cold Read"): They gave the AI a prompt saying, "You are an expert. Here is a word, tell me its job," without showing it any examples. It's like asking a tourist to translate a menu without giving them a phrasebook.
• Few-Shot (The "Cheat Sheet"): They gave the AI a tiny list of examples first (e.g., "This word is a noun, this one is a verb"). It's like handing the tourist a few key phrases before they start.
• Fine-Tuning (The "Apprenticeship"): They took the AI and trained it specifically on the medieval texts. It's like hiring the tourist as an intern for a month to study only these specific books.
• Cross-Lingual Transfer (The "Language Family Reunion"): This is the most interesting part. They trained the AI on a mix of languages. For example, they taught it on Catalan and Occitan together, then tested it on Occitan. It's like teaching someone Spanish and Italian, then seeing if they can understand Portuguese better because the languages are related.

3. The Results: Who Won?

The results were clear, but with some interesting twists:

• The AI Wins: In almost every case, the modern AI (LLMs) beat the old rule-based tools. The "genius polyglots" handled the messy, weird spellings much better than the "rule-following librarians."
• Training Matters: The AI did best when it was actually trained on the data (Fine-tuning) rather than just guessing.
• The "Goldilocks" of Mixing Languages:
  • For the smallest, hardest dataset (Medieval Occitan): The AI learned best when it studied all three languages together (Trilingual). It was like the student needing to hear all the related dialects to finally understand the tricky one.
  • For the medical text (Medieval French): The AI did best when it was paired only with Catalan (Bilingual). Adding the third language actually made it slightly worse. It's like a student learning French; sometimes adding a third language (like Spanish) helps, but sometimes focusing just on the closest relative (Italian/Catalan) is more effective for a specific, small task.

4. The "Why" Behind the Win

The researchers found that the AI was particularly good at figuring out words that have meaning and context, like adjectives, adverbs, and pronouns. The old tools struggled with these because they rely on rigid rules. The AI, however, could look at the whole sentence and say, "Ah, this word describes the noun, so it must be an adjective," even if the spelling was weird.

5. The Catch (Limitations)

The paper is honest about what it didn't do:

• Cost: The "genius polyglots" (the big AI models) are expensive to run and require powerful computers. The old tools are free and fast.
• Data Size: The datasets were very different in size (one had 2,000 words, another had 59,000). The researchers couldn't perfectly separate whether the AI did well because of the language or just because it had more data to learn from.
• Specificity: They only tested these three languages. We don't know yet if this works for other ancient languages that aren't related to Romance languages.

The Bottom Line

This paper proves that for reading messy, ancient medieval texts, modern AI is a powerful new tool. It doesn't just replace the old methods; it improves them significantly, especially when you teach it by mixing related languages. However, there isn't a "one-size-fits-all" strategy. Sometimes you need to mix all the languages together, and sometimes you need to focus on just two, depending on the specific text you are trying to read.

For historians and digital humanists, this means we can now build better tools to automatically organize and understand thousands of years of literature that were previously too messy for computers to handle.