💬 NLP

How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

This paper proposes a diagnostic framework that disentangles tokenization costs from semantic comprehension in historical Italian, revealing that while 17th-century texts incur high predictive uncertainty despite robust embedding similarity, this generative instability can be effectively mitigated by approximately 60% through simple temporal context prompting.

Original authors: Maria Levchenko

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maria Levchenko

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, modern librarian (a Large Language Model) who has read almost everything on the internet. You ask this librarian to help you organize a dusty, ancient library filled with books from 300 years ago. The paper asks: How confused does this modern librarian get when reading these old books?

The author, Maria Levchenko, argues that we usually think "old books are hard" as one big, messy problem. But this paper breaks that problem down into three distinct parts, using Italian and Russian history books as test cases.

Here is the breakdown using simple analogies:

1. The "Tokenization Tax": The Broken Zipper

Think of a language model like a person trying to read a book where the letters are glued together in weird ways.

The Problem: Modern computers read text by breaking words into small chunks called "tokens." Old books use old spelling (like the long "s" that looks like an "f" or Russian letters that don't exist anymore).
The Result: Because the computer doesn't recognize these old letters, it has to chop the words into tiny, inefficient pieces. It's like trying to zip up a jacket where the teeth are bent; you have to pull the zipper much harder and longer to get it closed.
The Finding: This "tax" costs the computer more energy and memory. Interestingly, this happens equally for 17th-century Italian and 18th-century Russian. Both languages force the computer to do extra mechanical work just to see the words.

2. The "Comprehension Tax": The Confused Translator

Now, imagine the computer has managed to unzip the jacket (read the words). Does it actually understand what it's reading?

The Surprise: Here is where the two languages act very differently.
- Russian: Even though the letters were weird and hard to unzip, once the computer saw them, it understood the meaning almost perfectly. It was like reading a book written in a strange font, but the story was still very familiar.
- 17th-Century Italian: This was a different story. Even after the computer unzipped the words, it was genuinely confused. The vocabulary was archaic, and the sentence structures were like Latin. The computer was "surprised" by what it read.
The Metaphor: Think of the Russian text as a modern recipe written in a weird font. You can read it easily. The 17th-century Italian text is like a recipe written in a language you barely know, using ingredients you've never heard of. The computer knows the words, but it can't guess what comes next.
Key Insight: Just because a text is hard to type (tokenize) doesn't mean it's hard to understand. The paper proves these are two separate problems.

3. The "Semantic Safety": The Blindfolded Artist

This is the most important finding for libraries.

The Question: If the computer is confused and can't predict the next word, does it still know what the sentence means?
The Answer: Yes! The paper found that even when the computer was struggling to "speak" or predict the text, it could still "see" the meaning perfectly.
The Analogy: Imagine an artist who is blindfolded and trying to draw a cat. They might struggle to get the lines right (high confusion), but if you ask them, "Is this a cat or a dog?" they will point to the cat with 99% certainty.
Why it matters: This means digital libraries can safely use these AI models to search and find old documents. The AI might stumble if asked to rewrite the text, but it is excellent at knowing what the text is about.

The Magic Fix: The "Time Travel" Hint

The paper tested a very simple trick to help the computer.

The Trick: Before showing the computer the old text, the researchers simply added a tiny note: "This is a text from 1687."
The Result: This simple hint reduced the computer's confusion by about 60%.
The Analogy: It's like telling a time-traveling tourist, "You are in the year 1687, so expect people to dress and talk differently." Once the computer knows the "time zone," it stops expecting modern grammar and adjusts its expectations, making the old text much easier to handle.

What About Famous Books?

The paper also warns us about using famous books (like The Betrothed by Manzoni) as test cases.

The Issue: These famous books are so popular that the AI has probably read them a thousand times during its training. They are like "cheat codes."
The Reality: If you test the AI on these famous books, it looks like a genius. But if you test it on obscure, dusty, non-famous archives (the "long tail"), it struggles much more. The paper says we shouldn't judge the AI's ability to handle history based on its performance on famous classics.

Summary

Old text is expensive to process (Tokenization Tax) because of weird letters.
Old text is confusing to predict (Comprehension Tax) if the style is very different from today.
But the AI still understands the meaning (Semantic Robustness), so it's great for searching archives.
A simple hint about the date fixes most of the confusion.
Don't trust famous books as the only test; real history is harder than the classics.

Technical Summary: How Surprising Is Historical Italian to Language Models?

Problem Statement

Large language models (LLMs) are increasingly integrated into digital library workflows for tasks such as OCR post-correction, indexing, and retrieval. However, the ability of these models to process historical language remains poorly understood. Current evaluations often treat "historical difficulty" as a monolithic barrier, conflating three distinct factors: orthographic variation, linguistic distance, and pretraining exposure. This conflation obscures whether models struggle due to mechanical encoding inefficiencies (tokenization) or genuine comprehension deficits (predictive uncertainty). Furthermore, reliance on canonical historical texts (e.g., Manzoni's I Promessi Sposi) as benchmarks may yield overly optimistic results due to high exposure in training corpora, failing to represent the challenges posed by non-canonical, pre-standardized archival materials.

Methodology

The authors propose a diagnostic framework that decomposes historical difficulty into four distinct dimensions:

Tokenization Cost: Measuring "token inflation" (the relative increase in token count compared to a modernized baseline) to quantify encoding inefficiency.
Predictive Uncertainty (Surprisal): Measuring perplexity ($PPL$) to assess the model's uncertainty in next-token prediction, serving as a proxy for comprehension difficulty.
Semantic Robustness: Calculating cosine similarity between embeddings of historical and modern sentence pairs to determine if archaic surface forms map to stable semantic spaces.
Context Sensitivity: Evaluating the impact of minimal temporal context prompts (e.g., "Italian text from 1687") on reducing perplexity.

Datasets:
The study evaluates three datasets spanning three centuries to isolate specific sources of difficulty:

17th-Century Italian (1610–1689): A newly curated corpus of five non-canonical texts (religious treatises, academic prose, etc.) digitized from original page images. This dataset features archaic orthography (e.g., long s ſ), Latin-influenced syntax, and low likelihood of verbatim pretraining exposure.
19th-Century Italian (I Promessi Sposi, Quarantana edition): A high-exposure control condition. As a canonical text, it serves as a "best-case" baseline to illustrate model behavior when orthography is standardized and lexical familiarity is high.
18th-Century Russian Civil Print Books: A contrastive stress test featuring extreme orthographic divergence (obsolete characters like yat, fita) but comparatively limited lexical and syntactic distance from modern Russian.

Models:

Tokenization: OpenAI o200k_base (GPT-4o/5 family) and Qwen 2.5 tokenizers.
Perplexity: Qwen 2.5-1.5B (base checkpoint) to ensure raw probability distributions.
Embeddings: OpenAI text-embedding-3-large, Google LaBSE, and Qwen 2.5-1.5B hidden states.

Key Results

1. Dissociation Between Encoding Cost and Comprehension

The study reveals a distinct decoupling between tokenization efficiency and predictive difficulty.

Tokenization Tax: Both 18th-century Russian and 17th-century Italian incur comparable tokenization penalties (approx. 25–30% inflation). In Russian, this is driven by byte-level fallback for obsolete characters; in Italian, it is driven primarily by the long s (ſ), which causes subword fragmentation despite being present in the vocabulary.
Predictive Difficulty: Despite similar encoding costs, the perplexity ratios diverge sharply.
- Russian: Shows only a modest increase in perplexity (1.25× modern baseline), indicating that orthographic disruption alone does not significantly confuse the model's predictive capabilities.
- 17th-Century Italian: Is on average 2.4× more surprising than its modern equivalent, with academic prose reaching 3.2×.
Conclusion: Tokenization cost is a poor proxy for historical complexity. High token inflation does not necessarily imply high comprehension difficulty, and vice versa.

2. Genre and Register Outweigh Chronology

Historical difficulty is not a linear function of time.

Within the 17th-century corpus, the most difficult text (Academic Preface, 1687) and the easiest text (Descriptive Portraits, 1689) were published only two years apart.
The academic text exhibited a perplexity ratio of 3.22×, while the descriptive text was 1.81×.
Finding: Register (e.g., structural fragmentation in academic prose vs. rhythmic redundancy in theological narrative) is a stronger determinant of model performance than chronological age. Canonical 19th-century texts behave as high-exposure baselines, masking the structural challenges of non-canonical materials.

3. Semantic Robustness Despite Generative Instability

High perplexity does not imply representational degradation.

Embedding similarity between historical and modern sentences remains robust across all datasets (Cosine Similarity $\ge$ 0.85).
The Qwen 2.5 model achieved a similarity of 0.938 for 17th-century Italian, despite the high perplexity observed in generation tasks.
Implication: Models can successfully map archaic surface forms to the correct semantic space even when they lack the probabilistic confidence to generate them token-by-token.

4. Mitigation via Temporal Context

A minimal, model-agnostic intervention significantly reduces difficulty.

Prepending a short temporal context prompt (e.g., "Italian text from 1687") reduced historical surprisal by approximately 60% for 17th-century Italian.
The effect was less pronounced for Russian (17% reduction), suggesting that context priming is most critical for texts with high distributional divergence (lexical/syntactic drift) rather than those with primarily orthographic noise.

Significance and Claims

The paper argues that "historical difficulty" is not a single, insoluble problem but a composite of interacting factors that must be evaluated separately:

Decoupling Metrics: The study demonstrates that tokenization efficiency (a mechanical tax) is uncorrelated with semantic robustness. Digital libraries can safely deploy LLMs for semantic retrieval tasks (search, clustering, classification) even when dealing with texts that incur high tokenization costs and high perplexity, as the models maintain stable internal representations.
Benchmark Validity: Reliance on canonical, high-exposure texts (like Manzoni) leads to systematic underestimation of the challenges posed by the "long tail" of archival materials. Evaluations must include non-canonical, pre-standardized texts to accurately assess model suitability.
Strategic Deployment:
- Retrieval: Can be deployed with minimal normalization, as semantic alignment remains high.
- Generation: Tasks like OCR correction or summarization remain sensitive to high surprisal and require specific mitigation strategies.
Cost-Effective Mitigation: The authors demonstrate that simple temporal prompting is a highly effective, cost-free strategy to bridge the gap between model expectations and historical registers, reducing predictive error by over half without the need for fine-tuning or aggressive text normalization.

In conclusion, the paper advocates for a shift in evaluation frameworks: moving beyond surface-level metrics like token counts to distinguish between the mechanical cost of encoding, the uncertainty of prediction, and the stability of semantic representation. This allows digital libraries to leverage LLMs effectively as "capable readers" of history, even if they are not yet "perfect speakers" of it.