💬 NLP

Pretraining Language Models for Diachronic Linguistic Change Discovery

This paper proposes an efficient pretraining pipeline using temporally segmented corpora to create specialized language models that outperform fine-tuned baselines in speed and historical accuracy, thereby enabling the discovery of diverse diachronic linguistic phenomena such as lexical, grammatical, and semantic change.

Original authors: Elisabeth Fittschen, Sabrina Li, Tom Lippincott, Leshem Choshen, Craig Messner

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elisabeth Fittschen, Sabrina Li, Tom Lippincott, Leshem Choshen, Craig Messner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a historian trying to understand how people spoke and thought in the 18th century. You want to build a time machine that only knows about that specific era.

Now, imagine you have a super-smart AI assistant (a Large Language Model, or LLM) that has read almost every book ever written, from ancient scrolls to modern tweets. This AI is incredibly fluent and knows a lot, but it has a major problem: it has no concept of time. It mixes up the past and the future. If you ask it about a "car" in 1750, it might confidently tell you about a Ford Mustang, even though cars didn't exist then. It's like asking a modern chef to cook a medieval feast, but they keep accidentally adding ketchup and microwaves to the recipe because they've seen them in every other cookbook.

This paper proposes a clever solution to fix this "time confusion."

The Problem: The "Omnivore" vs. The "Specialist"

Standard AI models are like omnivores. They eat everything—data from all time periods—to become smart. This makes them great at general conversation but terrible at historical research because they "contaminate" the past with future knowledge.

The researchers wanted to build specialists. They wanted models that only "ate" data from specific time periods (e.g., 1750–1820, 1820–1850, etc.) so they could truly understand the language of that specific slice of history without any "future leakage."

The Solution: Building a "Time-Traveling" Team

Instead of taking a giant, modern AI and trying to "fine-tune" it (which is like trying to teach a modern adult to forget everything they know about the internet and only remember the 1700s), the researchers decided to train new models from scratch.

Think of it this way:

The Old Way (Fine-tuning): Taking a modern university professor and trying to force them to unlearn everything they know about the 21st century so they can pretend to be a 1700s scholar. They might still slip up and use modern slang.
The New Way (Scratch Training): Hiring five different apprentices. You give the first apprentice only books from 1750–1820. You give the second apprentice only books from 1820–1850, and so on. You never let them see books from the future.

The "Baby" Trick

Training a model from scratch usually requires a massive amount of money and computer power (like building a supercomputer). But these researchers used a "Baby" trick (inspired by the "BabyLM" community). They used a distillation method, which is like having two master chefs teach a student chef. The student learns by watching the masters, allowing them to become very skilled even with a smaller "diet" of data.

This made the process twice as fast and much cheaper than the traditional method.

The Results: The "Time Leak" Test

The researchers tested their new "time-specialist" models against the old "modern" models using a clever game called a Cloze Test.

Imagine a sentence with a missing word:

"The man drove his ______ to the market."

The Modern Model (Fine-tuned): If you ask this model about 1750, it might guess "car" or "truck" because it knows those words exist in its training data, even though they didn't exist in 1750. It "leaks" future knowledge into the past.
The Time Specialist (Scratch): If you ask the 1750 model, it will guess "horse" or "wagon." It cannot guess "car" because it literally never saw the word in its training data. It respects the timeline perfectly.

Why This Matters

The paper found that while the "Time Specialist" models weren't quite as fluent in general conversation as the giant modern models, they were much better at detecting change.

They could spot exactly when a word changed meaning. For example:

The word "Station" originally meant a military camp or a stopping place.
In the 1840s, as trains were invented, the meaning shifted to "Railway Station."
The "Time Specialist" models could see this shift happening in real-time. The 1820 model thought "station" meant a camp. The 1850 model started thinking it meant a train stop. The modern model just sees both meanings mixed together and can't tell the story of the change.

The Big Takeaway

If you want to study history, literature, or how language evolves, you don't need a giant, all-knowing AI that mixes up the past and future. You need a team of smaller, focused AI models that are strictly trained on specific time periods.

It's the difference between having a general encyclopedia that tells you everything about everything (but gets the dates wrong), and having a team of specialized historians who can tell you exactly what people were thinking in 1790, without accidentally mentioning the internet.

In short: To understand the past, sometimes you have to forget the future.

1. Problem Statement

Large Language Models (LLMs) are increasingly used as tools for knowledge discovery in humanities fields like historical linguistics and literary studies. However, a fundamental methodological conflict exists:

The Conflict: Humanities research often relies on strict temporal or genre-based boundaries (e.g., analyzing language change from 1750–1820 vs. 1820–1850). Standard LLMs are pretrained on massive, heterogeneous corpora that span centuries, leading to anachronistic contamination. When a model trained on modern data is applied to historical texts, it "leaks" future knowledge (e.g., knowing the modern meaning of "car" when analyzing 18th-century texts), making it unsuitable for studying genuine diachronic change.
The Limitation of Current Solutions: Fine-tuning existing LLMs (even with parameter-efficient methods like LoRA or DoRA) does not fully eliminate this contamination because the base model's weights already encode future knowledge. Furthermore, fine-tuning large models is computationally expensive.

2. Methodology

The authors propose a "Train-from-Scratch" approach using efficient pretraining techniques to create models strictly bounded by historical periods.

A. Data Curation

Corpus: Project Gutenberg collection.
Time Range: 1750–1940.
Temporal Slicing: The corpus was divided into five consecutive historical slices (1750–1820, 1820–1850, 1850–1880, 1880–1910, 1910–1940).
Date Attribution: A pipeline was developed to estimate publication dates using WikiData author info, Project Gutenberg metadata, and LLM inference (using Llama3.3-70B quantized to 4 bits).
Token Budget: Each slice was constrained to 10 million tokens for training, with 5M for testing and 1M for validation.

B. Model Training Strategies

The study compares two primary approaches:

Scratch Models (Proposed):
- Architecture: BabyLlama-2 (345M parameters).
- Technique: Trained from scratch using a distillation approach (BabyLM community methods). Two teacher models are trained, and a student model is distilled from them.
- Constraint: Trained only on the specific historical slice, ensuring no future knowledge is present in the weights.
Fine-tuned Models (Baseline):
- Architecture: Llama3-8B backbone.
- Technique: Fine-tuned using DoRA (Weight-Decomposed Low-Rank Adaptation) adapters on the same historical slices.
- Constraint: Starts with a model pretrained on general, anachronistic data.

C. Evaluation Framework

The authors introduced novel metrics to evaluate Fluency vs. Historical Specificity:

Perplexity: Measured against in-period and cross-period test sets.
BLiMP (Benchmark of Linguistic Minimal Pairs): Used to measure general grammatical fluency.
Novel Cloze Evaluation (Word Sense Leakage):
- Constructed using the Oxford English Dictionary (OED) with example sentences and first attestation years.
- Leakage Metric: Measures how often a model trained on period $T$ correctly predicts a word sense that did not exist until a later period $T'$ .
- Recall-Normalized Leakage: Adjusts leakage scores based on the model's ability to recall pre-cutoff senses, isolating true "future knowledge" leakage.

3. Key Contributions

Efficient Pretraining for Historical Boundaries: Demonstrated that training small models from scratch on modest datasets (10M tokens) using BabyLM techniques is more effective for historical specificity than fine-tuning large models.
Quantification of "Time Leakage": Developed a rigorous metric to measure how much future knowledge "leaks" into historical models, proving that fine-tuned models retain significant anachronistic bias while scratch models do not.
Hypothesis Generation Pipeline: Created a workflow to detect linguistic change by contrasting perplexity and word sense acceptance across the battery of time-sliced models.
Computational Efficiency: Showed that scratch models train nearly 2x faster than DoRA-fine-tuned models while maintaining adequate performance for specific historical tasks.

4. Key Results

Historical Specificity (The "Leakage" Problem):
- Fine-tuned models consistently "leaked" information. They performed well on future word senses (e.g., predicting "car" in 1750 text) because their base weights contained modern data.
- Scratch models showed near-zero leakage. Their perplexity increased monotonically when tested on texts outside their training period, and they failed to predict future word senses, correctly reflecting the "arrow of time."
Fluency vs. Specificity Trade-off:
- Fine-tuned models achieved higher BLiMP scores (0.80–0.84) and lower perplexity overall, indicating better general fluency.
- Scratch models had slightly lower BLiMP scores (0.70–0.74) but were specialized to their era. The authors argue this trade-off is necessary for diachronic research; general fluency is less valuable than temporal accuracy.
Discovery of Linguistic Change:
- Case Study 1 ("Station"): Scratch models showed a sharp drop in perplexity for "station" (railway sense) specifically in the 1820–1850 slice, aligning with the historical adoption of rail technology.
- Case Study 2 ("Cholera"): Scratch models tracked the semantic shift of "cholera" from a general intestinal ailment (including pig diseases) to a specific human pathogen. Early models accepted the "pig" sense; later models rejected it. Fine-tuned models accepted the modern definition across all periods, missing the historical nuance.
- Case Study 3 ("End of the line"): Scratch models trained on 1750–1820 data surprisingly predicted the modern idiom "end of the line," suggesting the model captured the precursors (hereditary/writing contexts) that paved the way for the modern metaphor, a nuance lost in fine-tuned models.

5. Significance and Conclusion

Epistemological Rigor: The paper argues that for humanities research, "boundary accuracy" (ensuring a model only knows what was known at a specific time) is more critical than raw performance metrics.
Scalability: The approach is computationally feasible for academic budgets, requiring only modest GPU resources compared to fine-tuning 8B+ parameter models.
Future Directions: The authors suggest this method can be extended to other diachronic boundaries (e.g., genre, dialect) and larger corpora to uncover knowledge-level shifts in history and literature.

In summary, the paper establishes that training small models from scratch on restricted historical data is the superior method for studying language change, as it prevents the anachronistic contamination inherent in fine-tuning large, general-purpose LLMs.