💬 NLP

DHPLT: large-scale multilingual diachronic corpora and word representations for semantic change modelling

This paper introduces DHPLT, an open resource comprising large-scale, multilingual diachronic corpora across 41 languages with pre-computed embeddings and lexical substitutions, designed to address the scarcity of data for semantic change modeling beyond high-resource languages.

Original authors: Mariia Fedorova, Andrey Kutuzov, Khonzoda Umarova

Published 2026-02-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mariia Fedorova, Andrey Kutuzov, Khonzoda Umarova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to study how human language evolves, like watching a river change its course over decades. You need a massive collection of old letters, newspapers, and diaries to see how words like "mouse" (the animal vs. the computer device) or "cloud" (the sky vs. data storage) shifted their meanings.

The problem? For most of the world's languages, those "old letters" don't exist in a neat, organized library. They are scattered, messy, or simply non-existent.

Enter DHPLT. Think of DHPLT as a massive, time-traveling digital library built by researchers to fix this problem. Here is how it works, broken down simply:

1. The Source: The "Internet Time Machine"

Instead of waiting for librarians to organize history, the researchers went straight to the source: the Internet. They used a giant project called HPLT (High-Performance Language Technologies) that has already scraped billions of web pages.

The Analogy: Imagine the internet is a giant, chaotic ocean. HPLT is a net that scoops up clean, filtered water.
The Trick: To turn this ocean into a timeline, they didn't look at when the text was written (which is often hidden or missing). Instead, they looked at when the web crawler downloaded the page.
- If a bot downloaded a page in 2015, that page must have existed by 2015. It's like finding a time capsule: if you found it in 2015, the contents couldn't have been made after 2015.

2. The Collection: 41 Languages, 3 Time Capsules

The researchers didn't just grab random pages. They built three specific "time windows" for 41 different languages (from English and Spanish to Tamil and Georgian):

Window 1 (2011–2015): The "Early Internet" era.
Window 2 (2020–2021): The "Pandemic" era (when the world changed rapidly).
Window 3 (2024–Present): The "AI Boom" era.

Each window contains 1 million documents per language. That's a lot of reading material!

3. The "Target Words": The Stars of the Show

You can't read a million books to find one word. So, the researchers picked a "Wanted List" of about 18,000 interesting words for each language (nouns, verbs, and adjectives).

They didn't just give you the text; they did the heavy lifting for you. They pre-calculated three different types of "meaning maps" for these words:

Static Maps (Word2Vec): Like a dictionary definition that doesn't change based on context. It's the "average" meaning of a word in that era.
Contextual Maps (Token Embeddings): Like a high-definition photo. It shows how the word was used in a specific sentence. Did "AI" mean a robot in a video game, or a chatbot? This map knows the difference.
Substitutes: If you had to replace the word with another one in that sentence, what would you use? This helps track how the "vibe" of a word changed.

4. Why This Matters: Seeing the Shift

The paper shows that this tool works beautifully. They tested it with the word "AI":

In 2011–2015: The word was hanging out with "video games," "robots," and "cars."
In 2020–2021: It started hanging out with "chatbots" and "machine learning."
In 2024: It's now surrounded by "ChatGPT," "generative," and "LLMs."

They saw the exact same shift in Spanish ("IA") and Russian ("ИИ"). It's like watching a movie in fast-forward, seeing exactly when a word's personality changed.

The Catch (Limitations)

The researchers are honest about the flaws. Because they used "download dates" instead of "writing dates," a document downloaded in 2024 might actually have been written in 2010.

The Analogy: It's like finding a 2010 newspaper in a 2024 trash can. You know it exists in 2024, but it's not new.
The Result: The timelines aren't perfectly sharp, but they are "good enough" to see the big picture of how language evolves.

The Bottom Line

DHPLT is a gift to the scientific community. Before this, studying how language changes was like trying to build a house with only a few bricks from a dozen different countries. Now, they have a warehouse full of bricks from 41 different cultures, sorted by time, with blueprints already drawn up. It allows researchers to finally ask: "How does the world speak differently today compared to ten years ago?" and actually get an answer.

1. Problem Statement

The field of Lexical Semantic Change Detection (LSCD)—tracing how word meanings shift over time—is currently constrained by a lack of large-scale, multilingual diachronic corpora.

Scarcity of Data: Existing resources are limited to a dozen high-resource languages (mostly Indo-European) and often rely on manually curated historical archives (e.g., newspaper collections), which are non-existent for the vast majority of the world's languages.
Lack of Standardization: There is no unified, open-source resource that allows researchers to compare semantic change across diverse language families and time periods.
Computational Barriers: Generating semantic representations (embeddings, lexical substitutes) for specific target words across multiple time periods is computationally expensive, hindering immediate experimentation for researchers.

2. Methodology

The authors propose DHPLT (Diachronic HPLT), a resource derived from the HPLT v3.0 project (High-Performance Language Technologies), which utilizes web-crawled data.

A. Data Source and Temporal Signal

Source: Documents are extracted from the HPLT v3.0 dataset, which consists of cleaned, deduplicated, and language-identified web pages from Common Crawl and the Internet Archive.
Temporal Proxy: Since manual date labeling of billions of web pages is impossible, the authors use web crawl timestamps as the creation date signal.
- Logic: If a page was crawled in year $X$ , the content was created no later than $X$ . While this allows for some "contamination" (e.g., a 2024 crawl containing 2010 text), it provides a reliable "upper bound" for the text's age.
Time Periods: The data is split into three distinct periods with gaps to facilitate change detection:
1. 2011–2015 (Early period)
2. 2020–2021 (Pandemic period)
3. 2024–Present (Most recent)

B. Language Selection

From the original 198 languages in HPLT v3.0, 41 languages were selected based on:

Volume: At least 0.5 million documents per time period.
Model Availability: Existence of a corresponding HPLT T5 monolingual encoder-decoder model.

Coverage: 12 different language families (e.g., Indo-European, Afro-Asiatic, Uralic, Sino-Tibetan).

C. Data Processing Pipeline

Sampling: For each language and time period, 1 million documents are randomly sampled (0.5M if fewer are available).
Target Word Selection:
- Starting from the T5 model vocabulary, the authors filter for full words (not sub-word pieces) appearing as nouns, verbs, or adjectives.
- Frequency Threshold: Words must appear at least 10 times in each of the three time periods (30+ total).
- Script Verification: Words must be written in the language's primary script.
- Lemmatization: Tokens are mapped to lemmas to group morphological variants.
- Result: An average of ~18,600 target words per language.

D. Semantic Representations

To lower the barrier for entry, the authors pre-compute and release four types of representations for the target words:

Contextualized Token Embeddings:
- Generated using HPLT T5, XLM-R, and HPLT GPT-BERT models.
- 1,000 random occurrences per word for T5/XLM-R; 100 for GPT-BERT.
Lexical Substitutes:
- Generated using HPLT GPT-BERT models (masked language modeling) to provide top-15 substitutes for 100 random occurrences.
- Note: T5 models were not used for this due to their span-masking objective producing long sequences rather than single lexemes.
Static Word Embeddings (SWE):
- Word2Vec (SGNS) models trained on the DHPLT corpora for each language/time period.
- Alignment: Vector spaces for periods 1 and 2 are aligned to period 3 using Procrustes alignment to enable direct similarity comparisons across time.
Frequency Counts:
- Raw counts of target words across the three periods to control for frequency effects and plan computational resources.

3. Key Contributions

DHPLT Corpus: The first open collection of diachronic corpora covering 41 languages across 12 language families, with 1 million documents per language per time period (approx. 59 billion words total).
Pre-computed Representations: A massive reduction in computational overhead for researchers, providing ready-to-use embeddings (T5, XLM-R, GPT-BERT), lexical substitutes, and static word vectors.
Standardized Benchmarking: Establishes a unified framework for multilingual LSCD, moving beyond the "dozen high-resource languages" limitation.
Open Source Tools: Provides code to reproduce the dataset and generate custom subsets or target word lists.

4. Results and Validation (Sanity Checks)

The authors validated the utility of DHPLT by analyzing semantic shifts in specific words:

Case Study: "AI" (English) / "IA" (Spanish) / "ИИ" (Russian):
- 2011–2015: Semantics were dominated by gaming (NPCs, RPGs, FPS) and robotics.
- 2020–2021: Shifted toward IoT, machine learning, and chatbots.
- 2024: Dominated by Generative AI, LLMs, and ChatGPT.
- Observation: The static word embeddings (SWEs) clearly captured this trajectory, even with the "upper bound" limitation of crawl timestamps.
Case Study: "Remote" (English) / "Remoto" (Spanish):
- Showed a significant semantic shift between 2011–2015 and 2020–2021, moving from general spatial distance to specific "remote work" contexts.
Stability Check: Legal terms like "legislative" and "jurisdiction" showed minimal semantic change across periods, confirming the model's ability to distinguish between changing and stable concepts.
Cross-lingual Consistency: Similar semantic trajectories were observed across English, Spanish, and Russian, validating the multilingual applicability of the resource.

5. Significance and Limitations

Significance:

Democratization of LSCD: Enables research into semantic change for low-resource and non-Indo-European languages.
Long-term Dynamics: The three-period structure allows for the study of long-term, multi-point semantic dynamics rather than simple binary comparisons.
Efficiency: By providing pre-computed embeddings, the resource allows researchers to focus on modeling and analysis rather than data preprocessing.

Limitations:

Temporal Precision: The use of crawl timestamps as a proxy for creation date means the "upper bound" is not a precise creation date. Period 3 may contain older texts, potentially blurring the edges of semantic shifts.
Scope of Representations: Pre-computed representations are limited to selected target words, not the entire vocabulary, due to storage and compute constraints.
Model Temporality: The pre-trained models used for embeddings do not inherently encode time; they are static models trained on specific time slices. Future work could involve training models specifically on period-specific data.

In conclusion, DHPLT fills a critical gap in computational linguistics by providing a scalable, multilingual infrastructure for studying how language evolves, specifically targeting the discovery of semantic change across diverse linguistic contexts.