💬 NLP

Rethinking Metrics for Lexical Semantic Change Detection

This paper introduces Average Minimum Distance (AMD) and Symmetric Average Minimum Distance (SAMD) as robust alternatives to traditional metrics like APD and PRT for detecting lexical semantic change, demonstrating their superior performance across various languages, models, and representation spaces.

Original authors: Roksana Goworek, Haim Dubossarsky

Published 2026-02-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roksana Goworek, Haim Dubossarsky

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out how the meaning of a word has changed over time. Maybe you want to know how the word "awful" went from meaning "full of awe" to meaning "terrible," or how "mouse" went from just a small animal to also mean a computer device.

For a long time, linguists and computer scientists have used a specific set of tools to solve this mystery. They treat words like clouds of data points. If the cloud in 1900 looks very different from the cloud in 2020, the word has changed.

The Old Tools: The "Average" Approach
The two main tools used so far are like taking a class photo or calculating an average.

The "Class Photo" (PRT): You take every single time a word was used in 1900, find the exact center of that group, and do the same for 2020. Then you measure the distance between the two centers.
The "Average Distance" (APD): You take every single word from 1900 and measure its distance to every single word from 2020, then average all those numbers.

The Problem:
Imagine a classroom where 99 students are sitting in the same spot, but one student has moved to the back of the room to start a new club.

If you take the average, the center of the class barely moves. You might think, "Nothing changed!"
If you look at the average distance, the one student who moved is drowned out by the 99 who stayed put.

This is the problem with the old tools: they are great at spotting when everyone changes their mind, but they are terrible at spotting when a new idea emerges or an old idea disappears. They miss the subtle, local changes.

The New Tools: The "Matchmaker" Approach
The authors of this paper, Roksana Goworek and Haim Dubossarsky, say, "Let's stop looking at the averages and start looking at the individuals." They introduce two new tools: AMD and SAMD.

Think of these tools as matchmakers or dating apps for word usages.

AMD (Average Minimum Distance):
Imagine you have a list of all the times the word "mouse" was used in 1900 and a list for 2020.
- For every 1900 usage, the AMD tool asks: "Who is your closest soulmate in the 2020 list?"
- It finds the best match for each one, measures how far apart they are, and averages those distances.
- Why it's better: If a new sense of the word appears in 2020 (like the computer mouse), the old "animal" usages from 1900 will have to reach far to find a match, or they might not find a good one at all. This creates a high "distance" score, correctly flagging that a big change happened. It doesn't get distracted by the crowd.
SAMD (Symmetric Average Minimum Distance):
This is a stricter version of the matchmaker. It forces a one-to-one pairing.
- It says, "Okay, we have 100 usages in 1900 and 100 in 2020. We must pair them up perfectly, one-to-one, with no leftovers."
- It tries to make the best possible pairs to minimize the total distance.
- Why it's better: This is great for seeing if the shape of the usage has changed. If the 2020 crowd is clustered in a totally different way than the 1900 crowd, the "cost" to pair them up will be high.

The Big Experiment
The authors tested these new tools against the old ones using:

7 different languages (English, German, Chinese, etc.).
Many different AI models (some smart and specialized, some general-purpose).
Different "views" of the data: Sometimes they looked at the full, high-definition data, and sometimes they squashed the data down into a smaller, simpler version (like compressing a high-res photo into a thumbnail).

The Results: Why This Matters
Here is the punchline in simple terms:

The Old Tools Crumble Under Pressure: When the data was compressed or simplified (which happens often when dealing with older languages or limited computing power), the old "Average" tools failed miserably. They couldn't see the changes anymore.
The New Tools are Tough: AMD and SAMD kept working well, even when the data was messy or simplified. They are like a sturdy pair of hiking boots compared to the old tools' fancy dress shoes.
The "Definition" Twist: The authors also tried translating the word usages into simple text definitions (like asking an AI to define the word in different contexts). When they used this "definition space," the AMD tool became even better at spotting changes. It's like switching from looking at blurry shadows to reading clear text.

The Takeaway
The paper argues that we shouldn't rely on just one way to measure how words change. The old "average" methods are okay for big, sweeping changes, but they miss the nuances.

The new AMD and SAMD tools are like a magnifying glass. They let us see the small, local shifts in meaning—like a word gaining a new slang meaning or losing an old one—without getting lost in the noise of the crowd. They are more robust, more flexible, and they work even when we don't have perfect data.

In short: Stop averaging the crowd; start matching the individuals. That's how you truly understand how language evolves.

1. Problem Statement

Lexical Semantic Change Detection (LSCD) aims to quantify how word meanings evolve over time using diachronic corpora. While recent advancements have shifted from static word embeddings to contextualized language model embeddings (usage-level representations), the field relies heavily on a limited set of global aggregation metrics:

Average Pairwise Distance (APD): Computes the mean distance between all usage pairs across two time periods.
Prototype Distance (PRT): Computes the cosine distance between the centroids (averages) of usage distributions for two periods.

The Core Issue: These global metrics implicitly assume that semantic change manifests as a broad shift in the entire distribution of word usages. Consequently, they often fail to capture localized phenomena, such as:

The emergence of a new sense.
The disappearance of an older sense.
Changes affecting only a small subset of usages (which may be drowned out by the "bulk" of unchanged usages).
They are also highly sensitive to dimensionality reduction and less effective with non-specialized encoders.

2. Methodology

The authors propose two new metrics based on local correspondence rather than global aggregation.

A. New Metrics

Average Minimum Distance (AMD):
- Concept: For every usage instance in Corpus A, it finds the nearest neighbor in Corpus B (and vice versa).
- Calculation: It averages these minimum distances.
- Directionality: It allows for directional decomposition ( $A \to B$ and $B \to A$ ), enabling the analysis of asymmetric changes (e.g., sense broadening vs. narrowing).
- Mechanism: Matches each usage to its closest semantic counterpart, capturing local shifts.
Symmetric Average Minimum Distance (SAMD):
- Concept: An extension of AMD that enforces a one-to-one alignment between usage instances across periods.
- Calculation: It uses a greedy algorithm (or Hungarian algorithm) to match usages such that no two usages in one period map to the same usage in the other, minimizing the total distance.
- Benefit: Mitigates the "hubness" problem (where a few points become nearest neighbors to many others) and captures shifts in distribution concentration.

B. Experimental Setup

Datasets: Benchmarks across 7 languages (English, German, Swedish, Latin, Spanish, Norwegian, Chinese) from SemEval-2020, LSCDiscovery, and others.
Encoders:
- Specialized: XL-LEXEME (fine-tuned for Word-in-Context).
- Non-Specialized: Multilingual models (XLM-RoBERTa, mmBERT, RemBERT, m-E5) and Monolingual models (RoBERTa, GBERT, etc.).
Representation Spaces: To test robustness, embeddings were projected into four spaces:
1. Full: Original high-dimensional embeddings.
2. Definition-based: Projected onto distances to LLM-generated definitions (interpretable space).
3. PCA: Principal Component Analysis (variance-preserving).
4. Random: Randomly selected dimensions (stress test).

3. Key Contributions

Introduction of AMD and SAMD: New metrics that quantify semantic change via local usage-level correspondence rather than global distributional shifts.
Robustness to Dimensionality Reduction: Demonstrated that local correspondence metrics maintain performance even when embeddings are reduced to very low dimensions (e.g., 4–8 dimensions), whereas global metrics (APD/PRT) collapse.
Interpretability via Definition Spaces: Showed that AMD performs exceptionally well when embeddings are projected into a space defined by LLM-generated definitions, bridging the gap between quantitative metrics and human-understandable semantics.
Encoder Agnosticism: Proved that these metrics offer significant improvements over APD/PRT regardless of whether the underlying encoder is specialized (fine-tuned) or generic (pre-trained only).

4. Results

Overall Performance: AMD and SAMD consistently achieved higher Spearman correlations with gold-standard change scores compared to APD and PRT across all languages and encoders.
Dimensionality Reduction (Stress Test):
- PRT: Failed almost completely under PCA, dropping to near-zero correlation.
- APD: Dropped to ~0.30 correlation in low dimensions.
- AMD/SAMD: Maintained correlations near 0.40 even when reduced to 4 dimensions. SAMD showed slightly higher robustness under PCA.
Specialized vs. Non-Specialized Encoders:
- Specialized (XL-LEXEME): SAMD + PCA yielded the best results (0.694 correlation).
- Non-Specialized: AMD paired with Definition-based spaces yielded the strongest performance for monolingual encoders, suggesting that sense-aware organization in the embedding space benefits local matching.
Metric Behavior:
- Global metrics (APD/PRT) are sensitive to the "bulk" of data and fail when noise is introduced via dimensionality reduction.
- Local metrics (AMD/SAMD) are sensitive to specific usage shifts and remain stable even when the embedding structure is manipulated.

5. Significance and Implications

Beyond "One-Size-Fits-All": The paper argues against relying solely on APD or PRT. The choice of metric should depend on the research goal (e.g., detecting broad shifts vs. specific sense emergence) and the constraints of the data (e.g., low-resource languages requiring dimensionality reduction).
Discovery-Oriented LSCD: Global metrics often flag topical shifts as semantic changes. Local correspondence metrics (AMD/SAMD) are better suited for large-scale discovery because they are less vulnerable to confounds and better at identifying genuine, localized semantic evolution.
Interpretability: The success of AMD in definition-based spaces suggests a path toward transparent LSCD. By projecting embeddings into spaces defined by human-readable definitions, researchers can analyze why a change occurred, not just that it occurred.
Methodological Expansion: AMD and SAMD should be viewed as complementary tools that expand the toolkit for contextualized embedding analysis, particularly for scenarios involving reduced representation spaces or non-specialized models.

In conclusion, the authors demonstrate that shifting from global distributional aggregation to local usage correspondence provides a more robust, stable, and interpretable framework for detecting lexical semantic change.