💬 NLP

Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection

This paper critically re-evaluates SemEval-2020 Task 1, the leading benchmark for lexical semantic change detection, by identifying significant limitations in its narrow operationalization of change, substantial data quality issues, and restrictive design choices, ultimately arguing that it should be viewed as a partial tool rather than a definitive measure of progress while calling for future improvements in theory, transparency, and scope.

Original authors: Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelmana

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelmana

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a machine that can read old books and tell you exactly how the meaning of words has changed over the last 200 years. To test if your machine is smart, you need a "driver's license test" for it. In the world of computer science and linguistics, the most famous test for this is called SemEval-2020 Task 1.

For years, researchers have used this test as the gold standard to see who has the best "word-change detector." But in this paper, the authors are acting like a tough car inspector. They are saying: "Hold on a minute. This test isn't just a little flawed; the test itself is driving us in the wrong direction."

Here is a breakdown of their critique using simple analogies:

1. The "Lego Block" Problem (Operationalisation)

The Issue: The test assumes that word meanings are like Lego blocks. You either have a block (a specific meaning), or you don't. If you add a new block or lose an old one, the word has "changed."
The Reality: The authors argue that meaning is more like water flowing in a river. It doesn't jump from one distinct shape to another; it shifts gradually.

The Metaphor: Imagine the word "nice." In the past, it meant "foolish." Today, it means "pleasant." The test treats this as if the word suddenly swapped one Lego block for another. But in reality, the meaning slowly melted and reformed over centuries.
The Flaw: By forcing these smooth, flowing changes into rigid "Yes/No" boxes (Did it change? Yes/No), the test misses the subtle, messy, and gradual ways language actually evolves. It's like trying to measure the temperature of a cup of coffee by only asking, "Is it boiling or is it ice?"

2. The "Dirty Glasses" Problem (Data Quality)

The Issue: To test the machine, the organizers gave it a pile of old newspaper scans and books. But these weren't clean scans; they were full of dirt, smudges, and torn pages.
The Reality: The data is full of OCR noise (errors made when computers scan old text).

The Metaphor: Imagine asking a student to read a history book, but the pages are covered in coffee stains, the ink is faded, and some words are missing entirely.
- Sometimes the computer sees "blockade" but reads it as "bloqiiade."
- Sometimes a sentence gets cut off in the middle, like a radio signal fading out.
- Sometimes the computer mislabels a verb as a noun (calling "landing" a "land").
The Flaw: If your machine is trying to learn how words change, but the text it's reading is broken, it might think a word changed meaning when it actually just looked weird because of a smudge. It's like blaming the student for failing the test when the teacher handed them a book with half the pages torn out.

3. The "Tiny Sample" Problem (Benchmark Design)

The Issue: The test only asks the machine to check about 30 to 40 specific words per language.
The Reality: This is like judging a chef's entire career based on how well they cook one single egg.

The Metaphor: If you have a test with 40 questions, getting just one answer wrong changes your score by a huge amount (like 2.5%).
- If the test is too small, the results are just noise. A system might look "better" just because it guessed the right 40 words by luck, not because it's actually smarter.
- It's like a weather forecast that only looks at the temperature in one tiny park in London and claims to predict the weather for the whole world.
The Flaw: The test is too small to be statistically reliable. It's also limited to only four European languages (English, German, Swedish, Latin), so we can't be sure if the "smart machines" would work on languages like Chinese, Arabic, or Swahili.

The Big Picture: What Should We Do?

The authors aren't saying the test is useless. They are saying it's a rough sketch, not a finished painting.

They suggest we need to:

Stop thinking in Lego blocks: Accept that meaning changes gradually, like a river, not in jumps.
Clean the glasses: Make sure the old books are scanned perfectly before we use them to train AI.
Expand the test: Instead of checking 40 words, check thousands. And don't just check European languages; check the whole world.

In short: The current test is a useful starting point, but if we keep using it as the only measure of success, we might be building AI that is good at passing a broken test, but bad at understanding how human language actually works.

1. Problem Statement

SemEval-2020 Task 1 is widely regarded as the most influential shared benchmark for Lexical Semantic Change (LSC) detection. However, the authors argue that despite its popularity and utility, the benchmark suffers from critical flaws that compromise its validity, reproducibility, and generalizability. The paper identifies three primary areas of concern:

Operationalisation: The task reduces complex, gradual semantic shifts to discrete "sense gain/loss" or "sense redistribution," ignoring constructional, collocational, and discourse-level changes.
Data Quality: The underlying corpora contain significant noise (OCR errors, malformed characters, truncated sentences) and inconsistent linguistic annotations (POS tagging, lemmatization errors), particularly in German and Swedish datasets.
Benchmark Design: The use of small, curated target sets (31–48 words per language) creates high statistical uncertainty, while the limited language coverage (four European languages) restricts cross-linguistic generalizability.

2. Methodology

The authors employ a critical, multi-lens evaluative framework to re-examine the benchmark:

Theoretical Analysis: They critique the theoretical underpinnings of the task, specifically the reliance on the DURel framework and the assumption that meaning consists of discrete, stable senses. They argue this ignores the gradient nature of semantic change (e.g., prototype shifts, metonymy).
Quantitative Data Audit:
- The authors sampled the first 100 rows of each language-period subset (English, German, Swedish, Latin).
- They utilized the Google Gemini API to detect specific data quality issues (sentence cut-offs, merged sentences, OCR noise, broken word boundaries, metadata artifacts) and manually verified the results.
- They quantified the prevalence of these errors across the datasets.
Linguistic Re-analysis (Reparsing):
- They tested the impact of re-processing the data using modern parsers (Stanza, UDPipe) compared to the original gold standards.
- They investigated inconsistencies in lemmatization (e.g., historical spelling variations like Lyzeum vs. Lyceum) and Part-of-Speech (POS) tagging errors (e.g., verbs tagged as nouns).
Statistical Simulation:
- They calculated the Margin of Error (MoE) for the small test sets to demonstrate how sampling noise affects accuracy scores.
- They applied Fisher's z-transformation to Spearman rank correlations to illustrate the wide confidence intervals inherent in the small sample sizes (n=37 for English, etc.).

3. Key Findings & Results

A. Operationalisation Limitations

Discreteness vs. Continuity: The benchmark forces continuous semantic processes into binary categories (change/no change). This fails to capture gradual shifts, metonymy, or changes in constructional patterns that do not result in a distinct new "sense."
Annotation Subjectivity: The gold labels are derived from clustering decisions and arbitrary frequency thresholds. The authors cite the German word Seminar as a case where the gold label indicated "no change," despite strong evidence of a metonymic shift (institutional department $\to$ courses).
Temporal Restriction: By only comparing two time periods (C1 vs. C2), the task obscures the trajectory, onset, and reversibility of semantic change.

B. Data Quality Defects

The audit revealed pervasive noise, particularly in non-English datasets:

Swedish (Kubhist): The most problematic dataset, containing 106+ problematic rows in the first period alone, including severe OCR errors (e.g., bloqiiade-cillstZnd), broken word boundaries, and truncated sentences.
German (BZ/ND): Significant OCR noise (e.g., Jn instead of In), merged sentences from newspaper columns, and 59 empty lines in lemmatized files.
English (CCOHA): While cleaner, it suffers from erroneous POS tagging (e.g., the verb land tagged as a noun) and missed target words.
Impact: These errors distort frequency counts, corrupt context windows for embedding models, and invalidate gold-standard annotations that rely on specific POS pairs.

C. Statistical and Design Flaws

Statistical Noise: With only 37 targets for English, a single correct/incorrect prediction changes accuracy by 2.7%. The 95% Margin of Error for English is approximately 16.1%, meaning a model scoring 50% could statistically represent a true performance between 33.9% and 66.1%.
Rank Correlation Uncertainty: For a Spearman correlation of 0.5 on the English set, the 95% confidence interval is roughly [0.21, 0.71], rendering fine-grained comparisons between systems unreliable.
Frequency Bias: Despite controls, many systems correlate strongly with word frequency variables ( $r > 0.8$ ) rather than genuine semantic change.
Lack of Generalizability: The focus on four Indo-European languages limits the applicability of findings to typologically diverse or low-resource languages.

4. Key Contributions

Critical Re-evaluation: The paper provides the first comprehensive, multi-dimensional critique of SemEval-2020 Task 1, moving beyond system performance metrics to question the validity of the benchmark itself.
Empirical Evidence of Noise: It provides a quantitative table of data quality issues (OCR, sentence fragmentation, POS errors) across all four languages, challenging the assumption of "clean" gold data.
Statistical Rigor: It demonstrates mathematically that the small test sets render current evaluation metrics (accuracy, correlation) statistically unstable and prone to sampling noise.
Reparsing Analysis: It highlights the sensitivity of LSC systems to preprocessing choices, showing that modern parsers can outperform the original gold standards when applied to noisy historical data.

5. Significance and Recommendations

The paper argues that SemEval-2020 Task 1 should be treated as a partial test bed rather than a definitive measure of progress in LSC. The authors propose the following for future research:

Broader Operationalisation: Move beyond discrete sense inventories to include constructional, collocational, and discourse-level changes.
Transparent Preprocessing: Future datasets must include explicit denoising protocols, preprocessing logs, and released scripts to ensure reproducibility.
Robust Evaluation: Use larger, vocabulary-wide target sets to reduce statistical noise and avoid the "one-word changes everything" problem.
Cross-Linguistic Expansion: Include typologically diverse, non-European, and non-Latin script languages to test the true generalizability of LSC models.
Continuous Modeling: Shift from binary time-slice comparisons to continuous temporal modeling to capture the trajectory of semantic change.

Conclusion: The paper calls for a paradigm shift in how semantic change benchmarks are designed, emphasizing that valid progress requires better data hygiene, more realistic evaluation settings, and theories that accommodate the gradient nature of meaning change.