The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

Technical Summary: The LSCD Benchmark

Problem Statement
Lexical Semantic Change Detection (LSCD) is a complex, lemma-level task that automates the analysis of word meaning shifts over time. Currently, the field suffers from significant heterogeneity in modeling options, task definitions, dataset versions, preprocessing strategies, and evaluation metrics. LSCD is typically operationalized as a modular pipeline involving three sequential subtasks:

1. Word-in-Context (WiC): Determining semantic proximity between pairs of word usages.
2. Word Sense Induction (WSI): Clustering these usages into sense groups based on proximity.
3. LSCD: Comparing sense clusters across time periods to detect change (Binary or Graded).

This modularity, while reflecting the annotation process, has led to a lack of comparable evaluation conditions. Existing leaderboards often lack common model implementations, and previous comprehensive repositories (e.g., Schlechtweg et al., 2019) focused on type-based models which have since been outperformed by token-based contextualized embeddings and WiC-fine-tuned models. Consequently, it is difficult to reproduce results, select optimal model combinations, or evaluate the specific contribution of individual pipeline components.

Methodology
The authors present a benchmark repository designed to standardize LSCD evaluation across multiple languages and historical epochs. The methodology is built on the following pillars:

• Modular Evaluation Framework: The benchmark explicitly supports the evaluation of the three constituent tasks (WiC, WSI, and LSCD) independently and as a full pipeline. This allows researchers to isolate performance bottlenecks and test specific components (e.g., a specific WiC model) across different datasets.
• Dataset Integration: The repository integrates a wide range of human-annotated datasets (e.g., DWUG, NorDiaChange, RuSemShift, ChiWUG) covering languages such as German, English, Swedish, Spanish, Norwegian, Chinese, Italian, and Russian. These datasets are unified under a common preprocessing pipeline (tokenization, lemmatization, normalization) and a standard train/dev/test split (CoMeDi split).
• Implementation of State-of-the-Art (SOTA) Models: The benchmark implements token-based approaches, including:
  • Embedding Generation: Using BERT and XLM-R (with options for layer aggregation like averaging or concatenation).
  • WiC Models: Implementing both distance-based approaches (Euclidean, Manhattan, Cosine on embeddings) and neural classifiers (DeepMistake, XL-LEXEME, XL-DURel).
  • Clustering: Supporting correlation clustering on pairwise scores and standard clustering on embeddings.
  • Aggregation Measures: Implementing direct change detection methods that skip clustering, such as Average Pairwise Distance (APD), Cosine distance between average embeddings (COS), and DiaSense.
• Experimental Design: The authors conducted experiments focusing on Graded Change and COMPARE tasks, comparing SOTA WiC models (DeepMistake, XL-LEXEME, XL-DURel) combined with various aggregation measures. They also investigated the impact of dataset versions, preprocessing (lemmatization, normalization), and the discretization of graded WiC predictions.

Key Results
The experiments yielded several significant findings regarding the current state of LSCD:

1. Dominance of APD: For both Graded Change and COMPARE tasks, the Average Pairwise Distance (APD) metric consistently outperformed other aggregation methods (COS, DiaSense), confirming previous findings.
2. New SOTA Models: The recently published XL-DURel model (fine-tuned for ordinal WiC) demonstrated superior performance compared to DeepMistake and XL-LEXEME, establishing a new SOTA.
3. Discretization Benefits: Discretizing graded WiC predictions into ordinal values (via thresholding) improved performance for XL-DURel, aligning better with the human annotation process, though it had negligible or negative effects on XL-LEXEME.
4. WiC as a Determinant: There is a strong correlation between WiC performance and overall LSCD performance; however, the relationship is not absolute. Improvements in WiC ranking do not always guarantee better LSCD results, suggesting that score distribution plays a critical role in aggregation.
5. Data Reliability: Model rankings can shift significantly depending on the dataset version used. Older, less reliable versions of datasets (with fewer annotation rounds) led to different optimal model choices compared to newer, higher-quality versions, highlighting the risk of relying on outdated ground truth.
6. Robustness to Spelling: Current base embedders proved robust against historical spelling variations, with raw input (no preprocessing) often yielding the best results, contrary to the assumption that normalization is strictly necessary.

Significance and Claims
The paper claims that the primary significance of this work lies in the creation of a standardized, transparent, and reproducible benchmark for LSCD. By unifying evaluation procedures and datasets, the repository enables:

• Fair Comparison: Researchers can now compare models under identical conditions, eliminating the noise caused by varying preprocessing and dataset versions.
• Component Analysis: The modular design allows for the careful evaluation of increasingly complex model components, facilitating targeted optimization.
• Reproducibility: Transparent implementation ensures that results can be easily reproduced and extended.

The authors conclude that their results support a "trivial, but rarely acknowledged trend": models that more closely mimic the human annotation process (e.g., using ordinal WiC judgments and discretization) tend to achieve better performance. They emphasize that the field must move away from relying on unreliable, older dataset versions to avoid false conclusions about model efficacy. While the current benchmark focuses on token-based models and skips clustering-based approaches for conciseness, it lays the groundwork for future comparisons involving clustering models and additional datasets.