💬 NLP

In the LLM era, Word Sense Induction remains unsolved

This paper highlights persistent methodological flaws in Word Sense Induction (WSI) evaluation and demonstrates that despite exploring various unsupervised, LLM-based, and semi-supervised approaches, no current method surpasses the simple "one cluster per lemma" heuristic, indicating that WSI remains an unsolved challenge requiring better integration of lexicons and LLM capabilities.

Original authors: Anna Mosolova, Marie Candito, Carlos Ramisch

Published 2026-03-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Anna Mosolova, Marie Candito, Carlos Ramisch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Word Meaning" Puzzle

Imagine you are a librarian trying to organize a massive library where books are written in a language that changes meaning depending on the context.

Take the word "Bank."

In one sentence, it's a place where you keep money.
In another, it's the side of a river.
In a third, it's a place where you sit (a bench).

Word Sense Induction (WSI) is the task of teaching a computer to figure out these different meanings without being given a dictionary. The computer has to look at thousands of sentences and say, "Ah, when 'bank' is near 'money,' it's Group A. When it's near 'water,' it's Group B."

For a long time, researchers thought that with the rise of Large Language Models (LLMs) (like the AI you are talking to right now), this puzzle would be solved. They assumed these super-smart AIs could just "know" the meanings.

This paper says: "Not so fast." The authors argue that the puzzle is actually harder than we thought, and the current tools (including the smartest AIs) are failing when tested on real-world data.

The Problem: The "Fake Exam" vs. The "Real World"

The authors discovered that previous tests for these AI systems were rigged, like a student studying for a math test using only practice problems that are all easy algebra, while the real exam includes calculus.

The Old Way (The Rigged Exam): Previous datasets used to train and test these AIs were cherry-picked. They only included words that were known to be tricky (polysemous) and had lots of examples. They ignored simple words that only have one meaning.
- Analogy: It's like training a dog to fetch a ball, but you only ever throw the ball. You never test if the dog can ignore a stick. When the AI sees a simple word, it gets confused because it was trained to expect complexity.
The New Way (The Real World): The authors created a new test using SemCor, a massive, natural collection of text. This dataset includes simple words, complex words, and everything in between, just like real life.
- Analogy: This is like throwing the dog a mix of balls, sticks, leaves, and shoes to see what it actually does.

The Shocking Results: The "One-Size-Fits-All" Trick Wins

When the authors ran their tests on this "Real World" data, something bizarre happened.

The Smartest AI: They tried the most advanced clustering algorithms and the newest LLMs (like GPT-4 and Llama).
The Dumbest Baseline: They also tried a "dumb" method called 1cpl (One Cluster Per Lemma). This method simply says: "Don't bother grouping the words. Just put every single instance of the word 'bank' into one giant pile."

The Result: The "dumb" method (1cpl) beat the smart AI systems!

Why? Because in real life, most words are used the same way 90% of the time. If you have a word that appears 100 times, maybe 95 times it means "money bank" and 5 times it means "river bank." If you just group them all together, you get the "main idea" right most of the time. The fancy AI tries to split them up perfectly, makes mistakes, and ends up with a lower score.

The Takeaway: The current state-of-the-art AI is actually worse at this specific task than a simple rule that says "assume everything is the same."

The LLM Struggle: The "Over-Confident Student"

The authors also tried asking the LLMs directly to do the job (e.g., "Here are 50 sentences with the word 'bank'; please group them by meaning").

The Result: The LLMs struggled badly. They got confused, forgot the instructions, or gave irrelevant answers.
Analogy: Imagine asking a genius student to sort 500 cards by color. Instead of sorting them, the student starts writing a poem about the cards, or asks, "What is a card?" or just stops halfway through. The LLMs are great at chatting, but terrible at organizing large lists of data without specific training.

The Solution: The "Magic Dictionary" (Wiktionary)

So, if the AI is failing, how do we fix it? The authors found a way to boost performance, but it requires a little help from a human-made resource: Wiktionary (a free, online dictionary).

They tried three tricks:

Data Augmentation: Feeding the AI more examples of the word from other books (like Wikibooks) so it sees more patterns.
Must-Link Constraints: Telling the AI, "Hey, these two sentences definitely mean the same thing, so don't separate them."
The Dictionary Boost: Using the definitions and examples from Wiktionary to guide the AI.

The Winner: The system that used Wiktionary as a guide (both for training and for giving hints) finally beat the "dumb" baseline. It improved the score by about 3.3%.

Analogy: It's like giving the student a cheat sheet. The student (AI) is still struggling, but if you give them the dictionary (Wiktionary) and say, "Use this to help you sort," they finally get an A.

Summary of the Paper's Message

The Field is Broken: Current tests for Word Sense Induction are flawed because they use fake, unrealistic data.
AI isn't Magic Yet: Even the most advanced LLMs cannot automatically figure out word meanings better than a simple "group everything together" rule when looking at real-world text.
Human Knowledge is Still Needed: To make progress, we can't just rely on raw AI power. We need to combine AI with human-made resources like dictionaries (Wiktionary).
The Future: We need to stop pretending the problem is solved. We need better ways to mix the "common sense" of human dictionaries with the "pattern recognition" of AI.

In short: We thought the AI had grown up and could handle the job alone. The paper proves the AI is still a toddler who needs a dictionary and a little bit of human help to do its homework correctly.

1. Problem Statement

Word Sense Induction (WSI) is the task of automatically discovering the different senses of a word (lemma) from a corpus without relying on predefined sense inventories (like WordNet) or sense-labeled training data. While traditionally an alternative to Word Sense Disambiguation (WSD), the authors argue that WSI remains unsolved in the era of Large Language Models (LLMs).

The paper identifies three critical methodological flaws in current WSI research:

Unrealistic Datasets: Standard evaluation datasets (SemEval 2010, 2013) artificially over-represent polysemous (multi-sense) words and ignore monosemous (single-sense) words, which are frequent in natural language.
Evaluation Bias: Metrics and datasets often fail to reflect natural corpus distributions, leading to systems that perform well on benchmarks but fail on real-world data.
LLM Limitations: Despite their semantic capabilities, current LLMs struggle with the specific task of clustering instances of a lemma into distinct senses, often failing to follow instructions or hallucinating.

2. Methodology

A. New Evaluation Framework (SemCor-WSI)

To address dataset bias, the authors constructed a new evaluation framework based on SemCor 3.0.

Natural Distribution: Unlike SemEval datasets, this framework respects the original corpus distribution of lemma frequencies and polysemy levels (average polysemy $\approx$ 2).
Composition: It includes both monosemous and polysemous lemmas across Nouns, Verbs, and Adjectives, with instance counts ranging from 2 to several hundred.
Baselines: Two simple baselines were established:
- 1cpl (One Cluster Per Lemma): Treats all instances of a word as a single sense.
- 1cpex (One Cluster Per Example): Treats every instance as its own unique sense.

B. Experimental Setup

The authors evaluated several approaches on the SemCor-WSI dataset:

Unsupervised Clustering: Clustering contextualized embeddings from Pre-trained Language Models (PLMs) like BERT (base/large) and models fine-tuned for the Word-in-Context (WiC) task (MirrorWiC, BERT-Wikt).
Clustering Algorithms: Agglomerative Clustering with Silhouette score (AGsilh) and X-Means.
Direct LLM Prompting: Asking LLMs (GPT-4o, Llama 3.1/3.3) to directly cluster instances of a lemma.
Data Augmentation:
- Unsupervised: Adding unlabeled examples from Wikibooks, LLM-generated sentences, or Wiktionary exemplars.
- Semi-supervised: Using Wiktionary for fine-tuning embeddings, as "must-link" constraints (forcing examples from the same Wiktionary sense into the same cluster), and defining the number of clusters.

3. Key Contributions

Critique of Current Evaluation: The paper demonstrates that standard WSI datasets (SemEval 10/13) are biased toward polysemy, causing state-of-the-art (SOTA) systems to overfit to unrealistic distributions.
The "1cpl" Baseline Dominance: The authors prove that on a natural distribution dataset, the simplest heuristic (clustering all instances of a lemma into one group) often outperforms complex unsupervised clustering methods.
LLM Performance Analysis: Direct prompting of LLMs for WSI yields poor results compared to traditional embedding clustering, with LLMs struggling to handle large batches of instances and often "refusing" the task or generating irrelevant output.
Data Augmentation Success: The paper shows that simple data augmentation (adding unlabeled corpus examples) significantly boosts performance, allowing unsupervised methods to surpass previous SOTA systems (PolyLM).
Lexicon-LLM Synergy: The most effective approach combines LLMs/embeddings with lexical resources (Wiktionary) via semi-supervised techniques (fine-tuning, must-link constraints, and cluster count definition).

4. Key Results

Baseline Superiority: On the SemCor-WSI test set, the 1cpl baseline achieved an F-B3 score of 73.6.
- The previous SOTA system (PolyLM-large) scored 73.0.
- Most unsupervised clustering methods (BERT, MirrorWiC) scored between 69.7 and 71.8, failing to beat the 1cpl baseline.
Part-of-Speech (POS) Variation:
- Nouns and Adjectives: The 1cpl baseline is the best performer.
- Verbs: Clustering methods perform better than 1cpl because verbs are more polysemous, but the gap is narrow.
LLM Prompting: Direct LLM prompting performed poorly (F-B3 $\approx$ 29–66 depending on the model and metric), often failing to generate valid cluster assignments.
Impact of Augmentation:
- Adding Wikibooks examples (10–150 per lemma) systematically improved results.
- Semi-supervised Wiktionary approach: By combining BERT-Wikt embeddings, Wiktionary-based must-link constraints, and Wiktionary-defined cluster counts, the authors achieved a new SOTA of 76.0 (F-B3), surpassing both the 1cpl baseline and PolyLM.
Metric Sensitivity: The authors highlight that different metrics (F-B3 vs. NMI) can lead to different conclusions about which system is "best," emphasizing the need for instance-based metrics like F-B3 in full-corpus scenarios.

5. Significance and Conclusion

The paper concludes that Word Sense Induction is not yet solved, particularly in realistic, full-corpus scenarios.

The "Polysemy Bias" Trap: Current SOTA systems are optimized for datasets that over-represent multi-sense words. When applied to natural data where many words are monosemous, these complex systems fail to outperform the trivial "one cluster per word" approach.
Role of Lexicons: Purely unsupervised methods struggle. The most promising path forward involves hybrid approaches that leverage existing lexical resources (like Wiktionary) to guide LLMs and embedding models, rather than relying solely on statistical patterns in raw text.
LLM Limitations: While LLMs possess rich lexical semantics, they are not yet effective at the specific mechanical task of unsupervised sense induction via prompting. Better articulation between LLM capabilities and structured lexicons is required.

Final Takeaway: To advance WSI, the community must move away from artificial benchmarks and adopt evaluation frameworks that reflect natural language distributions. Furthermore, future progress likely depends on integrating external lexical knowledge (lexicons) with the semantic capabilities of LLMs, rather than relying on LLMs alone.