💬 NLP

CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation

This paper introduces Concept-Aligned Embeddings (CALE), a novel approach that extends Word-in-Context fine-tuning to include inter-lemma scenarios via a new Concept Differentiation task and dataset, resulting in models that achieve state-of-the-art performance in lexical semantic tasks while improving the spatial organization of embeddings.

Original authors: Bastien Liétard, Gabriel Loiseau

Published 2026-01-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bastien Liétard, Gabriel Loiseau

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of books, and inside every book, words are like actors on a stage. Sometimes the same actor (a word) plays different roles (senses) depending on the scene (context). Other times, different actors (different words) play the exact same role.

For a long time, computer programs trying to understand language were like stage directors who only paid attention to one actor at a time. They could tell you if the word "bank" meant a place to put money or the side of a river, but they struggled to realize that "bank" and "financial institution" were actually playing the same role in the story.

This paper introduces a new way to teach computers to understand language called CALE (Concept-Aligned Embeddings). Here is how it works, broken down simply:

1. The Old Way vs. The New Way

The Old Way (Word-in-Context): Imagine a teacher asking a student, "Does the word 'bat' mean the animal or the sports equipment in these two sentences?" The student only compares the word to itself. This is like only looking at one actor's costume changes.
The New Way (Concept Differentiation): The authors say, "Let's look at the whole cast." They ask the computer: "Do these two words, even if they are spelled differently, mean the same thing in this specific scene?"
- Example: In one sentence, "The doctor diagnosed the patient." In another, "The doctor identified the illness."
- The old way might struggle to see that "diagnosed" and "identified" are doing the same job here. The new way is trained specifically to spot that connection.

2. The Training Camp (The Dataset)

To teach the computer this new skill, the authors built a special training camp called SPCD.

They took a massive collection of text (SemCor) where words were already tagged with their "meanings" (concepts).
They created millions of pairs of sentences. Some pairs used the same word with the same meaning (e.g., "bank" as money in both). Some used the same word with different meanings (e.g., "bank" as money vs. "bank" as a river).
Crucially, they also paired different words that meant the same thing (e.g., "bank" and "financial institution").
The computer's job was simple: Look at two word usages and say "Yes, they are the same concept" or "No, they are different."

3. The Result: CALE

After training on this "Concept Differentiation" task, the computer models became CALE.

The Magic: Before training, the computer's internal map of words was a bit messy. Words that sounded alike or looked alike were often grouped together, even if they meant different things.
The Shift: After training, the computer reorganized its map. Now, words that share the same concept (the idea behind them) are pulled close together, like magnets.
- If two words mean the same thing, they sit in the same neighborhood, even if they are spelled differently.
- If a word has two meanings (like "bat"), those two meanings are pushed far apart, so the computer doesn't get confused.

4. Did it Work?

The authors tested these new models against older, famous models (like XL-LEXEME) on several challenges:

The "Same Word" Test: Can it tell if "bat" means the animal in two different sentences? Yes, it was very good.
The "Different Word" Test: Can it tell that "fast" and "quick" mean the same thing in a specific sentence? Yes, it was better than the older models.
The "Time Travel" Test: Can it detect if a word's meaning has changed over time (e.g., how the word "mouse" used to mean only an animal, but now also means a computer device)? Yes, it performed as well as or better than the best existing tools.

5. The Big Picture

The most interesting finding is about the shape of the computer's mind.

Before training, the computer's mind was organized around words (lemmas). It thought, "This is the word 'run', so it belongs here."
After training, the computer's mind became organized around ideas (concepts). It thinks, "This is the idea of 'moving quickly', so these words belong here."

The authors conclude that by teaching the computer to look at the big picture of meaning (concepts) rather than just the small picture of spelling (words), they created a more flexible and accurate tool for understanding language. They made the data and the new models available for others to use, hoping it helps future research into how computers understand the nuance of human speech.

Technical Summary: CALE – Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation

Problem Statement

Current research in computational lexical semantics relies heavily on contextualized language models (e.g., BERT, XLM-R) to study word meaning. However, standard pre-trained models are often used without fine-tuning specifically for lexical semantics due to a lack of annotated data. Existing fine-tuned approaches, such as those for the Word-in-Context (WiC) task, are limited because they only compare occurrences of the same lemma. Consequently, these models capture intra-word sense variation (polysemy) but fail to model inter-word semantic relations (synonymy) or the broader structure of lexical meaning where different words instantiate the same underlying concept.

The authors argue that to achieve a higher perspective, the focus must shift from "word senses" (conventionalized uses of a specific lemma) to "semantic concepts" (mental representations of categories, akin to Wikipedia entries or WordNet synsets). There is a need for a model that can generate multi-purpose vector representations where occurrences instantiating the same concept are similar, regardless of whether they share the same lemma.

Methodology

1. Task Definition: Concept Differentiation

The paper introduces Concept Differentiation, a binary classification task that generalizes WiC. Given two word occurrences, the model must determine if they instantiate the same semantic concept.

Scope: Unlike WiC, this task includes both intra-lemma pairs (same word, different senses) and inter-lemma pairs (different words, same or different concepts).
Goal: To train models that learn to compare usages in context, capturing subtle semantic distinctions or consistencies across the corpus.

2. Dataset: SPCD (SemCor Pairs for Concept Differentiation)

To support this task, the authors constructed a new dataset derived from SemCor (Miller, 1995), utilizing WordNet synsets as concept labels.

Construction: The dataset includes 70.3k occurrences for 1,902 target lemmas covering 5,899 concepts.
Pairing Strategy: Pairs are sampled to ensure a rigorous evaluation of generalization. The dataset is split by disjoint subsets of concepts and lemmas (5% for validation, 10% for test) to prevent models from memorizing word-concept associations.
Categories: Pairs are labeled based on four categories:
- Same Concept / Same Lemma (SC/SL)
- Same Concept / Different Lemma (SC/DL)
- Different Concept / Same Lemma (DC/SL)
- Different Concept / Different Lemma (DC/DL)
Balance: The test set contains 44k pairs with a label distribution of approximately 40% positive (same concept).

3. Model Architecture: CALE (Concept-Aligned Embeddings)

The authors propose CALE, a family of token-level models fine-tuned for Concept Differentiation.

Architecture: Based on a Siamese architecture inspired by SentenceBERT. It uses a pre-trained contextualized language model (e.g., XLM-R, ModernBERT) followed by a pooling layer to produce fixed-dimensional vectors.
Training Objective: A contrastive loss function is used to maximize cosine similarity for positive pairs (same concept) and minimize it for negative pairs (different concepts), enforcing a margin $m$ for dissimilar pairs.
Input Handling: To isolate the target word's meaning within a sentence, special delimiters <t> and </t> are placed around the target word. The model learns to base the contrast on the meaning of the delimited word, ensuring the embedding represents the specific target occurrence rather than the whole sentence.

Key Contributions

Concept Differentiation Task: A new supervised task that unifies the study of polysemy (within-lemma) and synonymy (inter-lemma) under a single framework of concept alignment.
SPCD Dataset: A rigorously constructed dataset with strict train/validation/test separation based on concepts and lemmas, enabling evaluation of generalization to unseen meanings and lexical items.
CALE Models: A set of fine-tuned models demonstrating that training on Concept Differentiation yields efficient, multi-purpose lexical representations.
Analysis of Embedding Space: Evidence that CALE fine-tuning shifts the representation space from being lemma-centric to concept-centric.

Experimental Results

The authors evaluated CALE models against baselines (XLM-R, XL-LEXEME) across three lexical semantic tasks:

1. Concept Differentiation (SPCD Test Set)

Performance: CALE models achieved the highest Balanced Accuracy (79.3%), outperforming the strong multilingual baseline XL-LEXEME (76.7%) and the raw XLM-R (70.9%).
Generalization: CALE models performed well on both Same-Lemma (SL) and Different-Lemma (DL) pairs, demonstrating the ability to handle both polysemy and synonymy.
Multilingual Transfer: Despite being fine-tuned only on English data, CALE models based on XLM-R performed competitively on non-English benchmarks, leveraging the cross-lingual capabilities of the base model.

2. Lexical Semantic Change Detection (LSCD)

Task: Detecting meaning shifts of a single word across time periods (Diachronic Word Usage Graphs).
Results: CALE models matched or slightly outperformed XL-LEXEME in 5 out of 6 languages (e.g., English, German, Swedish).
Implication: The broader Concept Differentiation objective did not degrade performance on the narrower, lemma-centric LSCD task; in fact, it appeared to enhance the detection of semantic change.

3. In-Context Lexical Similarity (CoSimLex)

Task: Predicting the similarity of two distinct words in specific contexts.
Results: CALE models achieved the highest correlation scores across all languages and subtasks (Contextual Changes and Similarity Ratings), consistently outperforming XLM-R and often matching or exceeding XL-LEXEME.
Significance: This confirms that CALE effectively captures inter-word semantic relations, a capability where WiC-trained models (like XL-LEXEME) were previously expected to struggle.

4. Spatial Organization Analysis

Concept-Centric Shift: Analysis of cosine distance distributions revealed that CALE successfully reverses the tendency of base models where polysemous pairs (DC/SL) are closer than synonymous pairs (SC/DL). In CALE, synonymous pairs are closer, indicating a shift to concept-centric organization.
WordNet Alignment: CALE embeddings showed the highest Spearman correlation (>0.50) with Wu-Palmer similarities in WordNet, indicating a stronger alignment with established conceptual hierarchies than other models.
Clustering: CALE models yielded the highest Silhouette scores, suggesting better conceptual clustering, though scores remained low overall due to the fine granularity of WordNet synsets.

Significance and Claims

The paper claims that Concept Differentiation serves as a promising, general-purpose training signal for lexical semantics. The primary significance lies in the demonstration that a single fine-tuning objective can produce representations that are effective for:

Multi-purpose applications: Handling both intra-lemma (sense) and inter-lemma (concept) tasks.
Cross-lingual transfer: Achieving strong performance on non-English tasks despite English-only fine-tuning.
Structural alignment: Organizing the embedding space to reflect the conceptual structure of resources like WordNet more accurately than previous state-of-the-art models.

The authors conclude that moving from a "word sense" focus to a "semantic concept" focus allows models to capture the broader structure of lexical meaning, making CALE a valuable tool for studies mapping word forms to concepts and various computational lexical semantic applications.

Limitations

The authors acknowledge several limitations:

Data Scarcity: Fine-tuning was performed exclusively on English data; robustness in low-resource or typologically distant languages remains unthoroughly evaluated.
Binary Simplification: The binary nature of the task may oversimplify semantic similarity, which is often graded.
Lack of Ontological Supervision: The models receive no direct supervision regarding ontological hierarchies (e.g., hypernymy) or fine-grained relations beyond the binary concept match.
Scaling: The study focused on architectural modifications rather than scaling model parameters or embedding dimensions.