💬 NLP

A graph-based analysis of semantic types and coercion in contextualized word embeddings

This paper proposes a graph-based analysis using Neighbor Type Probability and Entropy metrics on BERT and sense-enhanced embeddings to demonstrate that semantic type information and coercion phenomena are better captured in sense-enhanced models, enabling the distinction between matching and mismatching noun-context pairs.

Original authors: Long Chen, Deniz Ekin Yavas

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Long Chen, Deniz Ekin Yavas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, invisible library where every book represents a single word used in a sentence. In this library, books are grouped together on shelves based on how similar they feel to one another. This paper is like a map-making expedition into that library to see how well the shelves are organized.

The researchers wanted to answer a specific question: Do these digital "shelves" (called word embeddings) know the difference between what a word is and what a word is doing in a specific sentence?

Here is a breakdown of their journey using simple analogies:

1. The Problem: The "Pizza" Paradox

Think of the word "pizza."

What it is (Lexical Type): It's food. It belongs in the "Food" section of the library.
What it's doing (Contextual Type):
- In the sentence "I am eating a delicious pizza," the pizza is acting like food. This is a Match.
- In the sentence "I finished the pizza," the word "finished" usually expects an activity (like a race or a movie), not a slice of cheese. Here, the pizza is being forced to act like an activity. This is called Coercion. It's like asking a fish to climb a tree; the fish is still a fish, but the context is treating it like a climber.

The researchers wanted to see if the computer's "library" could tell the difference between a normal pizza sentence and a "coerced" pizza sentence just by looking at which other books were sitting on the same shelf.

2. The Experiment: Building Two Maps

To test this, they took 10 different categories of words (like animal, food, human, location) and found 20 sentences for each. They then built two different "maps" (graphs) of these sentences:

Map A (The Standard Map): Built using a standard AI model called BERT. This model is like a general librarian who knows a lot about words but might get distracted by how often words appear together (e.g., "student" and "school" appear together a lot, even if they aren't the same type of thing).
Map B (The Enhanced Map): Built using a Sense-Enhanced model. This is like a librarian who has a special dictionary (WordNet) that explicitly tells them the "job title" or "category" of every word.

They also created a trick: they covered the main word in the sentence with a "mask" (like a blank space) to see if the map could figure out the word's role just from the surrounding context, ignoring what the word actually is.

3. The Tools: Measuring the Neighborhood

Once the maps were built, they used two rulers to measure the shelves:

Ruler 1: The "Neighbor Match" (NTP): If you pick a word, how many of its neighbors on the shelf are from the same category?
- Example: If you pick "dog," do most of its neighbors also say "cat" or "horse" (Animal)? Or do they say "leash" or "park" (Location)?
Ruler 2: The "Chaos Meter" (NTE): How mixed up is the shelf?
- Low Chaos: The shelf is full of only "Animals."
- High Chaos: The shelf has "Animals," "Cars," "Clouds," and "Shoes" all mixed together.

4. What They Found

The results were like discovering that one librarian was much better at organizing the library than the other.

The Enhanced Map (Sense-Enhanced) was the best organizer.
When they looked at the "Match" sentences (where the word fits perfectly), the Enhanced Map put words with the same type right next to each other. For example, "students" were surrounded by other "humans," not by "classrooms." The standard map (BERT) got confused by how often words appeared together, mixing types up.
The "Coercion" Mystery:
When they looked at the "Coercion" sentences (where the word is forced into a new role, like the "stadium" roaring like a "human"):
- On the Enhanced Map, the word "stadium" still sat mostly with other "locations." It didn't magically jump to the "human" shelf. This suggests the computer still knows what the word is, even when the sentence treats it differently.
- However, when they used the Masked version (hiding the word), the map started to show the contextual role. The neighbors became more diverse, showing that the system could sense the "activity" the context was demanding, even if it wasn't perfect.
The "Unrestricted" Sentences:
For sentences where the context didn't care what the word was (e.g., "You need a [linguist]" could be replaced by almost anything), the "Chaos Meter" (NTE) went way up. The shelves were a mess of different types, which is exactly what you'd expect if the context doesn't restrict the word.

5. The Bottom Line

The paper concludes that:

Specialized Librarians Win: The "Sense-Enhanced" model organizes words by their true semantic type much better than the standard model.
Context Leaves a Trace: Even when a word is "coerced" into a new role, the graph shows a mix. The word keeps its original identity (it stays on the "Location" shelf), but the context adds a layer of diversity to its neighbors.
Detecting the Weirdness: By measuring how "mixed up" the neighbors are (Entropy), you can tell if a sentence is a normal match, a weird coercion, or a vague sentence.

In short, the researchers built a map of word relationships and proved that with the right tools, we can see exactly how computers understand the difference between what a word is and what a sentence wants it to be.

Technical Summary: A Graph-Based Analysis of Semantic Types and Coercion in Contextualized Word Embeddings

Problem Statement
Semantic type mismatch between a noun and its context is central to the linguistic phenomenon of coercion, where a noun's lexical semantic type conflicts with the type required by the surrounding context (e.g., "finish the pizza," where pizza is a food but finish typically selects for an activity). While theoretical frameworks like Generative Lexicon and Type Compositional Logic have proposed mechanisms to resolve these mismatches, computational work directly operationalizing these theories remains scarce. Existing studies on pre-trained language models (PLMs) largely focus on word prediction tasks to detect covert events in coercion, often yielding inconsistent results. This paper investigates a different question: How is information regarding semantic types reflected in the contextualized word embeddings of noun instances, specifically distinguishing between typical predication (matching) and coercion (mismatch)?

Methodology
The authors propose a graph-based analysis of contextualized word embeddings to examine the distribution of semantic types in the neighborhood of noun instances.

Dataset Construction:
- Semantic Types: Ten semantic types were selected: animal, artifact, activity, food, human, info, location, mood, process, state. These form a conceptual hierarchy compatible with WordNet supersenses.
- Instances: 20 sentences per noun were extracted from BookCorpus for 5–10 frequent nouns per type.
- Annotation: Sentences were annotated for Lexical Type (lt) (the noun's inherent type) and Contextual Type (ct) (the type required by the context, inferred by masking the noun).
- Categories: Sentences were classified into four types:
  - MATCHING: $lt = ct$ (e.g., "eating a pizza").
  - COERCION: $lt \neq ct$ due to predicate selection (e.g., "the stadium was roaring").
  - OTHER_MISMATCH: $lt \neq ct$ due to metaphor or metonymy.
  - UNRESTRICTED: $ct = \emptyset$ (context imposes no specific type).
Embeddings and Graph Construction:
- Models: Two models were used: standard BERT (base, uncased) and sense-enhanced BERT (fine-tuned on SemCor with WordNet supersense labels).
- Masking: To isolate contextual information from lexical information, target words were replaced with [MASK] tokens.
- Graphs: Four directed graphs were constructed ( $G_b, G_s, G_{mb}, G_{ms}$ ) based on cosine similarity between embeddings (last 4 layers averaged). Edges connect the $k=10$ nearest neighbors of different target words to avoid lexical overlap.
Proposed Metrics:
- Neighbor Type Probability (NTP): Measures the proportion of a node's neighbors belonging to a specific semantic type.
  - NTMRL (Lexical): $NTP(lt, v)$
  - NTMRC (Contextual): $NTP(ct, v)$
- Neighbor Type Entropy (NTE): Measures the diversity of semantic types among neighbors. Low entropy indicates neighbors share the same type; high entropy indicates diverse types.

Key Results

Embedding Quality: Graphs constructed with sense-enhanced embeddings ( $G_s, G_{ms}$ ) reflect semantic type information significantly better than standard BERT graphs. For example, in MATCHING sentences, the average NTMRL for artifact and human types increased substantially on $G_s$ compared to $G_b$ .
Lexical vs. Contextual Reflection:
- Unmasked graphs ( $G_b, G_s$ ) primarily reflect lexical types. In COERCION sentences, neighbors still predominantly share the noun's lexical type rather than the contextual type, suggesting the embeddings retain the noun's inherent type even in mismatch contexts.
- Masked graphs ( $G_{mb}, G_{ms}$ ) show higher sensitivity to contextual types, though the signal remains partial (NTMRC < 0.5).
Distinguishing Sentence Types via Entropy (NTE):
- MATCHING sentences exhibit low NTE (neighbors share the same type).
- UNRESTRICTED sentences exhibit the highest NTE (neighbors are highly diverse).
- COERCION sentences show intermediate behavior. On masked graphs ( $G_{mb}, G_{ms}$ ), COERCION sentences have significantly lower NTE than UNRESTRICTED sentences but higher than MATCHING sentences. This suggests that while coercion instances do not fully shift to the contextual type, they exhibit more type diversity than standard matching instances.
Type Hierarchy: The NTP analysis on $G_s$ revealed taxonomic proximities (e.g., process, state, and mood forming a cluster; info close to mood), allowing the induction of a new semantic type hierarchy.

Significance and Claims
The paper claims to be the first study to focus on semantic type relations and coercion at the representation level of contextualized embeddings, rather than relying solely on word prediction tasks.

Methodological Contribution: The proposed graph-based metrics (NTP and NTE) effectively distinguish between matching, coercion, and unrestricted sentences.
Theoretical Insight: The results suggest that in coercion, the lexical type is not completely overwritten by the context; rather, the graph structure reflects a balance where the lexical type remains dominant in unmasked embeddings, while masked embeddings partially capture the contextual shift.
Future Potential: The authors modestly suggest that with larger datasets, their method could automatically induce semantic type hierarchies and consistently detect mismatch sentences. They also note the potential to further explore the distinction between coercion and other mismatch mechanisms (metaphor/metonymy) given sufficient data.

The study concludes that sense-enhanced embeddings provide a richer representation of semantic types, and the proposed graph-based analysis offers a viable tool for investigating how lexical and contextual information interact within PLMs.