💬 NLP

Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis

This paper investigates how large language models encode semantic relations across different scales by combining linear probing with mechanistic interpretability techniques, revealing that hierarchical relations exhibit directional asymmetry in their feature robustness, signals are most prominent in mid-layer MLP pathways, and causal manipulation via sparse autoencoders is more effective in larger models.

Original authors: Andor Diera, Ansgar Scherp

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Andor Diera, Ansgar Scherp

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart robot librarian (a Large Language Model, or LLM) who has read almost every book in the world. You want to know: Does this robot actually understand how words relate to each other, or is it just memorizing patterns like a parrot?

For example, does it know that "happy" and "sad" are opposites? Does it know that a "dog" is a type of "animal," but an "animal" isn't a specific type of "dog"?

This paper is like a team of detectives (the researchers) trying to peek inside the robot's brain to see how it stores these relationships. They used three robots of different sizes: a tiny one (Pythia), a medium one (GPT-2), and a huge one (Llama 3.1).

Here is what they found, explained with some fun analogies:

1. The "Middle Child" Problem

The researchers looked at the robot's brain layer by layer, like peeling an onion.

The Finding: The robot doesn't store the meaning of "dog vs. animal" in just one specific layer. Instead, the information is scattered all over the middle layers of the brain.
The Analogy: Imagine trying to find a specific recipe in a massive library. You don't find it on just one shelf; the ingredients are scattered across the middle aisles. You have to look at the whole middle section to get the full picture. Also, the robot gets better at this as it gets bigger (the 8-billion-parameter robot is much sharper than the tiny one).

2. The "Opposites" vs. "Synonyms" Mystery

The team tested four types of relationships:

Synonyms: Words that mean the same thing (Happy = Joyful).
Antonyms: Words that mean the opposite (Happy vs. Sad).
Hypernyms: General categories (Dog is a type of Animal).
Hyponyms: Specific examples (Dog is a Beagle).

The Results:

Antonyms (Opposites) are easy: The robot is great at spotting opposites. It's like the robot has a bright red "OPPOSITE" button that lights up clearly.
Synonyms (Same meaning) are hard: This is the robot's biggest struggle. It's hard to tell if two words are truly the same or just similar. The signal is very faint and blurry.
The "Distributional Paradox": The researchers found that if you just look at how close two words are in the robot's math space, "Happy" and "Sad" look very close because they often appear in similar sentences (e.g., "I feel happy/sad today"). It's like two people who always sit at the same table at a party; they look like friends, even if they are enemies. The robot needs special tools to realize they are actually opposites.

3. The "One-Way Street" (Hierarchy)

This is the most interesting part. The relationship between "Dog" and "Animal" is not a two-way street; it's a one-way street.

Hypernymy (Generalizing): "Dog" $\rightarrow$ "Animal." The robot is very good at this. It's easy for the robot to say, "Yes, a dog is an animal."
Hyponymy (Specificizing): "Animal" $\rightarrow$ "Dog." The robot struggles here. If you ask, "Is an animal a dog?" it gets confused.
The Analogy: Think of the robot's brain as a map. It has a very clear, wide highway going from "Specific" to "General" (Dog $\rightarrow$ Animal). But the road going the other way (Animal $\rightarrow$ Dog) is a narrow, bumpy dirt path that is easy to get lost on. The robot is biased toward generalizing, not specializing.

4. The "Brain Surgery" Experiment

To prove they weren't just guessing, the researchers performed "brain surgery" on the robot using a tool called Sparse Autoencoders (SAE).

What they did: They identified tiny, specific "neurons" (features) responsible for these relationships. Then, they either turned them off (ablation) or forced them on in neutral sentences (patching).
The Finding:
- On the tiny robot, the surgery barely worked. The robot was too small to have clear, isolated "relationship neurons."
- On the huge robot (Llama 3.1), the surgery was magic. When they turned off the "Hyponymy" neurons, the robot instantly forgot that a dog is a beagle. When they turned on the "Antonym" neurons in a neutral sentence, the robot suddenly started seeing opposites where there were none.
The Takeaway: The big robot has distinct, physical "switches" for these concepts. The small robot is more like a muddy soup where everything is mixed together.

5. The "Redundancy" of Generalization

Here is a weird quirk they found:

Generalizing (Hypernymy) is redundant. The robot has many different neurons that can all say "Dog is an Animal." If you cut out one, the others take over. It's like having 100 people in a room all shouting the same fact; if you silence one, the fact is still heard.
Specificizing (Hyponymy) is fragile. The robot relies on a very small, specific group of neurons to say "Animal is a Dog." If you cut those out, the robot forgets the relationship immediately. It's like a house of cards; remove one card, and the whole structure collapses.

Summary for the Everyday Person

This paper tells us that Large Language Models do understand semantic relationships, but they do it in a messy, uneven way:

Bigger is better: The huge models have much clearer "neural switches" for these concepts than the small ones.
Opposites are easy; Same-ness is hard: It's easier for AI to spot differences than similarities.
Generalizing is easy; Specificizing is hard: AI is great at saying "A dog is an animal," but bad at saying "An animal is a dog."
It's not just math: The relationships aren't just about how close words are in a list; they are encoded in specific, active parts of the brain that can be turned on and off.

The researchers have provided a "map" and a "screwdriver" (the framework) for future engineers to tweak these brains, potentially making AI better at tasks like organizing taxonomies, understanding logic, or spotting subtle differences in meaning.

1. Problem Statement

While Large Language Models (LLMs) demonstrate impressive capabilities in natural language processing, it remains unclear whether they possess structured conceptual understanding of semantic relationships or merely rely on surface-level statistical co-occurrences. Existing interpretability research has largely focused on identifying individual concepts (e.g., "cat," "dog") but has not systematically investigated how semantic relations (synonymy, antonymy, hypernymy, hyponymy) are encoded, localized, and causally manipulated within model architectures.

The authors aim to answer three core questions:

Do semantic relation signals exist in LLMs?
Where are these signals localized across layers and components (Attention vs. MLP)?
How strongly can these signals be manipulated (causality) using sparse features?

2. Methodology

The study employs a multi-faceted approach combining linear probing, Sparse Autoencoders (SAEs), and activation patching across three models of varying scales:

Models: Pythia-70M (6 layers), GPT-2-124M (12 layers), and Llama 3.1 8B (32 layers).
Dataset: Constructed from WordNet 3.0, containing 1,000 pairs each for Synonymy, Antonymy, Hypernymy, and Hyponymy, plus 1,000 random pairs. Pairs were augmented with neutral contextual prompts.
Probing Strategy:
- Linear probes (multinomial logistic regression) were trained on three activation streams per layer: Attention output, MLP output, and Post-residual stream.
- Metrics included mean accuracy, peak accuracy, peak depth, and the Center of Mass (CoM) to determine signal distribution.
Mechanistic Interpretability (SAE & Patching):
- Pre-trained Sparse Autoencoders (SAEs) were used to disentangle features.
- Activation Patching: The authors identified top- $k$ $k$ SAE features associated with specific relations (ranked by probe weights) and performed two causal interventions:
  - Necessity: Ablating (zeroing) features in relation-bearing inputs to see if the signal collapses.
  - Sufficiency: Injecting features into neutral/random inputs to see if a relation signal emerges.
- Metrics: Semantic Logit Difference ( $LD_{sem}$ ), Flip Rate ( $\Delta FR$ ), and Drop Rate ($DR$).

3. Key Contributions

Systematic Scale Analysis: A comparative study of semantic relation encoding from 70M to 8B parameters, revealing consistent difficulty ordering and scale-dependent improvements.
Integrated Framework: A novel pipeline combining linear probing with SAE-guided causal interventions to move beyond correlation to probe-level causality.
Localization Insights: Identification that relational signals are diffuse (peaking in mid-layers) rather than localized to specific "relation layers," and are strongest in post-residual/MLP pathways rather than attention mechanisms.
Asymmetry Discovery: Uncovering a fundamental directional asymmetry in hierarchical relations (Hypernymy vs. Hyponymy) and a difficulty hierarchy across relation types.

4. Key Results

A. Probing Results (Localization & Difficulty)

Difficulty Ordering: Consistent across all models: Antonymy (easiest) > Hyponymy > Hypernymy > Synonymy (hardest).
Scale Effect: Accuracy improves significantly with model size (e.g., Antonymy approaches ceiling on Llama 3.1; Synonymy remains the most difficult even at 8B).
Localization:
- No single "relation layer" exists. Signals are diffuse, with Centers of Mass (CoM) consistently near the mid-layers (approx. 0.5 normalized depth).
- Component Analysis: The Post-residual stream carries the strongest signal. The MLP component generally holds more recoverable evidence than Attention, though Attention shows relation-specific advantages in smaller models.

B. Relation-Specific Findings

Antonymy vs. Synonymy:
- Despite high cosine similarity between antonyms in embedding space (the "distributional paradox"), linear probes can easily separate them. This suggests relations are encoded in specific internal activations rather than global vector similarity.
- Synonymy is the hardest to decode and least affected by patching, indicating a highly diffuse encoding.
Hypernymy vs. Hyponymy (Directional Bias):
- Hypernymy (General $\to$ Specific, e.g., Dog $\to$ Animal) is redundantly encoded: it is easy to strengthen (high sufficiency) but hard to suppress (low necessity).
- Hyponymy (Specific $\to$ General, e.g., Animal $\to$ Dog) relies on compact features: it is easily disrupted by ablation (high necessity) and collapses readily.
- Order Sensitivity: Reversing word order (e.g., "Animal relates to Dog" vs. "Dog relates to Animal") causes a significant accuracy drop for Hyponymy ( $\Delta \approx 0.24$ ) but a minimal drop for Hypernymy ( $\Delta \approx 0.08$ ), confirming a bias toward generalization.

C. Causal Interventions (SAE Patching)

Capacity Dependence: Strong, reliable causal effects (shifting probe predictions) were observed only in Llama 3.1. Smaller models (Pythia, GPT-2) showed weak or unstable shifts, suggesting that sparse feature manipulation requires sufficient model capacity.
Feature Specificity: Interventions using top- $k$ SAE features significantly outperformed random feature interventions, confirming that specific features drive relation signals.
Asymmetry Confirmation: The patching results mirrored the probing asymmetry: Hypernymy showed high sufficiency but low necessity, while Hyponymy showed high necessity.

D. Contextual Robustness

Relation signals are largely prompt-invariant. Changing prompt templates degraded mean accuracy (readout calibration) but left peak accuracy and causal feature availability largely intact.

5. Significance and Implications

Beyond Black-Box Evaluation: The study demonstrates that LLMs do encode structured semantic relations, but these are not simple vector similarities. They are distributed, component-specific, and often asymmetric.
Mechanistic Understanding: The findings challenge the idea of "relation layers," proposing instead a diffuse, mid-stack representation that is heavily reliant on MLP pathways.
Practical Applications: Understanding that Hypernymy is robust (redundant) while Hyponymy is fragile (compact) can guide targeted interventions for tasks like ontology completion, taxonomy-aware retrieval, and textual entailment.
Methodological Advancement: The paper provides a reproducible framework for linking sparse feature analysis (SAEs) with causal evidence (patching), setting a standard for future mechanistic interpretability research.

In conclusion, the authors establish that while LLMs capture semantic relations, the encoding is scale-dependent, diffusely distributed, and structurally biased (favoring generalization over specification), with causal manipulability emerging only in larger models.