💬 NLP

On the Persistent Effects of Lexicality in Large Language Mod

This paper investigates how lexical overlap consistently influences representations across the depth and various architectures of large language models, revealing a transitional mid-depth regime where both lexical and semantic signals degrade, and demonstrating the resulting negative impact on downstream tasks like summarization and model editing.

Original authors: Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A. B. Siddique, Hassan Sajjad

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hammad Rizwan, Muhammad Umair Haider, Nishant Subramani, Mona T. Diab, A. B. Siddique, Hassan Sajjad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a giant, multi-story library where every floor represents a different "layer" of thinking. As a sentence travels from the ground floor (the beginning) to the top floor (the end), the library is supposed to transform the raw words into deep, abstract meaning.

The common belief was that as you go up the floors, the library forgets the specific words used and focuses entirely on the meaning. However, this paper argues that the library never actually forgets the words. In fact, the words keep haunting the meaning all the way to the top.

Here is a breakdown of the paper's findings using simple analogies:

1. The "Word Shadow" Problem

The researchers found that even in the deepest, most "smart" parts of the model, the representation of a sentence is still heavily influenced by the specific words it contains, not just what it means.

The Analogy: Imagine you are trying to find a friend in a crowd. You describe them by their personality (semantic content). But the model is like a security guard who is so distracted by the color of their shirt (lexical overlap) that they confuse two people wearing the same shirt, even if one is your friend and the other is a stranger.
The Finding: If you take a sentence and change its meaning but keep many of the same words, the model thinks the two sentences are very similar. It fails to see that the meaning has changed because it is too focused on the "word shadow."

2. The "Middle Floor" Trap

The paper discovered a strange phenomenon happening in the middle of the library (the middle layers of the model).

The Analogy: Think of the model's processing as a journey through a tunnel.
- Bottom: You see the raw bricks (the words).
- Top: You see the finished sculpture (the meaning).
- Middle: There is a dark, foggy valley. Here, the model has lost the clear view of the bricks, but it hasn't yet built the clear sculpture. It's a "no-man's-land" where the model is actually worst at understanding both the words and the meaning.
The Finding: In this middle section, the model struggles to identify the original words and it struggles to understand the meaning. It's a transitional zone where the information is compressed and messy.

3. Training Doesn't Fix the "Word Shadow"

The researchers tested models that were specifically trained to be better at understanding meaning (like models used for search engines or chatbots).

The Analogy: It's like hiring a new librarian who is an expert at finding books by their plot summary. You'd expect them to ignore the cover art. But the paper shows that even these expert librarians still get tricked if two books have the same title words, even if the stories are completely different.
The Finding: No matter how much you train the model to understand semantics, the "word shadow" effect persists. The model still relies on surface-level word matches as a shortcut.

4. Real-World Consequences

The paper shows how this "word shadow" breaks real-world tools that rely on these models.

Summarization (The "Copy-Paste" Trap):
- The Scenario: You ask an AI to summarize a news article.
- The Glitch: The AI might generate a summary that is nonsense but uses the exact same words as the original article.
- The Result: Current tools that grade summaries often give high scores to these nonsense summaries because they look so similar to the original text (high lexical overlap), even though they convey no actual meaning. The model is rewarding "word matching" over "meaning matching."
Model Editing (The "Collateral Damage" Trap):
- The Scenario: You want to update the model to learn a new fact (e.g., "The capital of Country X is City Y").
- The Glitch: Because the model is obsessed with word patterns, updating it for Country X accidentally changes how it answers questions about Country Z, if Country Z's name sounds or looks similar to Country X.
- The Result: The update "leaks" to other topics just because they share similar words, breaking the model's ability to keep facts separate.

Summary

The paper concludes that we cannot assume Large Language Models have successfully moved from "thinking about words" to "thinking about meaning." The words are still there, pulling the strings, even in the deepest layers.

The Takeaway: If you build a system that relies on these models to understand meaning (like a search engine or a fact-checker), you must be very careful. The system might think two things are the same just because they share a few words, even if their meanings are totally different. We need to test these models with "tricky" questions that have similar words but different meanings to ensure they are actually understanding us.

Technical Summary: On the Persistent Effects of Lexicality in Large Language Models

Problem Statement
Large Language Models (LLMs) are increasingly utilized as general-purpose embedding engines for retrieval, clustering, and evaluation. A prevailing assumption in the field is that as representations progress through model depth, they transition from local lexical structures to abstract, sequence-level semantics. However, this paper argues that this assumption is fragile. Empirical evidence suggests that lexical overlap often persists in embedding geometries, causing models to conflate surface-form similarity with semantic equivalence. This "lexical influence" degrades the performance of downstream tasks that rely on meaning-driven representations, such as document retrieval and model editing, by causing models to prioritize shared tokens over semantic content. The authors note that the relationship between this lexical influence and semantic content, and its propagation through model depth and training regimes, remains poorly understood.

Methodology
The authors investigate the evolution of lexical and semantic representations across model depth using a multi-faceted approach involving adversarial stress tests, layer-wise probing, and information-theoretic analysis.

Adversarial Stress Tests: The study employs triplet-based stress tests using the CounterFact and SugarCrepe++ (SCPP) datasets. Each triplet consists of an anchor ( $a$ ), a meaning-preserving paraphrase ( $p$ ), and a lexical distractor ( $d$ ) that shares significant token overlap with the anchor but differs semantically. Success is defined as the model embedding the anchor closer to the paraphrase than to the distractor. This metric quantifies the failure rate driven by lexical overlap.
Layer-wise Probing:
- Lexical Decodability: Linear probes are trained on hidden states to predict the input token identity. This measures the linear accessibility of surface-form information at each layer.
- Semantic Fidelity: Sentence embeddings from each layer are evaluated on the Massive Text Embedding Benchmark (MTEB) using standard lightweight heads to assess semantic performance without task-specific fine-tuning of the backbone.
Information-Theoretic Analysis: The authors analyze prompt entropy across layers to characterize information compression and decompression dynamics, relating these shifts to the observed lexical and semantic behaviors.
Downstream Case Studies: The impact of lexical influence is examined in two practical scenarios: abstractive summarization evaluation (using BERTScore and BARTScore) and factual model editing (using AlphaEdit).

Key Results

Persistence of Lexical Influence: Lexical influence is not confined to shallow layers; it persists across the entire depth of models, including those trained specifically for semantic similarity (embedding models). While instruction tuning and metric learning improve overall embedding quality, they do not eliminate the susceptibility to lexical overlap adversaries.
The Mid-Depth Valley: A consistent "valley" is observed in the intermediate layers (roughly 40–60% of model depth). In this region:
- Lexical Decodability is at its lowest (token identity is least linearly recoverable).
- Semantic Fidelity stagnates or degrades across classification, retrieval, and semantic textual similarity tasks.
- Lexical Failure Rates peak, indicating that representations are poor for both surface form and meaning.
Non-Monotonic Progression: The transition from lexical to semantic representations is not a smooth, monotonic progression. Token identity decodability fluctuates, and semantic performance recovers only in the later layers, often after the low-entropy bottleneck.
Information-Theoretic Alignment: The mid-depth valley aligns with a low-entropy bottleneck and the immediate re-expansion phase. Contrary to some prior hypotheses that the low-entropy bottleneck hosts the strongest representations, the authors find that semantic performance is weakest here. Stronger, meaning-sensitive geometry emerges later during the re-expansion phase.
Downstream Implications:
- Summarization: Lexical overlap causes evaluation metrics like BERTScore to incorrectly rank non-summaries (distractors) higher than gold summaries, even when the distractors lack semantic coherence.
- Model Editing: Updates targeted at specific facts generalize disproportionately to lexically similar but semantically unrelated prompts. This "lexical skew" compromises the locality of edits, causing unintended changes in model behavior for distractors sharing surface tokens with the edit prompt.

Significance and Claims
The paper claims that lexical influence is a fundamental and persistent property of LLM representations that survives various training regimes, including pretraining, instruction tuning, and contrastive learning. The authors argue that the standard heuristic of using final-layer embeddings or assuming a clean lexical-to-semantic transition is insufficient.

The primary significance lies in demonstrating that:

Lexical bias is robust: It persists even in models explicitly trained to minimize it, suggesting that current training paradigms reduce but do not eliminate lexical shortcuts.
Mid-layer representations are transitional: The intermediate layers represent a regime where both lexical and semantic signals are weak, challenging the view that intermediate layers are uniformly optimal for feature extraction.
Evaluation and Editing are vulnerable: Practical applications relying on embedding geometry are systematically biased by surface-form overlap, leading to miscalibrated similarity scores and compromised model editing locality.

The authors conclude that lexically controlled stress tests should become a standard diagnostic for representation learning, automatic metrics, and model interventions to ensure robustness against overlap-driven semantic failures. They propose future work to dissect the mid-depth transition regime using nonlinear transformations to understand the structure formed there and prevent overlap-driven errors.

1. The "Word Shadow" Problem

2. The "Middle Floor" Trap

3. Training Doesn't Fix the "Word Shadow"

4. Real-World Consequences

Summary

More like this