💬 NLP

Priors Persist Through Suppression: A Stroop Paradigm for Lexical Override

This paper demonstrates that in language models, familiar lexical priors persist through explicit override rules rather than being replaced, causing Stroop-like interference that originates from and is mechanistically localized to the specific activation pathways binding definition targets to query words.

Original authors: Han-yu Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han-yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, well-read robot a new rule for a game. You tell it: "In this game, the word 'doctor' means 'forest'."

You then ask the robot: "Using only this rule, what word is related to 'doctor'?"

Ideally, the robot should say "forest." But because the robot has read millions of books before, it has a deep, automatic habit: when it hears "doctor," it immediately thinks of "hospital."

This paper is a study of what happens when the robot tries to follow your new rule while fighting its old habit. The researchers call this a "Stroop-style" test (named after a famous psychology experiment where people struggle to name the color of a word when the word itself spells a different color, like the word "RED" written in blue ink).

Here is the breakdown of their findings, using simple analogies:

1. The Old Habit Doesn't Just Disappear

The researchers found that when you tell the robot to ignore its old knowledge, it doesn't simply delete the old meaning and replace it with the new one. Instead, the old meaning lingers in the background.

The Analogy: Imagine you are trying to drive a car on a new road, but your muscle memory keeps trying to steer you toward your old home. You can force the car to stay on the new road, but your hands are still twitching toward the old direction. The stronger your old habit (the "lexical prior"), the harder it is to stay on the new road, even if you are trying very hard.
The Finding: The more strongly the robot usually associates "doctor" with "hospital," the more it struggles to say "forest," even when you explicitly told it the new rule.

2. The "Glitch" Happens in a Specific Spot

The researchers didn't just watch the robot's answers; they looked inside its "brain" (its internal computer code) to see where the struggle happened. They used a technique called "activation patching," which is like temporarily swapping a piece of a robot's brain with a clean version from a different robot to see if it fixes the error.

The Analogy: Think of the robot's brain as a factory assembly line. The researchers found a specific three-person team on the line that is responsible for this specific task:
1. The person who hears the new rule ("Doctor means...").
2. The person who holds the new meaning ("...Forest").
3. The person who hears the question word ("...Doctor?").
The Finding: When the researchers "patched" (replaced) the work of these three specific people with a clean version, the robot suddenly got the answer right. If they only fixed two of them, or if they swapped the "Forest" person with someone who was holding a different word, the robot still failed. This proves that the robot needs to bind the specific new meaning to the specific word to make it work.

3. The "Suppression" vs. "Preservation" Mystery

One of the most interesting discoveries was how the robot fixes the mistake. The researchers expected the robot to simply "turn down the volume" on the wrong answer ("hospital").

The Analogy: Imagine a radio with two stations playing at once: the old habit (Hospital) and the new rule (Forest).
- What they thought would happen: The robot would just turn the volume down on the "Hospital" station.
- What actually happened: The robot does turn down the volume on "Hospital," but it does that for almost any reason (even if the rule is slightly messed up). The real magic happens because the robot turns the volume UP on the correct answer ("Forest") specifically when the new rule is perfectly clear.
The Finding: The robot's ability to follow the new rule depends on preserving the new meaning, not just suppressing the old one. If the "new meaning" part of the brain gets corrupted, the robot collapses, even if the "old meaning" is still being suppressed.

4. It Happens in All Kinds of Robots

The researchers tested this on 11 different types of AI models (ranging from small to large, and from different companies).

The Finding: No matter how big or smart the robot was, or how they phrased the rule (as a game, a legal document, or a technical manual), the old habit always interfered. The stronger the robot's original habit, the more it struggled to follow the new rule.

Summary

The paper concludes that when an AI is asked to change the meaning of a word, it doesn't simply overwrite its old knowledge. Instead, it's a tug-of-war. The old habit keeps pulling, and the new rule has to fight to keep the answer correct. The AI succeeds not by erasing the past, but by successfully "binding" the new meaning to the word in a specific part of its brain, while simultaneously trying to quiet the old habit.

In short: You can tell a robot a new rule, but its old habits are still whispering in its ear, and the robot has to work hard to ignore them.

Technical Summary: Priors Persist Through Suppression

Problem Statement
Language models (LMs) are frequently required to follow local definitions that temporarily override their pretrained lexical priors (e.g., a glossary defining "port" as "socket"). While these overrides often succeed, they are not clean replacements of meaning. Instead, the pretrained lexical prior continues to operate, creating a residual bias toward the familiar meaning even when a local rule is applied. This "override failure" manifests as a quiet interference in next-token probabilities, which is critical in domains requiring strict adherence to document-specific definitions, such as legal, medical, or technical documentation. The paper posits that contextual override is a competition where the prior persists rather than being erased, and it seeks to quantify this interference and locate its mechanistic source.

Methodology
The authors introduce a Stroop-style paradigm to isolate lexical-prior interference at the item level.

Stimuli: For each item, a query word $w$ is remapped to a contextual target $t$ (e.g., "doctor" means "forest"). A lexical-prior distractor $d$ (e.g., "hospital") competes with the target.
Control: A matched neutral control replaces the query word $w$ with a semantically weak word $n$ (e.g., "class") while keeping the target $t$ and distractor $d$ identical. This design isolates the interference caused specifically by the query word's prior.
Behavioral Metric: The interference effect ( $\Delta$ ) is measured as the difference in relative target preference between the neutral and conflict prompts: $\Delta = E_n[S(x_{neutral})] - S(x_{conflict})$ , where $S(x) = \log P(t|x) - \log P(d|x)$ .
Scope: Behavioral analysis covers 11 open-weight models (Qwen, Gemma, OLMo, Mistral families) ranging from 1B to 9B parameters. Mechanistic analysis focuses on a tractable subset of 5 aligned models using activation patching via TransformerLens.

Key Results

Behavioral Persistence of Priors:
- Across all 11 models and four conflict families (antonym, arbitrary, polysemy, domain-definition), the interference effect is positive, indicating that lexical conflict reduces the preference for the contextual target.
- The effect is most pronounced in document-relevant settings (arbitrary, polysemy, domain-definition), which are 2–4 $\times$ larger than the canonical antonym remapping.
- Lexical-Prior Strength as a Predictor: A controlled regression analysis reveals that the strength of the distractor's ordinary lexical prior (measured as $\log P(d) - \log P(t)$ in a neutral context) positively predicts the magnitude of interference ( $\beta \approx 0.114, p < 10^{-9}$ ). This holds even after controlling for answer prior, token frequency, tokenization, prompt style, and model scale. Priors "leak" through override in proportion to their strength.
Mechanistic Localization via Activation Patching:
- Source Triplet: Activation patching identifies a specific source-position triplet—(definition subject, definition target, query word)—that nearly fully recovers the conflict effect when patched from a clean run into a corrupted run (aggregate recovery $R \in [0.92, 1.06]$ ). This triplet outperforms any source-position pair.
- Binding vs. Identity: A "definition-target swap" control (keeping subject and query matched but swapping the target activation with a donor item) causes recovery to collapse. This confirms the triplet performs binding (associating the specific word with its new value) rather than simple identity matching.
- Logit Decomposition: The recovery of the target-distractor margin is driven by target preservation, not distractor suppression.
  - Under matched binding, the target logit is preserved while the distractor logit drops.
  - Under corrupted binding (swap or item-mismatched controls), the distractor logit drops similarly to the matched condition, but the target logit collapses.
  - This indicates that distractor suppression is a broad effect of source-position perturbation, whereas the specific contribution of binding is the preservation of the contextual target.

Significance and Claims
The paper claims a convergence between behavioral and mechanistic evidence on a single "lexical-prior channel":

Behaviorally: The strength of the lexical prior predicts which items resist override.
Mechanistically: The same channel (specifically the definition-target position within the source triplet) is where the causal effect of override repair lands.

The authors argue that lexical override is not a clean substitution but a partial competition where the prior remains active. The study provides a controlled paradigm to measure this interference and a mechanistic explanation for how models bind new definitions to existing tokens, distinguishing between the suppression of competing priors and the preservation of the intended contextual meaning.

Limitations
The authors note that the paradigm uses synthetic, short-form stimuli and English tokens. The mechanistic findings are established on antonym remapping within game-rule wrappers on smaller models (1B–2B). It remains an open question whether the specific source-triplet structure and the functional partition between target preservation and distractor suppression generalize to larger models (7B+), other conflict families, and complex, long-form document contexts.