💬 NLP

Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence

By leveraging sparse autoencoders to identify and intervene upon semantically unrelated yet interfering polysemantic features in small models, this study demonstrates that such interference patterns generalize across scale and architecture, enabling predictable behavioral control in larger black-box language models and revealing a convergent, higher-order organization of internal representations.

Original authors: Bofan Gong, Shiyang Lai, James Evans, Dawn Song

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bofan Gong, Shiyang Lai, James Evans, Dawn Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Swiss Army Knife" Problem

Imagine you have a giant Swiss Army knife. It has a blade, a screwdriver, a corkscrew, and a toothpick. Now, imagine that in a computer brain (a Large Language Model or LLM), the "blades" are the neurons.

In a perfect world, one neuron would be just a "blade," another just a "screwdriver." But in reality, these models are so efficient that they cram many tools into a single neuron. One neuron might act as a "blade" when you talk about cutting, but also act as a "screwdriver" when you talk about fixing a car.

This is called Polysemanticity. It makes the model smart and efficient, but it also makes it messy. Because one neuron holds two different ideas, changing the "blade" part accidentally twists the "screwdriver" part.

The Discovery: The "Secret Knock"

The researchers asked a scary question: If we tap on the "blade" part of a neuron, can we accidentally make the model talk about "screwdrivers," even if we never asked for it?

They found that yes, we can.

They discovered that in these computer brains, two completely unrelated ideas (like "Beethoven" and "Frustration") can be secretly linked inside the same neuron. Even though they seem totally different to us, the model's internal wiring treats them as neighbors.

The Analogy:
Think of the model's brain as a giant, crowded dance floor.

The Neurons are the dancers.
The Concepts are the songs they are dancing to.
Polysemanticity means one dancer is trying to dance to two different songs at once (a jazz song and a heavy metal song).
The Interference: If you push the dancer to dance harder to the jazz song, they might accidentally start stomping their feet to the heavy metal beat, even though you didn't ask for that.

How They Tested It (The Four Levers)

The researchers tried to "hack" the model using four different levers to see if they could force these accidental connections to happen:

The Feature Lever (The Internal Map): They used a special tool (called a Sparse Autoencoder) to map the dance floor. They found two songs that seemed unrelated but shared a dancer. They pushed the dancer toward one song, and the model started singing the other song.
The Token Lever (The Gradient): Instead of looking at the map, they looked at the specific words (tokens) that made the dancer move. They found that pushing the words associated with one idea (like "License") would accidentally make the model talk about another idea (like "Politics"). This worked even better than the map.
The Prompt Lever (The Whisper): They didn't touch the model's brain at all. They just added a few specific words to the beginning of the conversation (like a secret code). This "whisper" was enough to make the model switch topics unexpectedly.
The Neuron Lever (The Super-Neuron): They found some dancers who were holding so many songs at once (500+ concepts!) that they were "Super-Neurons." If you turned these dancers up, the whole dance floor went crazy. If you turned them off, the floor barely noticed.

The Shocking Twist: It Works on Bigger Models Too

The most surprising part of the paper is what happened next.

They found these weird, accidental links in two tiny, simple models (like a child's toy). Then, they took the "secret codes" (the specific words or nudges) they found in the toy models and tried them on massive, advanced models (like Llama-3 or Gemma, which are like adult geniuses).

The Result: It worked.

Even though the big models were trained differently and were much smarter, they had the exact same hidden wiring. The "secret knock" that worked on the toy also opened the door on the genius.

The Analogy:
Imagine you find a loose floorboard in a small shed. You push it, and the ceiling fan turns on. You then go to a massive, high-tech skyscraper. You find a loose floorboard in the exact same spot, push it, and the skyscraper's ceiling fan turns on too.

This suggests that all these models, no matter how big or small, are built on the same fundamental "blueprint" of how they organize ideas.

Why Does This Matter?

It's Not Random: We used to think these weird connections were just random glitches. This paper proves they are systematic. There is a hidden structure to how AI thinks that we don't fully understand yet.
It's a Vulnerability: If you can trick a small model into saying something weird, you can probably trick a big model too, using the same "secret code." This is a safety risk.
It's a Tool: If we understand these hidden links, we can use them to control AI. We could potentially "nudge" a model to be more helpful or less toxic without retraining the whole thing.

The Bottom Line

AI models are like super-efficient libraries where books are stacked in a chaotic way. One book might be about "Cooking" and "Math" at the same time. The researchers found that if you pull the "Cooking" book, the "Math" book falls out too.

Even more importantly, this chaotic stacking happens in the same way in small libraries and giant libraries. If we learn the rules of this chaos, we can learn how to control the library, or at least understand why the books keep falling off the shelves.

1. Problem Statement

The paper addresses the challenge of polysemanticity in Large Language Models (LLMs), a phenomenon where individual neurons or groups of neurons encode multiple, distinct, and often semantically unrelated concepts simultaneously. While this "superposition" allows models to represent more features than they have neurons (enhancing capability), it creates a "black box" problem for interpretability and control.

The authors identify a specific, under-explored vulnerability: Polysemantic Interference. This occurs when two features that are semantically distinct (orthogonal in human symbolic space) share activation directions in the model's internal vector space, causing them to interfere with one another. The core research questions are:

Can manipulating semantically unrelated but interfering features reliably steer model behavior?
Do these interference structures persist across different model architectures and scales (transferability)?
Is there a systematic, higher-order organization to these interference patterns that transcends human intuition?

2. Methodology

The study employs a multi-layered approach combining Sparse Autoencoders (SAEs), mechanistic interventions, and cross-model evaluation.

A. Feature Extraction and Clustering

Models: The authors use pre-trained SAEs from Neuronpedia for two small models: Pythia-70M and GPT-2-Small.
SAE Features: They treat SAE features as explicit directions in the activation space ( $A_\ell$ ).
Semantic vs. Activation Space: They define a "Human Symbolic Manifold" ( $M$ ) using text embeddings (via GPT-4o-mini) to measure surface semantic similarity ( $S$ ). They compare this with the cosine similarity of feature directions in activation space ( $I$ ).
Agglomerative Clustering: To handle inconsistent semantic granularity (e.g., "dog" vs. "Labrador"), they cluster SAE features based on semantic similarity (using cutoffs from 0.15 to 0.40) to identify distinct, higher-level feature groups.

B. Intervention Strategies

The authors test four types of interventions to manipulate the "next-token prediction distribution":

Feature-Direction Steering: Projecting SAE feature vectors directly into the model's activation space to steer the model.
Token-Gradient Steering: Computing gradients of the model's output with respect to the top-activating tokens of an interference feature. This creates a steering vector that does not require SAE access to the target model.
Prompt Injection: Prepending text snippets that strongly activate interference features to the input prompt.
Neuron Intervention: Masking or amplifying specific neurons based on their "polysemanticity" (the number of distinct features they encode).

C. Evaluation Metrics

Weighted Cosine Similarity ( $\Delta c$ ): Measures the semantic alignment between the model's output distribution and a target feature's token set.
Weighted Overlap: Measures the raw probability mass assigned to target tokens.
Cross-Model Transfer: Interventions distilled from small models (Pythia/GPT-2) are applied to larger, black-box instruction-tuned models (Llama-3.1-8B/70B and Gemma-2-9B) to test generalizability.

3. Key Contributions

Demonstration of Covert Manipulation: The paper proves that models can be steered toward a target semantic direction by manipulating features that are semantically unrelated to the target but interfere with it in activation space.
Cross-Model Transferability: Interventions derived from small models successfully manipulate larger, instruction-tuned black-box models. This suggests a shared, stable polysemantic topology across architectures and training regimes.
Discovery of "Super-Neurons": The authors identify a class of highly polysemantic neurons (connected to $>500$ features) that act as critical junctions. Amplifying these neurons causes exponential behavioral shifts, while masking them has a surprisingly reduced effect.
Counterintuitive Regularities: Post-hoc analysis reveals that only ~27.7% of interfering feature pairs have plausible higher-order semantic associations (e.g., thematic or causal links) detectable by LLMs. The majority of interference patterns are opaque to human intuition, suggesting latent knowledge structures.

4. Key Results

Interference Effectiveness:
- Feature Steering: Semantically dissimilar but interfering features reliably shift next-token predictions. Stronger interference scales lead to larger shifts.
- Token-Gradient Superiority: Token-gradient steering yields roughly 10 $\times$ larger effects than direct SAE feature steering and is more robust across models.
- Prompt Injection: While weaker in latent-space manipulation, prompt injection shows massive gains (10 $\times$ –1000 $\times$ ) when measured by weighted overlap, indicating it concentrates interference on specific tokens.
Generalization to Large Models:
- Interventions using high-interference tokens from Pythia/GPT-2 successfully steered Llama-3.1-8B/70B and Gemma-2-9B.
- Success rates for specific feature types (e.g., location, emotion, numbers) were significantly higher than random baselines, even without access to the target model's internals.
Neuron Vulnerability:
- Neurons with higher polysemanticity are more vulnerable to intervention.
- "Super-Neurons" (encoding $>500$ features) exhibit asymmetric vulnerability: amplification drastically alters behavior, but deactivation (masking) has a diminished effect, suggesting they are robust hubs that require specific activation to function.
Semantic Opacity:
- In a paired-choice evaluation, LLMs (GPT-5-mini) identified latent associations in high-interference pairs only 64.3% of the time (significantly above chance but far from perfect), confirming that these structures are largely counterintuitive.

5. Significance and Implications

Theoretical Insight: The findings challenge the view that polysemanticity is purely stochastic or an artifact of training. Instead, they suggest a convergent, higher-order organization of internal representations that persists across scales. This supports the "Platonic Representation Hypothesis," implying that models learn shared semantic topologies.
Safety and Control: The work reveals a systematic vulnerability where models can be manipulated via "noise" (semantically unrelated inputs). This has implications for:
- Adversarial Attacks: New vectors for jailbreaking or steering models without needing internal access (via token gradients/prompts).
- Alignment: Offering new methods for targeted unlearning or steering behavior in black-box models.
Future Directions: The paper proposes that these "weak signals" and counterintuitive interference patterns may serve as testable hypotheses for latent knowledge structures in both artificial and human cognition, bridging the gap between mechanistic interpretability and cognitive science.

In summary, the paper demonstrates that the "noise" of polysemantic interference is actually a structured signal that can be exploited to predict and control model behavior across different scales, revealing a deep, shared architecture of meaning in LLMs.