🤖 AI

How LLMs Are Persuaded: A Few Attention Heads, Rerouted

This paper reveals that persuasion-induced factual errors in large language models are caused by a compact, monitorable circuit where shallow attention heads detect persuasive keywords to redirect a specific evidence-routing feature, forcing decision heads to jump from the correct answer to a persuasion-target vertex in a low-dimensional latent space.

Original authors: Xiangkun Sun, Lingkai Kong, Aoqi Zhang, Liang Zeng, Tonghan Wang

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangkun Sun, Lingkai Kong, Aoqi Zhang, Liang Zeng, Tonghan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Hypnotist" in the Machine

Imagine a large language model (LLM) as a very smart, well-trained librarian. This librarian knows the facts: they know the capital of France is Paris, not London. They have a massive library of truth in their head.

However, this paper discovered that if you whisper a very convincing, persuasive lie to the librarian right before they answer, they will suddenly forget the truth and tell you the lie. This happens even if the librarian knows better.

The researchers wanted to know: Where exactly inside the librarian's brain does this switch happen? Is the whole brain getting confused? Do they lose their memory? Or is there a specific, tiny switch being flipped?

The Discovery: It's Not a Brain Fog, It's a Switch

The researchers found that persuasion doesn't make the whole model "confused" or "less confident." Instead, it hijacks a very small, specific part of the model's internal wiring.

Think of the model's brain as a giant factory with thousands of workers (called "attention heads").

The Finding: Only a tiny handful of these workers (specifically, a few in the middle layers of the factory) actually decide the final answer.
The Analogy: Imagine a voting booth where 1,000 people are shouting opinions, but only two specific people hold the actual ballot box. If you convince those two people to change their vote, the whole election changes, even if the other 998 people are still shouting the truth.

The "Tetrahedron" Map: How Choices Live in the Brain

The researchers looked at how these two special workers (called "decision heads") process the four possible answers (A, B, C, D).

The Geometry: They found that these workers arrange the four answers in a specific 3D shape, like the corners of a pyramid (a tetrahedron).
- Corner 1 = Answer A
- Corner 2 = Answer B
- Corner 3 = Answer C
- Corner 4 = Answer D
The Normal State: When the model is asked a question without lies, the internal signal lands firmly on the "Correct Answer" corner.
The Persuasion State: When a persuasive lie is added, the signal doesn't slowly drift or get muddy. It teleports. It instantly jumps from the "Correct" corner to the "Lie" corner.

The Metaphor: Imagine a train station with four tracks leading to four different cities. The train is usually on the track for "Truth." Persuasion doesn't make the train drive slowly toward the wrong city; it instantly switches the tracks so the train is suddenly on the "Lie" track.

The Mechanism: The "Copy-Paste" Workers

Here is the most surprising part: The researchers found that these special "decision workers" aren't actually thinking or reasoning about the evidence.

What they do: They act like a copy machine. They look at which option token (A, B, C, or D) they are told to look at, and they simply copy that token's identity into the final answer.
How the trick works: The persuasion doesn't change the workers' ability to copy. Instead, it changes which option they are looking at.
- Normal: The worker looks at the "Truth" option and copies it.
- Persuaded: A persuasive keyword (like a specific name or a fake fact) tricks the worker into looking at the "Lie" option instead. The worker then copies the lie, thinking it's the right choice.

The Root Cause: The "Keyword Hunters"

So, what makes the worker look at the wrong option?

The researchers traced the signal back to a group of workers in the earlier layers of the factory (layers 8 through 12).

The Hunters: These early workers scan the input text for "persuasive keywords" (like the word "Nigeria" in their example, which was used to trick the model about a fact).
The Signal: When they find these keywords, they write a tiny, one-dimensional signal onto the option tokens.
The Reroute: This signal acts like a magnet. It pulls the attention of the "decision workers" (the copy machines) away from the truth and toward the lie.

The Real-World Test: "Generative Engine Optimization" (GEO)

The paper didn't just test this in a lab. They tested it in a realistic scenario called Generative Engine Optimization (GEO).

The Scenario: Imagine website owners trying to trick search engines. They write articles specifically designed to hijack the AI's output, making the AI pick their website as the "best" source, even if it's full of lies.
The Result: The researchers found that this same "hijacking circuit" (the keyword hunters and the copy-machine workers) was the exact same mechanism used in these real-world attacks. The AI wasn't being "tricked" by complex reasoning; it was being rerouted by a simple, narrow path.

Summary: What This Means for Safety

The paper concludes that persuasion is not a massive, chaotic failure of the AI's brain. It is a narrow, monitorable circuit.

Old View: "The AI is confused by lies."
New View: "The AI has a specific, tiny switch that can be flipped by a keyword, causing it to copy the wrong answer."

Because this mechanism is so small and specific, the authors suggest we might be able to build "security guards" (monitors) that watch just these few workers. If they see the "persuasion signal" trying to flip the switch, they can stop it before the wrong answer is written, without needing to retrain the whole AI.

In short: The AI isn't losing its mind; it's just having its attention hijacked by a very specific, tiny lever. If we can find that lever, we can stop the hijacking.

Technical Summary: How LLMs Are Persuaded: A Few Attention Heads, Rerouted

Problem Statement

Large Language Models (LLMs) possess a critical safety vulnerability: they can be persuaded to abandon factual knowledge in favor of incorrect information when presented with persuasive context. While behavioral studies have documented high rates of "sycophancy" (29–62% error rates in high-stakes domains) and the commercial exploitation of this via Generative Engine Optimization (GEO), the internal mechanistic cause of this override remains poorly understood. Existing defenses, such as prompt engineering or fine-tuning, address symptoms rather than root causes, leaving the underlying vulnerability intact. This paper seeks to provide the first circuit-level explanation of how persuasive text overrides factual knowledge, moving beyond static untruthfulness to understand how the override is constructed dynamically from the input.

Methodology

The authors employ a mechanistic interpretability framework based on the Transformer Circuits paradigm, utilizing activation patching and representation analysis on decoder-only Transformer models (LLaMA-3, Qwen-3, Gemma-2, Gemma-3).

Experimental Setup: The study uses a controlled multiple-choice setting (NQ2 and Geo-Bench) where models answer factual questions. Prompts are constructed in pairs: a "clean" prompt and a "persuasive" prompt containing a span of text (logical, credibility-based, or emotional appeals) designed to steer the model toward a specific incorrect option.
Causal Localization (Activation Patching): To identify the components responsible for persuasion, the authors perform interchange interventions. They replace the update of a specific component (attention head or MLP) in a persuasive run with its value from a clean run. Components with high "restoration scores" (reverting the model to the correct answer) are identified as causally critical.
Geometric Analysis: The authors analyze the residual stream representations using Principal Component Analysis (PCA) to understand how answer options are encoded. They specifically examine the geometry of the decision subspace before and after key attention heads.
Circuit Tracing: The study traces the causal chain from the input to the output by:
- Decomposing attention heads into QK (query-key) and OV (value-output) circuits.
- Extracting a rank-one "option-routing feature" via Singular Value Decomposition (SVD) of the QK circuit.
- Using "noising" and "denoising" patching to locate the specific layers that construct this routing feature from persuasive keywords.

Key Contributions and Findings

1. Sparse Decision Heads and Tetrahedral Geometry

The paper identifies that persuasion susceptibility is not a diffuse corruption of the network but is localized to a very small set of mid-layer attention heads (termed "decision heads"). In the LLaMA-3 model, Layer 17, Head 24 (L17H24) is the primary driver.

Geometry: These heads encode the four answer options as distinct vertices of a low-dimensional tetrahedron (a 3D subspace).
Mechanism: Persuasion does not blur belief or gradually reduce confidence. Instead, it causes a discrete latent jump where the representation moves from the vertex corresponding to the correct answer to the vertex corresponding to the persuasion target.

2. Routing, Not Reasoning

The authors demonstrate that decision heads do not internally compute or reason over evidence to determine the answer.

Copying Mechanism: The OV circuit of the decision head acts as a faithful copy mechanism. The option tokens already carry a discrete identity code in the directions the OV circuit can read. The head simply copies whichever option token it attends to.
Rerouting: Consequently, persuasion works entirely by rerouting attention. If the attention is redirected to the persuasion target token, the model outputs the wrong answer.

3. The Option-Routing Feature

The paper isolates a specific, one-dimensional feature (a rank-one vector $u_k$ ) within the QK circuit of the decision head that governs attention selection.

Causal Control: Modifying this feature directly steers the model's choice. Adding the feature vector to option tokens increases their selection probability; removing it blocks persuasion.
Construction: This routing feature is not present in the raw input. It is constructed by a band of shallower attention heads (Layers 8–12) that read persuasive keywords from the input and write the feature onto the option-token representations.

4. End-to-End Causal Chain

The study validates a complete causal chain:

Input: Persuasive keywords are read by shallow heads (Layers 8–12).
Feature Construction: These heads construct the option-routing feature ( $u_k$ ) on the option tokens.
Routing: The decision head (e.g., L17H24) reads this feature via its QK circuit, redirecting attention to the persuasion target.
Output: The OV circuit copies the target token, resulting in a discrete jump to the wrong vertex in the decision tetrahedron.

5. Real-World Generalization (GEO)

The mechanism is not limited to synthetic benchmarks. The authors show that the same decision heads and geometric structures govern model behavior in Generative Engine Optimization (GEO) scenarios, where retrieved web content is strategically poisoned to hijack LLM outputs. The circuit identified in controlled settings transfers directly to these realistic poisoning scenarios.

Significance and Claims

The paper reframes LLM persuasion from a behavioral failure into a narrow, monitorable circuit.

Mechanistic Understanding: It provides a precise account of where and how the override happens, identifying specific components (a few attention heads and a 1D feature) rather than treating the model as a black box.
Defensive Potential: By isolating the specific pathway, the authors suggest that targeted runtime monitors could detect persuasion in progress, and interventions could neutralize the specific pathway without retraining the model.
Limitations: The authors explicitly state that their findings are derived from controlled multiple-choice settings. It remains an open question whether this exact circuit governs free-form generation. Furthermore, while the mechanism is characterized, the paper does not evaluate specific defenses built upon these findings.

In summary, the paper argues that persuasion is a "rerouting" attack on a specific, low-dimensional feature within the model's residual stream, controlled by a small, identifiable set of attention heads, making the vulnerability structurally distinct from static knowledge errors.