💬 NLP

What Drives Representation Steering? A Mechanistic Case Study on Steering Refusal

This paper investigates the causal mechanisms of steering vectors in large language models through a case study on refusal, revealing that they primarily function by interacting with the OV circuit to modulate attention scores, which allows for significant sparsification and the identification of semantically interpretable concepts.

Original authors: Stephen Cheng, Sarah Wiegreffe, Dinesh Manocha

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stephen Cheng, Sarah Wiegreffe, Dinesh Manocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) like a giant, super-smart orchestra. Every time it answers a question, hundreds of musicians (neurons) play together to create a symphony of words. Usually, this orchestra plays whatever the conductor (the user) asks for.

But sometimes, the orchestra has a "safety rule" built into its sheet music: "If the request is dangerous, stop playing and say 'I can't do that'."

Steering Vectors are like a secret wand that a researcher waves over the orchestra to change the music without rewriting the whole score. You can wave it to make the orchestra refuse a harmless request, or wave it the other way to make the orchestra ignore its safety rules and play a dangerous tune.

This paper asks a very simple but deep question: How does this magic wand actually work inside the orchestra?

Here is the breakdown of their findings, using some fun analogies:

1. The "Secret Shortcut" (The Circuit)

The researchers wanted to know: When we wave the wand to make the AI say "No," which specific musicians are actually doing the work? Is it the whole orchestra, or just a few key players?

They found that the "refusal" signal doesn't need the whole orchestra. It only needs a tiny, specific secret shortcut (about 10% of the musicians).

The Analogy: Imagine a massive highway system. You might think traffic jams happen everywhere, but the researchers found that the "refusal" traffic is only flowing through one specific, narrow tunnel. If you block that tunnel, the traffic stops. If you leave it open, the traffic flows, even if you close 90% of the rest of the highway.

2. The "Volume Knob" vs. The "Sheet Music" (OV vs. QK)

Inside the AI, there are two main ways information flows:

The QK Circuit (Query & Key): This is like the musicians reading the sheet music to decide who to listen to.
The OV Circuit (Output & Value): This is like the musicians actually playing the notes and the volume.

The paper discovered something surprising: The steering wand almost entirely ignores the sheet music (QK). It doesn't change who the musicians are listening to. Instead, it hijacks the volume knobs (OV).

The Analogy: It's like a sound engineer who doesn't tell the band to change the song or the lyrics. Instead, they just walk up to the mixing board and crank the volume on the "Refusal" channel to 100% and the "Compliance" channel to 0%. The band is playing the same notes, but the output is completely different because of the volume mix.

3. The "Hidden Meaning" (Decoding the Wand)

When you look at the steering wand (the vector) itself, it looks like a giant, messy list of random numbers. It's impossible to read.

The Analogy: It's like looking at a bag of mixed-up Lego bricks and trying to guess what the final castle looks like.

However, the researchers found a way to "translate" the bricks. By breaking the wand down, they found that specific parts of it correspond to clear concepts like "Forbidden," "Illegal," or "Dangerous."

The Takeaway: Even though the wand looks like gibberish, it is actually made of very clear, understandable ideas. They just needed the right decoder ring to see them.

4. The "Sparse Signal" (The 90% Rule)

The researchers asked: "Do we need the whole wand, or just a tiny piece of it?"
They found that you can throw away 90% to 99% of the numbers in the steering wand, and it still works almost perfectly!

The Analogy: Imagine you have a giant map to a treasure. You think you need every single road marked. But the researchers found that if you just keep the three main highways, you can still get to the treasure. The rest of the map is just noise.
Why this matters: This means we can make steering much faster and cheaper because we don't need to carry around the whole heavy wand; just the tiny, important core is enough.

5. Different Methods, Same Path

The paper tested three different ways to create these wands (some by math, some by training). They expected them to be totally different.

The Analogy: It's like three different people trying to get to the same coffee shop. You'd expect them to take three different routes.
The Result: They all ended up taking the exact same secret tunnel. No matter how you make the wand, the AI uses the same internal machinery to process the "Refusal" command.

Summary

This paper is a "mechanical autopsy" of how AI safety works. They found that:

It's local: Safety steering only uses a tiny, specific part of the brain.
It's a volume knob: It changes the output intensity, not the internal logic.
It's efficient: You can strip away 99% of the data and it still works.
It's universal: Different methods all use the same internal path.

Why should you care?
If we understand exactly how the "safety switch" works, we can build better switches that are harder to break (making AI safer) and easier to fix when they get stuck saying "No" to harmless questions (making AI more helpful). It turns the "black box" of AI into a machine with visible, understandable gears.

1. Problem Statement

Large Language Models (LLMs) are increasingly aligned with human intent using steering vectors, a lightweight technique that modifies hidden activations at inference time to induce specific behaviors (e.g., refusal, persona control, or jailbreaking). While effective, the mechanistic basis of how these vectors interact with internal model components to alter outputs remains poorly understood.

The Gap: There is no clear explanation of which internal circuits (subgraphs of the model) are responsible for propagating steering effects, how they interact with attention mechanisms, or whether different steering methodologies (e.g., Difference-in-Means vs. Learning-based) utilize the same underlying pathways.
The Goal: To provide a causal, mechanistic explanation of steering vectors by analyzing the "refusal" concept—a critical safety capability—using circuit discovery techniques adapted for steered generation.

2. Methodology

The authors propose a novel framework to extend mechanistic interpretability (specifically activation patching) to multi-token steered generations.

A. Multi-Token Activation Patching Framework

Traditional circuit discovery operates on single-token tasks with clean vs. corrupted inputs. The authors adapt this for steering:

Setup: They treat the steered generation (with the vector added) as the "clean" input and the base generation (without the vector) as the "corrupted" input.
Edge Attribution Patching with Integrated Gradients (EAP-IG): They calculate the Indirect Effect (IE) of edges in the computational graph. Instead of a single forward pass, they approximate the gradient of the steering effect across a sequence of tokens by linearly increasing the steering coefficient ( $\alpha$ ) during the integration process.
Circuit Extraction: They identify a minimal set of edges (a "circuit") that, when patched, recovers the steered behavior with high faithfulness (defined as the ability to reproduce the logit difference between steered and base models).

B. Steering Vectors Analyzed

The study compares three distinct methodologies for obtaining refusal steering vectors on Gemma 2 2B and Llama 3.2 3B:

Difference-in-Means (DIM): A non-learning method computing the difference in mean activations between harmful and harmless datasets.
Next Token Prediction (NTP): A learning-based method optimizing a language modeling objective on prompt-response pairs.
Preference Optimization (PO): A learning-based method using contrastive responses (preferred vs. dispreferred).

C. Ablation and Decomposition

Circuit Ablation: They freeze specific components (QK attention scores, OV values, MLPs) to measure performance degradation.
Steering Value Vector (SVV) Decomposition: They mathematically decompose the direct effect of the steering vector on attention heads, isolating an input-invariant component ($svv$) that flows through the OV circuit (Output-Value).
Sparsification: They test if steering vectors can be compressed by zeroing out dimensions based on their contribution to the steering effect (gradient-based or IE-based sparsification).

3. Key Contributions & Results

A. Circuit Interchangeability and Overlap

Shared Circuits: Despite different training methodologies (DIM, NTP, PO), the resulting steering vectors leverage highly interchangeable circuits.
Overlap: Circuits derived from different methods show >90% overlap in their edges.
Functional Equivalence: A circuit discovered via DIM can be used to steer a model using NTP or PO vectors with nearly identical faithfulness (recovering >85% of the effect). This suggests a shared "backbone" for refusal steering regardless of how the vector is derived.
Sparsity: Only ~10% of the model's total edges are required to recover 85% of the steering behavior, indicating that refusal steering is highly localized to specific subnetworks.

B. Mechanism of Action: The OV Circuit Dominance

The study reveals a surprising asymmetry in how steering vectors interact with the attention mechanism:

OV vs. QK: Steering vectors primarily interact with the OV circuit (Value $\to$ Output projection).
QK Insignificance: Freezing the QK circuit (Query $\to$ Key attention scores) has a negligible effect on steering performance (only an 8.75% drop in Attack Success Rate).
Conclusion: Steering vectors bypass the attention scoring mechanism (QK) and directly modulate the value passed to the output (OV).

C. Semantic Interpretability via SVV Decomposition

The Problem: Raw steering vectors (especially NTP and PO) are often uninterpretable via standard "Logit Lens" techniques.
The Solution: By decomposing the vector into Steering Value Vectors (SVVs) for individual attention heads, the authors uncover semantically meaningful concepts.
Findings: SVVs reveal tokens related to "forbidden," "illegal," "unsafe," and "refusal." This decomposition makes the steering mechanism interpretable even when the raw vector is not.

D. Sparsification and Shared Subspace

High Sparsity Tolerance: Steering vectors can be sparsified by 90–99% (keeping only 1–10% of dimensions) while retaining most performance.
Convergence: Different steering methods (DIM, NTP, PO) converge on the same small subset of important dimensions.
Methodology: Gradient-based sparsification (filtering dimensions by their normalized contribution to the steering effect) outperforms random dropout or magnitude-based pruning.

4. Significance and Implications

Mechanistic Understanding: The paper moves beyond "black box" alignment, proving that steering vectors operate through specific, localized, and interchangeable circuits, primarily via the OV attention pathway.
Efficiency: The discovery that steering vectors are highly sparse (90%+ compressible) suggests that future alignment interventions could be significantly more efficient, requiring fewer parameters or dimensions to achieve the same behavioral shift.
Interpretability Tool: The SVV decomposition provides a new tool for interpreting high-dimensional vectors in the residual stream, applicable beyond steering to sparse autoencoders and model editing.
Safety and Robustness:
- Defense: Understanding that QK circuits are largely ignored by steering vectors could inform new defense mechanisms that specifically protect the OV pathway.
- Offense (Jailbreaking): The findings also imply that targeted, sparse interventions could be used to more effectively jailbreak models by focusing on the specific dimensions and circuits identified.
Generalizability: The multi-token activation patching framework is a reusable tool for analyzing any behavior modification in LLMs, not just refusal.

Conclusion

This paper establishes that representation steering is not a diffuse phenomenon but a targeted manipulation of specific, sparse circuits within the LLM. By demonstrating that different steering methods converge on the same functional pathways (OV circuit) and dimensions, the authors provide a roadmap for more precise, efficient, and interpretable model alignment and intervention.