💬 NLP

Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Models

This paper introduces a "control-window law" that establishes a budget-normalized framework to predict when single-neuron interventions in language models will coherently steer behaviors like refusal without causing output collapse, revealing that controllability is a typed, budgeted property rather than a simple scalar dose.

Original authors: Hongliang Liu

Published 2026-06-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongliang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: It's Not About How Hard You Push, It's About Where You Push

Imagine a massive, complex machine (a Language Model) that can do many things: speak different languages, do math, or refuse to answer dangerous questions. Scientists have discovered that sometimes, you can change what this machine does by tweaking just one tiny switch (a single neuron) inside it.

However, previous experiments were messy. Sometimes turning that switch worked perfectly; other times, it made the machine go crazy and repeat gibberish. The big question was: Why does it work sometimes and break other times?

This paper says the answer isn't just about "how much" you turn the switch. It's about a specific Goldilocks Zone called the Control Window.

The Analogy: Tuning a Radio vs. Breaking a Speaker

Think of the machine's internal state as a radio signal.

The Neuron: A specific knob on the radio.
The Dose: How hard you turn that knob.
The Goal: You want to tune the radio to a specific station (e.g., "Answer in Chinese" or "Refuse this request").

The paper argues that turning the knob has three possible outcomes, depending on how hard you turn it:

Too Soft (Inert): You turn the knob, but the station doesn't change. Nothing happens.
Just Right (The Control Window): You turn the knob to a precise spot. The radio switches cleanly to the new station. The music is clear, and the behavior changes exactly as intended.
Too Hard (Collapse): You turn the knob too far. The signal overloads, the speakers blow out, and the radio starts screaming static or repeating the same word over and over. The machine "collapses."

The Law: The paper proves that for any single neuron, there is a specific "window" between "too soft" and "too hard." If you can find that window, you can control the behavior. If the window doesn't exist (because the "too hard" point is reached before you even get to the "change" point), you can't control that behavior with that neuron.

Key Concepts Explained Simply

1. Leverage is Not Reach

The paper makes a crucial distinction: Leverage vs. Reach.

Leverage: How much the machine's internal math changes when you push the switch.
Reach: Whether that change actually results in a useful, coherent behavior change.

The Analogy: Imagine pushing a heavy door.

High Leverage, Low Reach: You push the door with all your might (high leverage), but you push it in the wrong direction. The door doesn't open; instead, the hinges break, and the door falls off (collapse).
Low Leverage, High Reach: You push the door gently in the exact right spot. It swings open smoothly.

The paper found that the "smartest" neurons (the ones that actually control behavior) often have low leverage. They are subtle. If you look at the machine using standard "gradient" tools (which measure how hard you have to push), these subtle neurons look invisible or unimportant. But they are the ones that actually work.

2. The "Budget" and the "Ceiling"

The paper introduces a way to measure the "dose" (how much you push) based on the machine's own size, rather than using a random number.

The Budget: Think of this as the machine's "energy capacity" at that specific moment.
The Ceiling: This is the point where the machine breaks. The paper shows you can predict this ceiling just by looking at the machine's blueprint (weights) before you even touch it.

The Prediction: If the "trigger" needed to change the behavior is lower than the "ceiling" where the machine breaks, you have a Control Window. If the trigger is higher than the ceiling, the window is closed, and you can't control it with that neuron.

3. The "Bypass" vs. The "Real Harm"

When testing on "refusal" (making the AI say "I can't do that"), the paper found a surprising split.

Coherent Bypass: The AI stops saying "I can't do that" and starts talking fluently about the topic.
Actionable Reach: The AI actually provides the dangerous instructions or harmful content.

The Analogy: Imagine a security guard at a bank.

Bypass: You trick the guard into stepping aside. He lets you walk in, but you just stand there looking at the walls. You got past the guard, but you didn't steal anything.
Actionable Reach: You get past the guard and actually open the vault.

The paper found that for some neurons, you can get the "Bypass" (the guard steps aside) easily, but you never get the "Actionable Reach" (the vault opens). The AI might talk fluently about a dangerous topic but refuse to give the actual steps to do it. This means a simple "did it say no?" test is not enough; you have to check if it actually did the bad thing.

What This Means for the Field

It's Predictable: You don't have to guess. You can look at the machine's math, calculate the "ceiling," and know in advance if a neuron is a "controller" or a "breaker."
Old Tools Were Wrong: Standard methods that look for the "loudest" neurons (high gradients) are actually finding the ones most likely to break the machine. The real controllers are the quiet, subtle ones.
Safety Check: This gives model creators a new way to test safety. Instead of just asking "Can we break it?", they can measure "How wide is the window where it breaks?" If the window is tiny or non-existent, the model is robust.

Summary

This paper turns the mystery of "tweaking one neuron to change AI behavior" into a precise science. It says: Don't just push hard. Find the specific, budgeted amount of push that sits between "doing nothing" and "breaking the machine." If that spot exists, you have control. If not, you don't. And just because you can make the AI talk differently doesn't mean it will actually do what you want.

Technical Summary: Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Models

1. Problem Statement

While aligned language models gate critical behaviors (e.g., refusal, language routing, arithmetic operator choice) through sparse feed-forward (FFN) neurons, there is currently no theoretical framework predicting when intervening on a single neuron will coherently control a behavior versus causing the model output to collapse into degenerate text.

Two fundamental gaps exist in the current literature:

Attribution-Intervention Disagreement: Neurons ranked as important by local gradient attribution often fail to causally control behavior, while true controllers are frequently invisible to gradients. High "leverage" (the ability of a dose to move the model state) does not predict "reach" (the coherent behavioral effect).
Lack of Predictive Theory: No existing principle explains why a specific intervention flips one behavior coherently while collapsing another. Interventions at fixed magnitudes yield contradictory verdicts depending on the arbitrary dose chosen, preventing single-neuron control from being a predictable, falsifiable phenomenon.

2. Methodology: The Budget-Normalized Control-Window Framework

The authors propose a framework that treats single-neuron intervention as a dose-response relationship, analogous to pharmacology. The core innovation is normalizing the intervention dose against the model's internal geometry rather than using fixed magnitudes.

The Control Coordinate

A single-neuron intervention adds a scalar multiple $t$ of a write direction $v$ (the down-projection column of an FFN neuron) to the residual stream $r_L$ :
$r(t) = r_L + t v$
Because the residual stream passes through normalization, only the direction of the perturbed stream matters. The effect reduces to a single control coordinate: the alignment between the perturbed residual and the write direction, $\cos \theta_v$ .

The Coherence Budget and Universal Drive

The dose is normalized by the coherence budget $B$ , defined as the ratio of the residual norm to the write direction norm:
$B = \frac{\|r_L\|}{\|v\|}$
The dimensionless coupling $g = t/B$ drives the control coordinate along a universal saturation curve:
$\cos \theta_v = \frac{g}{\sqrt{1 + g^2}}$
This curve describes how the residual rotates toward the write direction as the dose increases.

The Control-Window Law

Coherent control exists if and only if the behavior's trigger threshold ( $\cos \theta_{trig}^v$ ) lies below the collapse ceiling ( $\cos \theta_{coll}^v$ ):
$\text{Coherent Control} \iff \cos \theta_{trig}^v(\tau) < \cos \theta_{coll}^v$

The Trigger: A behavior-dependent threshold where the behavior flips. It varies by behavior class (e.g., $\approx 0.3$ for language routing, $\approx 0.45$ for refusal) and rollout horizon $\tau$ .
The Ceiling: A geometry-dependent threshold where the output collapses. It is predicted from the model weights and a single forward pass, scaled by a collapse coefficient $m^*$ (related to the residual's participation ratio).

3. Key Contributions

A Control-Window Law on One Coordinate: The paper establishes that the effect of any dose reduces to a single coordinate, $\cos \theta_v$ , following a universal drive curve. Coherent control is strictly bounded by a trigger (behavior-specific) and a ceiling (geometry-specific).
Prospective Ceiling Prediction: The collapse ceiling can be predicted before intervention using only weights and one generic forward pass. On 15 held-out neurons, the predicted ceiling had a Mean Absolute Error (MAE) of 0.14 (0.07 in bulk layers).
Three Mechanisms for Closed Windows: The paper identifies three reasons why a neuron might fail to control a behavior:
- Geometric Close: The collapse ceiling is lower than the trigger (observed in Gemma models).
- Drive-Capped Close: Post-FFN normalization caps the neuron's reach below the trigger (observed in Gemma).
- Decoupled Close: The write occurs too late in the network depth to propagate before the output is generated.
Generality Across Behaviors: The law governs benign mode switches (language routing, operator choice) and safety behaviors (refusal). The trigger values differ by behavior class, but the underlying geometry remains consistent.
Levers vs. Reach (Gradient Blindness): The paper demonstrates that local gradient attribution anti-predicts control. True controllers write "off-axis" (perpendicular to the immediate readout), carrying near-zero first-order gradients but significant second-order curvature. Consequently, gradient-ranked neurons are often collapse-prone, while the most effective controllers are invisible to standard attribution.
Typed Outcomes: Intervention success is not a scalar but "typed." A neuron can achieve coherent bypass (flipping refusal in fluent text) without achieving strict-actionable reach (providing harmful, operational content). These two outcomes decouple, particularly in refusal scenarios.

4. Results and Validation

Predictive Accuracy: The framework successfully predicted the "open" or "closed" verdict for 11 out of 15 held-out neurons. The four "misses" were not violations of the law but informative edge cases (e.g., geometric closes, drive-capped closes, or weak opens).
Collapse Coefficient: The collapse coefficient $m^*$ was found to be architecture-dependent. It is $\approx 1.4\text{--}1.7$ for Qwen and Llama families but drops to $0.31$ for Gemma-3, correlating with the residual's participation ratio (concentration).
Discovery Efficiency: A forward-only contrastive screen combined with the control-window law recovered 4/4 known controllers that gradient ranking missed (0/4). The contrastive screen provides recall, while the window law provides precision.
Refusal Audit: On a cohort of refusal neurons, the study showed that while many neurons could bypass refusal coherently, only a subset (3 of 6 audited Llama pivots) produced genuine actionable harmful content. Furthermore, actionable content often commits late in the rollout (after $\approx 96\text{--}128$ tokens), distinguishing it from early-token bypass.
Two-Dimensional Window: The control window is shown to be a surface over dose and rollout time. Some behaviors (like language routing) are token-local, while others (like operator switching) require a rollout horizon to manifest.

5. Significance and Claims

The paper claims to transform single-neuron steering from a collection of anecdotes into a predictable, falsifiable law of deep learning mechanics.

Unification: It unifies prior findings (super weights, refusal neurons, arithmetic mechanisms) by placing them on the same control window, resolving apparent contradictions between control, collapse, and invisibility.
Defensive Utility: It provides a "two-axis safety diagnostic" (coherent-bypass margin vs. strict-actionable potency) allowing model owners to audit robustness. A model can have a wide bypass window but zero actionable reach, or a closed window entirely.
Methodological Shift: It prescribes a new standard for intervention: budget-normalized dosing read at the behavior's rollout horizon. This corrects the record on neurons previously labeled as destructive due to fixed-magnitude overshoot.
Theoretical Insight: It clarifies that "leverage is not reach." True controllers operate by seeding off-axis perturbations that downstream circuits amplify, rendering them invisible to first-order gradient analysis.

The authors conclude that while the collapse ceiling is predictable from geometry, the trigger value remains a measured quantity dependent on behavior class and depth, representing the primary open problem for a fully first-principles theory. The work establishes a measurement discipline that separates behavioral routing from capability reach, offering a rigorous tool for auditing model controllability.