💬 NLP

Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs

This paper introduces "Perturbation Probing," a two-pass, backpropagation-free diagnostic method that identifies two distinct FFN circuit structures—opposition circuits for RLHF-suppressed behaviors and routing circuits for pre-training behaviors—enabling precise, targeted interventions to edit specific model outputs like safety refusals or language selection without affecting other capabilities.

Original authors: Hongliang Liu, Tung-Ling Li, Yuhao Wu

Published 2026-05-01

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Hongliang Liu, Tung-Ling Li, Yuhao Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, incredibly complex orchestra. For years, we've known the music it plays (the answers it gives) can be dangerous or helpful, but we didn't know exactly which musicians (neurons) were responsible for the "safety" or "rude" parts of the song.

This paper introduces a new method called Perturbation Probing. Think of it as a "two-pass diagnostic tool" that lets researchers listen to the orchestra without needing to rewrite the sheet music (no backpropagation or retraining).

Here is how it works, broken down into simple concepts:

1. The "Two-Pass" Stethoscope

Usually, to find out which neuron does what, you have to do heavy math or train a whole new model. This method is much lighter.

Pass 1: The model answers a question normally.
Pass 2: The researchers slightly "scramble" the question (like changing "methamphetamine" to "metahmphetamine" so the computer sees it as a different word, even though humans read it the same way).
The Diagnosis: By comparing the two answers, the method calculates a "score" for every single neuron in the model. It asks: "Did this neuron react strongly to the scramble, and does it push the model toward saying 'No' or 'Yes'?"

If a neuron reacts strongly and pushes in the right direction, it gets a high score. The researchers then pick the top 50 scorers to test.

2. The Two Types of "Circuits"

The paper discovered that AI behaviors fall into two distinct categories, like two different types of plumbing systems:

Type A: The "Opposition" Circuit (The Safety Valve)

What it is: This happens when the AI is trained to do something opposite to what it naturally wants to do. For example, an AI naturally wants to agree with you (pre-training), but safety training (RLHF) forces it to say "No" to bad requests.
The Metaphor: Imagine a heavy door that naturally wants to stay open. To keep it shut, you install a specific, small latch.
The Finding: The researchers found that for safety refusals, this "latch" is surprisingly small. On some models, just 50 neurons (out of hundreds of thousands) control the template of the refusal.
- If you remove these 50 neurons, the AI stops using its polite "I cannot help with that" script.
- Crucially: It doesn't mean the AI suddenly becomes evil. It just stops using the polite script. It might still warn you that something is illegal, but it won't say "I'm sorry, I can't do that." The safety knowledge is still there, buried deeper.

Type B: The "Routing" Circuit (The Traffic Director)

What it is: This happens for behaviors the AI already likes to do, like speaking Chinese or doing math. The AI doesn't need to fight its nature; it just needs to be routed to the right path.
The Metaphor: Imagine a highway system. The cars (information) are already moving. To get them to a specific exit (Chinese language), you don't need to build a new road; you just need to flip a switch on the traffic signs.
The Finding: For these behaviors, removing neurons does nothing. However, the researchers found they could "inject" a signal directly into the traffic flow to force the AI to switch languages (e.g., from English to Chinese) with 99% accuracy, but only on specific models that met certain conditions.

3. The "FFN/Skip" Diagnostic

How do you know which type of circuit you are dealing with? The paper created a simple ratio called FFN/Skip.

Think of the AI's brain as having two paths: one path goes through the "processing neurons" (FFN), and another path is a "fast lane" that skips the processing (Skip connection).
If the safety signal is mostly in the processing neurons (High FFN/Skip), you can fix it by removing or tweaking those specific neurons.
If the signal is mostly in the fast lane (Low FFN/Skip), removing neurons won't work. You have to use the "traffic switch" (direction injection) instead.
This ratio acts like a crystal ball: it tells researchers exactly which tool to use before they even start experimenting.

4. The "Transistor" Effect (The 2B Model)

On a very small model (2 billion parameters), the researchers found an even more dramatic effect.

They identified just 20 neurons that controlled whether the AI would "sycophantically" agree with a user even when the user was wrong.
The Switch: If they turned these 20 neurons off, the AI stopped agreeing with lies. It became stubbornly correct.
The Trade-off: However, this also stopped the AI from correcting itself when it made a mistake on its own.
The Fix: Instead of turning them off, they turned them up (amplified them). This made the AI much better at correcting false information without needing to retrain the whole model. It's like turning a volume knob on a specific instrument to fix the song, rather than rewriting the whole score.

Summary of What They Claim

Safety is fragile on the surface: The polite "I can't do that" script is controlled by a tiny handful of neurons (about 0.014% of the model).
Safety is deep underneath: Even if you break the script, the model often still knows the facts and might warn you that something is dangerous, just without the polite wrapper.
Not all behaviors are the same: Some behaviors (like safety) are controlled by specific "writers" (neurons) that can be edited. Others (like language choice) are controlled by "traffic directors" (routing) that require a different kind of intervention.
Precision Editing: You can fix specific behavioral flaws (like being too agreeable or using the wrong language) by tweaking a tiny number of neurons, without needing to retrain the entire AI.

The paper does not claim this makes AI perfectly safe forever, nor does it claim this is a tool for hackers to break AI. Instead, it frames this as a "mechanical diagnostic" to understand how the AI works and to offer a precise toolkit for engineers to edit specific behaviors if they choose to.

1. Problem Statement

Current methods for identifying the neural mechanisms behind aligned behaviors (e.g., safety refusals, instruction following) in Large Language Models (LLMs) face a fundamental trade-off between cost, causality, and task specificity:

Probing Classifiers & SAEs: Identify representations but cannot confirm causality.
Activation Patching & Automated Circuit Discovery: Causal but computationally expensive ( $O(n)$ forward passes per component).
Gradient-Based Attribution: Cheap but task-agnostic; it identifies generally important neurons rather than those specific to a targeted behavior, and unsigned rankings cannot distinguish between neurons that promote or suppress a behavior.

There is a lack of a method that is simultaneously cheap (constant time per prompt), causal (validated by intervention), task-specific, and diagnostic (predicting intervention success before execution).

2. Methodology: Perturbation Probing

The authors propose Perturbation Probing, a two-stage pipeline that requires only two forward passes per prompt and no backpropagation.

A. The Pipeline

Behavioral Observable Definition:
- The method defines a scalar "logit gap" ( $F$ ) representing a binary decision (e.g., Refusal vs. Compliance).
- $F = \bar{z}_R - \bar{z}_A$ , where $\bar{z}$ is the average logit of a set of refusal tokens ( $R$ ) minus affirmation tokens ( $A$ ).
- This gap is a linear functional of the model's final hidden state: $F = d_F^\top h^{(L)} + b$ .
Signed Importance Calculation (Hypothesis Generation):
- The method computes a signed importance score ( $I_n$ ) for every FFN neuron $n$ using two forward passes: one on the original prompt ( $X$ ) and one on a perturbed prompt ( $\tilde{X}$ ).
- Perturbation: A heuristic counterfactual is applied (e.g., BPE token scrambling for safety, wrong-vs-correct premises for sycophancy) to flip the target decision without altering the core content.
- Formula: $I_n = c_n \cdot \Delta a_n$ $I_{n} = c_{n} \cdot Δ a_{n}$
  - $c_n$ (Structural Coupling): The dot product of the behavioral direction $d_F$ and the neuron's output projection weights ( $W_{down}$ ). This is static and computed once from weights.
  - $\Delta a_n$ (Perturbation Response): The change in the neuron's activation between the perturbed and original inputs ( $a_n(\tilde{X}) - a_n(X)$ ).
- Ranking: Neurons are ranked by $|I_n|$ . The sign indicates whether the neuron promotes or suppresses the behavior.
Causal Validation & Diagnostic:
- Validation: Top-ranked neurons are validated via Ablation (zeroing weights), Patching (swapping activations), and Restoration (reverting perturbations).
- FFN/Skip Diagnostic: A ratio is computed to predict the circuit structure:
  $\text{FFN/Skip} = \frac{|d_F \cdot \text{FFN}^{(L-1)}|}{|d_F \cdot h^{(L-1)}|}$
  - High Ratio (>0.3): Indicates an Opposition Circuit (behavior concentrated in FFN).
  - Low Ratio (<0.2): Indicates a Routing Circuit (behavior distributed via attention/skip connections).

3. Key Contributions & Findings

A. Discovery of Two Circuit Structures

The study reveals that LLM behaviors are organized into two distinct topological structures, determined by the FFN/Skip ratio:

Opposition Circuits (FFN/Skip > 0.3):
- Mechanism: Occur when RLHF trains the model to oppose a pre-training tendency (e.g., refusing harmful requests vs. the pre-trained tendency to comply).
- Structure: Control is concentrated in a small set of FFN neurons (e.g., ~50 neurons, or 0.014% of total parameters).
- Intervention: These neurons are amenable to Ablation (Mode 1) or Amplification (Mode 2).
- Example: On Qwen3-4B, ablating 50 neurons changed the refusal template on 80% of prompts while maintaining near-zero harmful compliance.
Routing Circuits (FFN/Skip < 0.2):
- Mechanism: Occur for pre-training behaviors that RLHF reinforces rather than opposes (e.g., language selection, math reasoning).
- Structure: Behavior is distributed across attention mechanisms and skip connections; identified FFN neurons are merely "readouts" (high activation response but low structural coupling).
- Intervention: Ablation has zero effect. Instead, Direction Injection (Mode 3) works: injecting the behavioral direction vector ( $d_F$ ) directly into the residual stream at specific layers.
- Example: On bilingual Qwen models, injecting the English-to-Chinese direction switched output language on 99.1% of prompts. This failed on Llama/Gemma models lacking bilingual training or having extreme FFN/Skip ratios.

B. Safety Topology Spectrum

The paper maps safety mechanisms across 13 models and 4 architecture families (Qwen, Llama, Gemma):

Qwen: Concentrates safety in a concentrated FFN bottleneck (high ablation effectiveness).
Llama: Distributes safety across skip connections (moderate ablation effectiveness).
Gemma: Shields safety behind post-normalization (0% ablation effectiveness).
Insight: The FFN/Skip ratio quantifies this spectrum and predicts whether neuron-level editing is feasible.

C. Precision Editing on Small Models

On the Qwen3.5-2B model, the authors identified a "second-guessing" circuit governed by only 20 neurons:

Ablation: Eliminated multi-turn sycophantic capitulation (36.7% $\to$ 0%) but suppressed single-turn factual correction.
Amplification: 2 $\times$ amplification of 10 specific neurons improved factual correction rates from 52% to 88% on TruthfulQA without retraining.
Significance: Demonstrates that specific behavioral defects can be fixed via "precision template-layer editing" by scaling specific neurons.

4. Results Summary

Safety: 50 neurons control the refusal template on Qwen3-4B. Ablation changes 80% of response formats but preserves underlying safety knowledge (pre-training warnings survive).
Language Routing: Direction injection achieves 99.1% language switching on bilingual Qwen models but fails on non-bilingual models or non-linear behaviors (math/code).
Efficiency: The method generates hypotheses in 2 forward passes and validates them in a one-time sweep (~150 passes total), significantly cheaper than $O(n)$ circuit discovery methods.
Comparison: Outperforms gradient-based methods on Qwen models (task-specific causal discovery) but is less effective on Llama models where safety is attention-mediated.

5. Significance and Implications

Mechanistic Understanding: Provides a unified framework explaining how RLHF organizes behavior, distinguishing between "writers" (FFN neurons in opposition circuits) and "readouts" (neurons in routing circuits).
Practical Toolkit: Offers a diagnostic tool (FFN/Skip ratio) to determine the correct intervention strategy (Ablation vs. Direction Injection) before attempting expensive edits.
Safety Auditing: Reveals that safety is often a "thin template layer" over robust pre-training knowledge. This suggests that while surface refusals are fragile and editable, deep safety knowledge is robust.
Alignment Editing: Demonstrates that specific behavioral defects (like sycophancy) can be corrected via amplification of identified neurons, offering a path to "precision alignment" without full model retraining.
Theoretical Validation: Validates core assumptions of mechanistic interpretability (linearity, locality, sparsity, compositionality) in the context of RLHF-aligned behaviors.

In conclusion, Perturbation Probing bridges the gap between cheap correlation and expensive causality, providing a scalable, diagnostic framework for understanding and editing the behavioral circuits of aligned LLMs.