Technical Summary: Ablation-Reversible Heads Don't Transfer

Problem Statement

In mechanistic interpretability (MI), a common workflow for assigning functional roles to attention heads involves combining three lines of evidence:

1. Selective Necessity: Ablating the head damages a specific capability while sparing others.
2. Linear Decodability: The head's activations linearly encode the requested computation.
3. Ablation Reversibility: Restoring the head's clean activations after ablation recovers performance.

The paper argues that this evidence stack is insufficient for claiming a head implements a specific computation (e.g., "this head performs addition"). The authors posit that these four properties—selective necessity, linear decodability, ablation reversibility, and interventional generalizability—are distinct and routinely dissociate. Specifically, a head may be necessary and reversible within a single prompt context without carrying a transferable semantic state that can redirect behavior in a new context. Conflating these properties leads to over-confident role claims and misleading interpretations of transformer circuits.

Methodology

The authors introduce KID (Knowing / Intent / Doing), a role-assignment vocabulary, and pair it with a three-stage empirical pipeline to test for interventional generalizability.

The KID Framework

• Knowing: The process of recognizing what a prompt asks for. A head fills this role if its activations at prompt positions carry decodable computation information and its ablation damages prompt-level recognition.
• Intent: A computation-selection state occurring after recognition but before execution. A head fills this role if it satisfies "Knowing" criteria and is interventionally generalizable (its state, when patched into a different prompt, redirects behavior toward the source computation).
• Doing: Answer execution and generation. A head fills this role if its activations at answer positions are necessary and ablation reversibility is achieved by restoring answer-side activations.

The Three-Stage Pipeline

1. Capability-Selective Screening (CSS): Identifies small sets of attention heads whose targeted ablation selectively damages a measured capability. This establishes behavioral importance but not representational claims.
2. Singular Value Decomposition (SVD): Inspects the information linearly present within CSS-selected heads using counterbalanced prompt controls (e.g., same answer string, different computation) to separate semantic structure from surface artifacts.
3. Activation Transduction Assays: The critical test for interventional generalizability. Clean activations from a source prompt (Computation A) are patched into a target prompt (Computation B) under matched controls.
   • Controls: The pipeline employs same-computation controls (varying surface form) and same-answer controls (sharing the answer string but differing in computation) to ensure that observed effects are due to computation selection rather than context or answer familiarity.
   • Full-Trajectory Restore: To distinguish "Knowing/Intent" from "Doing," the authors separately restore activations at prompt positions versus answer positions after ablation.

Key Results

The study was applied to three instruction-tuned 7–8B models (Qwen2.5, Llama-3, Mistral) across five computation families (arithmetic, comparison, digit properties, dates, times).

1. Systematic Dissociation of Properties

The primary finding is that ablation reversibility does not imply interventional generalizability.

• Heads that pass the prompt-side ablation-reversibility check (restoring performance when activations are patched back into the same prompt) routinely fail to transfer the computation when their activations are patched into a different prompt.
• Across all tested model-family and rank combinations, no head satisfied all four evidence properties required for an "Intent" role.

2. Role Taxonomy

The authors propose a preliminary taxonomy of roles for CSS-selected heads, none of which currently satisfy the full criteria for an interventionally generalizable computation-selection state:

• Answer-Side Logit-Bias Heads: Selectively necessary and reversible, but only when restoring activations at answer positions. These heads bias the output distribution toward plausible wrong answers or the reference answer but do not store the computation selection.
• Prompt-Side Stabilizers: Necessary for maintaining the computation trajectory through the prompt (preventing drift). They are reversible at prompt positions but inert or negative under activation transduction. They maintain context rather than selecting a specific computation.
• Prompt-Primary, Context-Broad Heads: Reversible at prompt positions, but activation transduction transfers broad source context (evidenced by movement in same-answer controls) rather than a specific computation label.
• Soft Computation-Pattern Carriers: Identified via SVD (e.g., in a Qwen compare head), these subspaces carry abstract ordered-selection information across related families. While they show a "soft" causal handle, they fail to produce answer-ordering changes on adversarial low-margin rows, falling short of a clean computation switch.

3. Methodological Implications

• Same-Answer Controls: The paper highlights the "same-answer control" as a critical, underused check. It exposes cases where patches move the model's output by transferring answer-string familiarity rather than a computation-selection state.
• Role Assignment Requires Transduction: Behavioral screening (CSS) and linear decodability are necessary preconditions for interpretability claims but are insufficient for role assignment. Only activation transduction under matched controls provides evidence for a transferable computation-selection state.

Significance and Claims

The paper claims to provide a rigorous stress test for mechanistic role claims, demonstrating that the standard evidence stack (necessity + decodability + reversibility) is prone to "interpretability illusions."

• Refutation of Common Assumptions: The authors show that heads often function as prompt-trajectory stabilizers or answer-side logit-bias heads rather than as dedicated computation-selection units ("Intent").
• Operationalizing Rigor: The paper operationalizes a path toward auditable MI by gating role claims on interventional generalizability tests. It suggests that if a computation-selection state exists in attention heads, it is likely rare or distributed, as no CSS-localized head in the current experiments met the full criteria.
• Future Direction: The methodology clarifies that finding a computation-selection state requires full-trajectory role localization to identify prompt-side candidates, followed by activation transduction under same-answer controls. If such heads are not found, the computation-selection state may reside in non-attention components (MLPs, residual stream) or be distributed across many heads.

The paper concludes that while CSS identifies real, compact, and often-decodable heads, the dissociation of necessity, decodability, and reversibility from generalizability necessitates a more cautious approach to assigning semantic roles to transformer components.