💬 NLP

Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning

This paper challenges the assumption that function-vector heads are a homogeneous group by revealing they consist of two opposing sub-populations—writers that promote correct rules and cancellers that suppress them—which are invisible to magnitude-only ranking but significantly impact model performance when identified and ablated.

Original authors: Han-yu Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han-yu Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a large language model (like the ones that chat with you) as a massive orchestra trying to solve a puzzle based on a few examples. In this orchestra, there are specific musicians called attention heads. For a long time, researchers believed that the most important musicians were simply the ones playing the loudest notes. They assumed that if a musician was playing very loudly, they were definitely helping the orchestra play the right tune.

This paper, however, discovers that this assumption is wrong. It turns out that the "loudest" musicians actually belong to two completely different, opposing groups: The Writers and The Cancellers.

Here is the breakdown of what the paper found, using simple analogies:

1. The Big Misunderstanding: Volume vs. Direction

Previously, researchers looked at these musicians and ranked them by volume (how much they contributed to the answer). They assumed the top 20 loudest musicians were all "helpers."

The authors of this paper realized that volume isn't enough; you need to know the direction.

The Writers: These musicians play a loud note that pushes the orchestra toward the correct answer.
The Cancellers: These musicians also play a loud note, but they are pushing the orchestra toward the wrong answer. They are actively trying to cancel out the correct answer.

The Analogy: Imagine a tug-of-war.

Writers are the team pulling the rope toward the finish line.
Cancellers are a second team pulling the rope just as hard, but in the opposite direction.
If you only look at how hard they are pulling (magnitude), you think they are both equally important "pullers." But if you look at the direction, you see they are fighting each other.

2. The Discovery: Two Populations

The researchers tested this on several different models (ranging from small to large) and different types of logic puzzles. They found that in almost every case, the "top" musicians split into these two opposing camps.

The "Writers" push the probability of the correct answer up.
The "Cancellers" push the probability of the correct answer down.

When the researchers silenced (ablated) the Cancellers, the model actually got better at solving the puzzles. It was like removing a saboteur from the tug-of-war team; suddenly, the "Writers" could pull the rope to the finish line without resistance.

3. Why Previous Research Missed This

The paper explains that earlier studies (like one by Todd et al., 2024) used a method that only looked at the "loudness" of the contribution. Because of this, they accidentally picked up a mix of Writers and Cancellers, but the mix changed depending on the puzzle.

On some puzzles, the "Cancellers" were louder, so the old method thought they were the main helpers.
On other puzzles, the "Writers" were louder, so the old method thought they were the main helpers.

The paper shows that by ignoring the "sign" (direction), previous research was essentially mixing up the heroes and the villains, thinking they were all just "loud helpers."

4. Are the Cancellers Just Mistakes?

The researchers wanted to make sure the "Cancellers" weren't just broken parts of the model or accidental noise. They ran many tests to prove they are real, functional parts of the system:

They read different things: Writers focus on the "labels" (the answers) in the examples. Cancellers focus on the "format" (the punctuation and spacing). They are looking at different parts of the page.
They are content-driven: The Cancellers aren't just randomly suppressing answers; they are specifically trying to suppress the answer that was just demonstrated in the examples.
They are not "Copy Suppressors": The paper ruled out the idea that these are just generic "don't copy" mechanisms. They are specific to the logic of the task.

5. The "L11.H4" Character Study

The paper zooms in on one specific musician, L11.H4, to see how it works.

This musician is a "Canceller" on the logic puzzles (it tries to push the wrong answer).
However, if you change the puzzle to a vocabulary task (like matching antonyms), this same musician flips its role and becomes a "Writer" (it helps the correct answer).
The Lesson: This musician isn't inherently "good" or "bad." Its job is to suppress what was just demonstrated. If the demonstration is the right answer, it suppresses it (Canceller). If the demonstration is the wrong answer (in a different context), it suppresses the wrong answer, effectively helping the right one (Writer).

6. The Takeaway

The main conclusion is that we cannot treat the most "active" parts of an AI model as a single, uniform group.

Old View: "These are the top 20 most important heads. Let's use them to steer the model."
New View: "These top 20 heads are actually a civil war. Half are pushing the model to the right place, and half are pushing it to the wrong place. If you want to steer the model, you need to silence the Cancellers, not just amplify the Writers."

The paper does not claim this makes the model "safe" or ready for real-world medical or legal use. It simply states that to understand how these models learn rules from examples, we must recognize that their internal "musicians" are often fighting each other, not working together.

Technical Summary: Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning

1. Problem Statement

In the field of mechanistic interpretability, Function-Vector (FV) heads are typically identified as a sparse set of attention heads with large causal contributions to in-context learning (ICL) rule tasks. The standard identification pipeline (e.g., Todd et al., 2024) ranks these heads by the magnitude of their causal contribution (e.g., Direct Logit Attribution or AIE) and treats the top-ranked set as a homogeneous functional class.

This paper challenges the implicit assumption that high-magnitude heads uniformly push the model toward the demonstrated rule. The authors posit that magnitude-only ranking obscures a critical structural heterogeneity: the top set may contain heads with opposing causal effects. If this assumption fails, standard aggregation methods (like steering vectors) may inadvertently mix opposing forces, leading to suboptimal or contradictory interventions.

2. Methodology

The authors propose a sign-preserving identification pipeline and a rigorous validation framework across three model families (Pythia, Qwen2.5, GPT-2) and six scales (410M–12B).

Refined Direct Logit Attribution (DLA): Instead of ranking by absolute magnitude, the authors compute the signed contribution of each head to the correct-minus-incorrect logit difference ( $\Delta \ell$ ). They apply a permutation-based False Discovery Rate (FDR) control ( $q=0.10$ ) to identify a candidate set of FV heads that includes both positive and negative contributors.
Path Patching Validation: Candidates are validated using path patching to isolate direct effects. Heads are partitioned into two populations based on their direct effect magnitude relative to the prompt swing:
- Writers ( $W$ ): Heads with a positive direct effect ( $\ge +5\%$ of the swing), pushing the correct token's logit up.
- Cancellers ( $C$ ): Heads with a negative direct effect ( $\le -5\%$ of the swing), pushing the correct token's logit down.
The Canonical Verdict: The existence of two opposing populations is verified via a four-condition test on group ablations (zeroing or mean-ablation):
1. Ablating $W$ significantly reduces the correct logit ( $\Delta \ell_W < 0$ ).
2. Ablating $C$ significantly increases the correct logit ( $\Delta \ell_C > 0$ ).
3. The signs are opposite.
4. Joint ablation ( $W \cup C$ ) shows attenuation (the effect is smaller than the sum of individual effects), indicating they operate on the same readout but in opposing directions.
Artifact Rejection: The authors rule out six alternative explanations for the "canceller" signal:
1. Induction Overlap: Canceller heads are not simply induction heads.
2. Attention Sinks: The effect is not an artifact of attending to format prefixes or BOS tokens (verified via per-source DLA bucketing).
3. Generic Importance: Cancellers are not just high-attention heads with random signs.
4. Rank-1 Copy-Suppression: The mechanism is not a simple rank-1 suppression of copied tokens (unlike GPT-2 L10.H7).
5. V-Cascade: The effect is not a result of inhibiting an upstream writer via a specific V-composition cascade.
6. Rank-Nearest Controls: Random heads with similar magnitude do not exhibit the same cancellation behavior.

3. Key Results

Population Split: Across 15 cells (6 main Pythia cells + 9 extensions), the FV head set splits cleanly into writers and cancellers. The canonical verdict holds in 13/15 cells.
Asymmetric Overlap with Prior Work: Magnitude-only ranking (Todd et al., 2024) is biased toward the locally dominant sub-population.
- On the Hierarchical task, Todd's top-20 captures 64% of cancellers but only 4% of writers.
- On the Modular task, Todd's top-20 captures 59% of writers but only 8% of cancellers.
Causal Impact of Cancellers: Zero-ablating the canceller population yields a consistent logit gain of +0.13 to +0.29 nats and an accuracy improvement of +2 to +7 percentage points in 6/6 main cells.
Structural Distinction:
- QK Sources: Writers concentrate attention on demonstration labels; cancellers shift attention mass toward format-prefix tokens and away from labels.
- OV Geometry: The write directions (OV) of writers and cancellers are nearly orthogonal (perpendicular fraction mean 0.96), allowing them to coexist at the same readout without fully canceling each other out in the residual stream.
- Layer Distribution: Cancellers cluster in early-to-mid layers, often situated downstream of a writer in the same cell, ruling out late-stage suppression.
Case Study (L11.H4): The dominant canceller in Pythia-410M is identified as a "distributed-bias content reader." Its role flips based on the task template (canceller on rule tasks, writer on antonym tasks), supporting the hypothesis that it suppresses "what was demonstrated" rather than performing unconditional logit suppression. It does not fit the rank-1 copy-suppression or V-cascade templates.

4. Significance and Claims

The paper claims that the FV head population is not a homogeneous class but a structural property comprising two opposing sub-circuits.

Methodological Correction: The standard "magnitude-only" ranking is insufficient for identifying the full functional landscape of FV heads. It systematically misses the sub-population that opposes the dominant signal, leading to incomplete circuit models.
Intervention Implications: The paper distinguishes between ablation, transplant, and additive steering.
- Ablation: Removing cancellers improves performance.
- Transplant: Replacing heads with their task-mean (which effectively neutralizes the specific per-prompt behavior of cancellers) improves performance compared to writer-only transplants.
- Steering: The authors explicitly state they do not claim that an aggregated steering vector (combining $W$ and $C$ ) is the correct operation for practical steering. Because $W$ and $C$ have opposing signs, aggregating them into a single vector may dilute the steering effect. The population-level identification serves to define which heads to intervene on, not necessarily how to aggregate them.
Generality: The phenomenon is observed across different model architectures (GPT-NeoX, Llama-style, Conv1D) and scales, suggesting it is a fundamental property of how transformers implement in-context rule following, rather than an artifact of a specific model or task.

5. Limitations

The authors maintain modesty regarding the scope of their findings:

Task Scope: The primary analysis is limited to short, closed-label, synthetic ICL tasks (hierarchical and modular rules) and two vocabulary-ICL templates (antonyms, country-capital). It does not cover long-form generation or open-ended instruction following.
Mechanism Generalization: While the existence of the two populations is a population-level finding, the specific mechanistic characterization of the dominant canceller (L11.H4) as a "distributed-bias content reader" is not extrapolated to the entire canceller population.
Steering Claims: The paper makes no positive claim that vector-based steering using the identified populations improves practical model control beyond the specific logit shifts observed. The accuracy gains from ablation do not automatically translate to a proven advantage for additive steering vectors without further tuning.