🤖 machine learning

Spectral Probe-Circuits: A Three-Step Recipe for Identifying Attention-Head Circuits in Pretrained Transformers

This paper introduces a three-step, unsupervised methodology for identifying causal attention-head circuits in pretrained transformers, demonstrating that a spectral signal-based approach successfully isolates task-specific induction circuits across diverse model scales, architectures, and training pipelines.

Original authors: Yongzhong Xu

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongzhong Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a giant library of books (a neural network) that has been reading millions of stories to learn how to speak. Inside this library, there are thousands of tiny librarians (called "attention heads") who decide which words to pay attention to when forming a sentence.

This paper presents a three-step recipe to find the specific group of librarians responsible for a particular trick, like "induction" (spotting a pattern like "A, B... A, B" and predicting the next B). The authors want to know: Which specific librarians are doing this job, and can we prove it?

Here is the recipe, explained simply:

Step 1: The "Activity Meter" (The Spectral Signal)

The Problem: You can't just look at the library and know who is working. You need a way to spot librarians who are actually thinking about the content of the story, rather than just staring blankly or following a rigid rule.

The Solution: The authors invented a "participation ratio" meter.

The Analogy: Imagine a librarian who always points to the same shelf, no matter what book you ask about. That's boring; they aren't really thinking. Now imagine a librarian who looks at a different shelf for every single book you ask about. That librarian is actively processing the content.
The Tool: The authors measure how "spread out" a librarian's attention is over time. If a librarian's attention jumps around to many different places depending on the story, their "meter" goes up. This helps them find librarians that are doing specialized work without needing to know what specific task they are doing yet. It's like finding the most active workers in a factory just by watching how much they move around.

Step 2: The "Job Description" Screen

The Problem: The "Activity Meter" from Step 1 finds all the busy workers. But we want to find the workers doing one specific job (like induction). The meter might highlight a worker who is busy doing something else entirely.

The Solution: They apply a "screen" to filter the list.

The Analogy: You have a list of 100 active workers. You ask them: "Who is looking at the word immediately before the current one?" (Previous-token) or "Who is looking at the pattern 'A... A'?" (Induction).
The Tool: They check the attention patterns of the active workers. If a worker is looking at the right place for the specific task (e.g., looking back at the first "A" when they see the second "A"), they get a "ticket" to the candidate list. This turns a general list of "busy workers" into a specific list of "induction workers."

Step 3: The "Fire Drill" (Causal Verification)

The Problem: Just because a worker is looking at the right place doesn't mean they are causing the result. Maybe they are just watching, and someone else is doing the real work.

The Solution: They perform a "fire drill" (ablation).

The Analogy: You tell the specific group of "induction workers" identified in Step 2 to stop working (you zero out their output).
- The Test: Does the library stop predicting the next word correctly?
- The Control: To make sure you aren't just breaking the library by randomly firing people, you also fire a different group of workers from the same department who were not on your list.
- The Result: If the library crashes only when you fire the "induction workers," but keeps working fine when you fire the random group, you have proven that your specific group was the one doing the job.

What Did They Find?

The Recipe Works Everywhere: They tested this on models ranging from very small (51 million parameters) to quite large (1 billion parameters). In every single model, they found a tiny team of 3 to 6 librarians responsible for the induction trick.
It's Predictive: They could find these specific teams during the training process, before the model was even finished. The "Activity Meter" lit up for the right workers before the model even started getting good at the task.
The "Specialist" Ratio is Constant: No matter how big the library gets, the percentage of librarians doing these specialized, high-level jobs stays roughly the same (about 17–19%). The rest of the library handles general stuff.
Different Models, Different Teams: Even though the job (induction) is the same, different models use different teams of librarians to do it. One model might use workers in the first few rows; another might use workers in the middle rows. But the recipe finds the right team for each specific model.

Summary

This paper gives us a reliable, three-step method to find the "brain cells" in AI that do specific tricks. It doesn't just guess; it finds the active workers, filters them by job description, and then proves they are essential by turning them off and watching what breaks. It shows that even as AI models get huge, the core teams doing the clever work remain small and consistent.

Technical Summary: Spectral Probe-Circuits

Problem Statement

Mechanistic interpretability of pretrained transformers typically relies on post-hoc analysis: identifying attention-head circuits after a model is fully trained by ablating heads and observing performance degradation on a fixed, pre-specified task. This workflow is computationally expensive (requiring forward passes over fully trained networks for every condition) and inherently reactive, as the target capability must be defined before the search begins. Furthermore, existing methods often struggle to distinguish between general content-dependent computation and task-specific causal circuits, particularly in larger models where "attention sinks" (heads attending to the first token) dominate the landscape.

Methodology: A Three-Step Recipe

The paper proposes a three-step recipe for identifying attention-head circuits that operates per-head, requires no task labels or attribution gradients, and can be applied during training or on final checkpoints.

Step 1: Per-Head Spectral Signal (The General Indicator)

The first step identifies heads performing sustained, content-dependent computation using a spectral metric derived from random matrix theory.

Metric: For each head at each training checkpoint, the Participation Ratio (PR) of the per-head attention output matrix is computed. The PR measures the effective rank (dimensionality) of the output distribution.
Trajectory Feature: The method integrates the time-evolution of this signal: $I(L, H) = \sum_t \max(PR(L, H, t) - 1, 0) \cdot \Delta \log(\text{tokens}_t)$ .
Function: This "PR-integral" ranks heads that maintain high effective rank (spanning many directions) over time. It serves as a general specialization indicator, filtering out heads that collapse to low-rank (content-independent) states or remain noisy. Crucially, it does not distinguish what the head is doing, only that it is doing specialized computation.

Step 2: Task-Pattern Screen (The Task-Specific Filter)

The general spectral signal is specialized to a target task by measuring attention patterns against canonical targets.

Mechanism: An "all-head" screen measures the selectivity of every head for specific attention patterns (e.g., induction, previous-token, duplicate-token, first-token/BOS, self, local).
Selectivity Definition: Selectivity is the ratio of attention to target positions versus a uniform baseline.
Thresholds: Heads are classified if their selectivity exceeds a threshold (e.g., $\ge 30\times$ for assignment, $\ge 50\times$ for induction circuit membership).
Function: This step converts the general "specialization" signal into a candidate circuit for a specific capability. It addresses the limitation of Step 1, which in attention-sink-dominated models would rank generic content-dependent heads higher than task-specific ones.

Step 3: Causal Verification (The Falsifiable Claim)

Candidate circuits are validated via group ablation with rigorous controls to ensure the effect is causal and not an artifact of layer composition.

Procedure: The per-head slice of the residual stream at the attention output projection is zeroed out.
Controls:
1. Matched-Random: Ablating an equal number of heads in the same layers that were not selected by the screen. This rules out the hypothesis that "ablating any heads in these layers causes degradation."
2. Upper Bound: Ablating all heads in the selected layers. This establishes the maximum possible effect for that layer.
Criterion: A circuit is deemed causal if the screen-picked heads produce a degradation significantly larger than the matched-random control, while the upper bound saturates near zero.

Key Results

1. Cross-Architecture and Cross-Scale Replication

The recipe was validated across an 8 $\times$ parameter range (51M to 1B-active / 7B-total) and three architecture families (dense GPT-NeoX, dense Llama-style, and Llama-style MoE) trained on different data pipelines (TinyStories, FineWeb, Pile, DCLM).

Induction Circuit: A small induction circuit (3–6 heads) was found to be causally necessary for synthetic induction tasks in every model tested.
Sublinear Scaling: While total head counts increase linearly with model size, the size of the induction circuit remains sublinear (3–11 heads), indicating that the specific capability does not scale linearly with model capacity.

2. The Conserved Fraction of Specialized Heads

Across the 8 $\times$ scale range (124M to 410M), the fraction of heads performing identifiable specialized computation is conserved at $\sim$ 17–19%.

This fraction was recovered independently using two methods: (1) classifying top- $K$ spectral picks into capability classes, and (2) counting heads with induction selectivity above a null-distribution noise floor.
In larger 1B-class models, this fraction varies more significantly (4.7% to 27.3%) and correlates with the dominance of BOS-attractors (attention sinks) rather than model scale, suggesting the conservation is a property within specific architecture/data families.

3. Predictive Power and Seed Specificity

Seed Specificity: In a 51M-parameter model trained with six different random seeds, the recipe identified distinct, seed-specific circuits for the same task. Ablating the circuit identified for Seed A had no effect on Seed B's model, confirming the method finds the model's actual implementation rather than a universal pattern.
Predictive Timing: The spectral signal (Step 1) rises before the capability selectivity crosses its functional threshold. The recipe can identify the circuit using only a small fraction (0.3–2%) of the total training tokens, preceding the full emergence of the capability.

4. Robustness to Data Shifts

In a model trained on FineWeb and then continued on OpenWebText, the induction circuit identified at the FineWeb endpoint remained invariant (Jaccard = 1.00) through the data shift. The heads maintained their selectivity and PR trajectories, suggesting the method identifies structural specialization rather than corpus-specific statistics.

5. Compositional Dependencies

The method successfully identified the "previous-token" circuit (8–9 heads) and demonstrated its causal role in induction. Ablating the previous-token circuit destroyed induction performance, confirming the standard compositional structure where previous-token heads build the key vectors used by induction heads.

Significance and Claims

The paper positions itself as the methodology anchor for a three-paper program. Its primary claims are modest and focused on the validity of the recipe itself:

The Recipe is a Recipe, Not a Universal Screen: The authors explicitly state that Step 1 (spectral signal) is a general specialization indicator, not a task-specific detector. Step 2 (screen) provides task specificity, and Step 3 (ablation) provides causal falsifiability. The combination is what makes the method robust.
Honest Framing of Causal Claims: The paper emphasizes that the "matched-random" differential is what makes the causal claim honest. Without this control, one cannot distinguish between a specific circuit and a general layer effect.
Developmental Precedence: The spectral signal precedes the emergence of capability selectivity, offering a potential window for monitoring circuit formation during pretraining (detailed in a companion paper).
Limitations: The authors note that for composed tasks (e.g., Indirect Object Identification), the primary attention pattern may decouple from the task-causal structure across different model families, requiring a broader set of candidate screens. Additionally, the method relies on fixed evaluation batches, and the specific circuit heads may vary across re-trains (though the fraction of specialized heads appears stable).

In summary, the paper provides a reproducible, label-free framework for isolating small, causally necessary attention-head circuits, demonstrating that a conserved fraction of model capacity is dedicated to specialized computation that scales sublinearly with model size.