🤖 AI

Finding Interpretable Prompt-Specific Circuits in Language Models

This paper introduces ACC++, an improved circuit-tracing method that extracts interpretable, prompt-specific attention circuits from a single forward pass, revealing how language models utilize distinct, language-dependent signals to solve tasks like indirect object identification across different linguistic contexts.

Original authors: Gabriel Franco, Lucas M. Tassis, Azalea Rohr, Mark Crovella

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriel Franco, Lucas M. Tassis, Azalea Rohr, Mark Crovella

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, bustling city where millions of tiny workers (neurons) pass notes to each other to answer a question. For a long time, we could see what the city produced (the answer), but we had no idea how the workers decided to send those notes. It was like watching a magic show without knowing the tricks.

This paper introduces a new tool called ACC++ that acts like a high-tech "X-ray vision" for these models. It doesn't just guess; it traces the exact path of information as it flows through the model to solve a specific problem.

Here is a simple breakdown of what the paper does and finds:

1. The Problem: The "Black Box" City

When a model answers a prompt (like "When Mary and John went to the store, John gave the bottle to..."), it has to figure out that the answer is "Mary."

Old way: Researchers tried to figure out which workers were involved by turning them on and off one by one (like pulling fuses in a house to see which light goes out). This was slow, messy, and often gave a blurry picture with too many "false alarms."
The new tool (ACC++): Instead of pulling fuses, ACC++ listens to the "whispers" between workers. It identifies the specific, low-volume channels (called signals) that carry the crucial information needed to make a decision.

2. How ACC++ Works: The "Signal Detective"

Think of the model's attention mechanism (where it decides what to focus on) as a radio station.

The Old Method: It tried to find the whole radio tower that was broadcasting.
ACC++: It zooms in to find the exact frequency (a specific signal) and the exact microphone (the worker) that is speaking on that frequency.
The Magic: It does this in a single pass, instantly. It strips away the noise and leaves you with a clean, simple map showing exactly which workers talked to which, and what they said.

3. The Big Discovery: The "Prompt-Specific" Blueprint

The most exciting finding is that the model doesn't use the same blueprint for every question. It changes its strategy based on how you ask the question.

The "Name Game" Analogy (The IOI Task):
The researchers tested the model with a game: "When [Name A] and [Name B] went to the store, [Name B] gave a gift to [Name A]." The model needs to pick Name A.

Scenario A: "When Mary and John went..." (Mary is first).
Scenario B: "When John and Mary went..." (John is first).

The paper found that the model uses completely different internal circuits for these two scenarios, even though the task is the same.

In one case, the model uses a "Second Item" detector.
In the other, it uses a "Structural Pattern" detector.
The Takeaway: The model is flexible. It has a toolbox of different strategies and picks the right one based on the exact wording of your prompt.

4. The Multilingual Mystery: Same Workers, Different Languages

The researchers also tested the model with different languages (English, Spanish, French, Portuguese).

The Finding: The model uses the same group of workers (the same internal components) to solve the problem in all languages.
The Twist: However, the messages (signals) those workers pass to each other are language-specific.
The Analogy: Imagine a construction crew building a house. They use the same crew members (workers) whether they are building in New York or Paris. But in New York, they speak English and pass blueprints in English; in Paris, they speak French and pass blueprints in French. The structure of the work is the same, but the language of the communication changes.
Bonus Discovery: The paper found that the "distance" between the circuits for different languages matches how similar the languages are to each other. Spanish and Portuguese circuits look more alike than English and French circuits, just like the languages themselves.

5. Why This Matters (According to the Paper)

Clarity: It turns a messy, confusing web of connections into a clean, readable map.
Interpretability: The tool can even translate these internal "whispers" into plain English. For example, it can tell you, "This worker is looking for the second item in a list," or "This worker is recognizing a proper name."
Efficiency: It finds these answers much faster and with less "noise" than previous methods.

Summary

This paper gives us a new pair of glasses that lets us see exactly how an AI thinks, word by word. It reveals that the AI isn't just a static machine; it's a dynamic problem-solver that rearranges its internal team and changes its communication style depending on how you ask it a question and what language you use. It proves that we can understand the "mechanics" of these models without needing to break them apart.

Technical Summary: Finding Interpretable Prompt-Specific Circuits in Language Models

Problem Statement

Understanding the internal computational mechanisms of Large Language Models (LLMs) remains a central challenge in mechanistic interpretability. While prior work has established that models operate via "circuits"—causal pathways of components writing and reading information from the residual stream—existing methods for tracing these circuits face significant limitations. Traditional approaches often rely on activation patching, counterfactual inputs, or replacement models, which are computationally expensive, require multiple forward passes, and can introduce noise or artifacts. Furthermore, previous circuit-tracing methods often produce large, noisy graphs that obscure the specific low-dimensional signals used for communication, making it difficult to derive human-understandable explanations for model behavior on a per-prompt basis.

Methodology: ACC++

The paper introduces ACC++ (Attention-Causal Communication++), an improved circuit-tracing method that builds upon the principle of attention-causal communication (ACC). ACC++ is designed to identify the specific signals (contents of low-dimensional subspaces) that cause an attention head to attend to a specific token pair.

Core Mechanism

ACC++ operates on a single forward pass without requiring replacement models, patching, or counterfactual inputs. The method proceeds in four steps for a given attention head $(\ell, a)$ attending to a destination-source token pair $(d, s)$ :

Decomposition: The destination residual stream $x_d$ is decomposed into a sum of outputs from upstream components (embeddings, previous attention heads, MLPs).
Candidate Signal Construction: Using the Singular Value Decomposition (SVD) of the head's Query-Key (QK) matrix $\Omega = W_Q W_K^\top$ $Ω = W_{Q} W_{K}^{⊤}$ , the method projects upstream component outputs onto the singular vector directions. This creates a set of candidate signals, where each signal corresponds to a specific upstream component writing into a specific singular direction (a rank-1 candidate).
- Technical Advance: To handle model heterogeneity (e.g., bias terms and Rotary Positional Embeddings/RoPE), ACC++ unifies the attention computation into a single bilinear form $\tilde{x}_d^\top \Omega \tilde{x}_s$ by folding bias and RoPE rotations into the token representations. This allows for a single SVD analysis per head regardless of architecture.
Attribution and Scoring: The contribution of each candidate signal to the post-softmax attention weight $A_{ds}$ is scored using Integrated Gradients (IG). Unlike prior work that used linear surrogates for pre-softmax scores, ACC++ explicitly evaluates the nonlinear impact of signal removal on the full Softmax distribution.
Greedy Counterfactual Search: Starting with the full set of candidates, the algorithm greedily removes the highest-scoring signals until the attention weight $A_{ds}$ falls below a threshold $\tau$ . The removed set constitutes the causal circuit for that specific attention decision.

Circuit Construction

A full circuit for a prompt is constructed by recursively applying ACC++: starting from components with a direct effect on the target logit, the method traces upstream attention firings, identifying the causal signals for each, until no further upstream components remain.

Key Contributions

1. Conceptual and Technical Advances

Precise Signal Identification: ACC++ identifies signals as the cross-product of upstream components and singular vector directions, rather than identifying components and channels independently. This yields lower-dimensional, more precise signals.
Nonlinear Attention Accounting: By defining the counterfactual objective directly on the post-Softmax attention weight (rather than a linear surrogate), ACC++ avoids spurious components and identifies smaller, more accurate circuits.
Unified QK Analysis: The method handles bias and RoPE by transforming tokens, enabling a consistent SVD-based analysis across different model architectures (e.g., GPT-2, Pythia, Gemma).

2. Automatic Interpretability

The authors demonstrate that a substantial fraction of ACC++ signals are interpretable. They employ an automated pipeline (adapted from [18]) that:

Retrieves high-activation contexts for each signal from a dataset (The Pile).
Uses an LLM to generate a short natural-language description of the signal.
Validates the description using an LLM-as-a-judge protocol to distinguish signal from noise.
Results show that many signals admit short, coherent natural-language descriptions (e.g., "second item in a parallel pair," "proper noun," "structural delimiter").

3. Prompt-Specific Circuit Analysis

ACC++ enables the analysis of circuits on a per-prompt basis, revealing that model behavior is not static but varies significantly with prompt structure and language.

Results and Insights

1. Circuit Compactness and Quality

Across three models (GPT-2 Small, Pythia-160M, Gemma-2 2B) and three tasks (Indirect Object Identification, Greater-Than, Gender Pronoun), ACC++ circuits are 2x to 10x smaller than those produced by the original ACC method.

Dimensionality: The majority of ACC++ signals are rank-1 (using a single singular direction), compared to higher-dimensional signals in prior work.
Causal Validity: Ablation studies confirm that removing ACC++ signals significantly impacts model performance on the target task while minimally perturbing the residual stream (cosine similarity $\approx 0.99$ ), confirming their causal role.

2. Sensitivity to Prompt Structure (IOI Task)

In the Indirect Object Identification (IOI) task, the paper finds that circuits are highly sensitive to prompt templates:

GPT-2: Circuits cluster strictly by high-level template structure (ABBA vs. BABA order of names). The same "name-mover" head (9,9) uses fundamentally different incoming signals depending on the template (e.g., "second item in a pair" vs. generic structural cues).
Pythia: Circuits cluster by low-level wording changes (e.g., "While" vs. "Then"), indicating sensitivity to surface-level phrasing rather than just structural order.
Gemma-2: Shows a more complex, less template-dependent clustering, suggesting a reliance on finer-grained prompt details.

3. Multilingual Mechanisms (mIOI Task)

In a multilingual IOI setting (English, Spanish, French, Portuguese):

Component Reuse: Models reuse the same attention heads and MLPs across languages.
Language-Specific Signals: While components are shared, the signals flowing through them are often language-specific.
Linguistic Correlation: The distance between circuits of different languages correlates strongly with the typological distance between those languages (syntactic and genetic distances). For instance, Spanish and Portuguese circuits are more similar to each other than to English or French circuits.

Significance and Claims

The paper claims that ACC++ provides a practical, scalable tool for mechanistic interpretability that does not rely on expensive counterfactual interventions. Its primary significance lies in:

Scalability: Making per-prompt circuit analysis viable at scale by requiring only a single forward pass.
Interpretability: Providing a direct link between low-dimensional communication channels and human-readable natural language descriptions, bridging the gap between mathematical mechanisms and semantic understanding.
Nuanced Understanding: Revealing that model "circuits" are not monolithic but are dynamic assemblies of components and signals that adapt to prompt structure and language. This challenges the view of a single universal circuit for a task, showing instead a spectrum of prompt-specific mechanisms.

The authors note limitations, including the heuristic nature of the threshold $\tau$ , the lack of tracing for FFN-induced signals, and the fact that automated interpretations are suggestive rather than conclusive. They emphasize that the multilingual findings are based on a small number of language pairs and should be viewed as indicative of a broader trend rather than definitive proof.