💬 NLP

Automated Attribution Graph Interpretation via Probe Prompting

This paper introduces "probe prompting," a transparent rule-based pipeline that groups features in attribution graphs into concept-aligned supernodes using Cross-Prompt Activation Signatures, successfully validating their causal steering behavior across factual domains with minimal computational overhead.

Original authors: Giuseppe Birardi, Gonçalo Paulo

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Giuseppe Birardi, Gonçalo Paulo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, bustling city with millions of workers (neurons) passing notes to each other to answer a question. We know exactly how the notes are passed (the math), but the city is so huge and the notes so complex that it's impossible for a human to look at the map and say, "Ah, this specific group of workers is responsible for the fact that the capital of Texas is Austin."

This paper introduces a new way to map that city, called Probe Prompting. Here is how it works, using simple analogies:

1. The Problem: A City Too Big to Read

Previously, researchers tried to understand the model by looking at "attribution graphs." Think of these as blueprints showing which workers influenced the final answer.

The Issue: Even for a simple question like "What is the capital of Texas?", the blueprint has thousands of lines and nodes. It's like trying to read a novel by looking at every single letter individually. It takes hours to trace just one path, and doing this for thousands of questions is impossible.
The Old Way: Some researchers tried to ask an AI to label these workers ("This one is about 'Texas'"). But often, the AI was just guessing based on what the worker usually does, not what it actually did in that specific moment.

2. The Solution: The "Probe" Detective

The authors created a transparent, rule-based system called Probe Prompting. Instead of guessing, they treat the model like a suspect in a detective game.

The Setup: They take a specific question (e.g., "The capital of Texas is...") and generate a small set of "probe" questions that are almost identical but swap the details (e.g., "The capital of Florida is...", "The capital of New York is...").
The Test: They watch how specific workers (features) react to these probes.
- Analogy: Imagine you have a security guard in a building. To see what they do, you don't just watch them once. You ask them, "What do you do when a person named 'John' walks in?" Then, "What about 'Mary'?" Then, "What about 'Bob'?"
- If the guard only reacts when "John" walks in, they are a John-Detector.
- If they react to any name, they are a Name-Detector.
- If they react when someone says "is" or "the," they are a Grammar-Helper.

3. Grouping the Workers: "Supernodes"

Based on these reactions, the system groups thousands of individual workers into Supernodes.

The Magic: Instead of looking at 5,000 individual workers, the system says, "Okay, these 50 workers all act like 'Texas Detectors,' so let's call them one big team: Team Texas."
The Result: The massive, confusing city map is now compressed into a simple, readable diagram with just a few named teams (e.g., Team Texas, Team Austin, Team "Capital").

4. The Proof: The "Swap" Experiment

How do we know these labels are real and not just a lucky guess? The authors perform a Causal Swap.

The Experiment: They take a question about Dallas (where the answer should be Austin) and try to force the model to answer Minnesota (where the answer is St. Paul).
The Method: They "turn down the volume" on the workers labeled Team Dallas and "turn up the volume" on the workers labeled Team Minnesota.
The Outcome:
- When they used their Probe Prompting labels, the model successfully changed its answer from Dallas to Minnesota about 73% of the time.
- When they used a Random Control (picking random workers to swap, ignoring the labels), the model almost never changed its answer correctly.
- When they used an Influence Control (picking the "loudest" workers regardless of what they do), the model also failed to change the answer correctly.

5. Key Findings in Plain English

It Works: The system successfully identified which workers actually control specific facts. By grouping them, they made the model's "brain" readable.
Less is More: Sometimes, trying to control every part of the sentence (the city name, the state name, and the capital) actually confuses the model. It works better if you only tweak the specific parts related to the answer (the state and the capital).
It's Transparent: Unlike some AI methods that are "black boxes," this system uses clear, human-readable rules. If a researcher wants to check the work, they can see exactly why a worker was grouped with a specific team.

Summary

The paper presents a tool that turns a chaotic, unreadable map of a language model's brain into a clean, labeled diagram. It does this by testing how the model's parts react to slightly different questions, grouping similar parts together, and proving that these groups actually control the model's behavior by successfully swapping answers in a controlled experiment.

What the paper does NOT claim:

It does not claim this works for every type of AI task (like writing code or having a conversation); it was tested specifically on factual questions (like capitals, book authors, and company founders).
It does not claim to fix the model or make it smarter; it only helps humans understand how it works.
It does not claim to work on all languages or all model sizes yet; the tests were done on a specific English model (Gemma-2-2B).

Technical Summary: Automatically Interpreting Attribution Graphs via Probe Prompting

Problem Statement
While the computational pathways from input to output in Large Language Models (LLMs) are theoretically known, the sheer density of these computations renders them opaque to human interpretation. Existing methods for mechanistic interpretability, such as Sparse Autoencoders (SAEs) and Cross-Layer Transcoders (CLTs), successfully decompose dense model activations into sparse features. However, the resulting attribution graphs remain too large for manual analysis; a single short factual prompt can generate graphs with 600–5,000 nodes. Current automated interpretation techniques, which often rely on LLM scoring of corpus activations, fail to provide causal validation or circuit-level interaction insights. There is a critical need for automated, behavior-tied grouping methods that can scale from individual case studies to large catalogs of circuits without requiring manual tracing.

Methodology: Probe Prompting
The paper introduces probe prompting, a transparent, rule-based pipeline designed to convert complex attribution graphs into a compact set of concept-aligned "supernodes." The methodology proceeds through four distinct stages:

Attribution Graph Generation: Using Neuronpedia and CLT dictionaries, the system generates attribution graphs for seed prompts (e.g., "The capital of the state containing Dallas is").
Probe Prompt Generation: For each seed, an LLM generates a small set of concept-targeted probe prompts. These probes maintain the seed's syntactic structure (prepositions, answer position) but vary the content entities.
Cross-Prompt Activation Signatures (CPAS): The system measures feature activations across the probe set. Instead of relying on a single prompt peak (which is ambiguous), it aggregates metrics such as peak consistency, sparsity, and cosine similarity relative to the seed. This aggregation forms a CPAS for each feature.
Rule-Based Supernode Formation: A deterministic decision tree maps CPAS metrics to one of four functional roles:
- Semantic-Dictionary (SEM-DICT): Features consistently peaking on a single semantic token (e.g., "Texas").
- Semantic-Concept (SEM-CONC): Features peaking on a family of related tokens (e.g., artist names).
- Relationship (REL): Features with diffuse activation patterns encoding relations rather than entities.
- Say-X (SAY-X): Features peaking on functional tokens (e.g., "is", "the") in predictable positions relative to a semantic target, effectively promoting the output.
  Features sharing the same role and name are merged into a supernode.

Causal Validation Protocol
To validate the utility of these supernodes, the authors employ a causal intervention protocol involving entity-swap experiments. For a source-target pair (e.g., Dallas $\to$ Austin), the system ablates features associated with the source and amplifies features associated with the target. Success is measured by whether the model's prediction redirects to the target entity.

Crucially, the paper introduces two rigorous baselines to ensure the results are not artifacts of feature count or graph influence:

Matched-Random Control: A random set of features with the same count and layer distribution as the labeled set, drawn from outside concept-aligned supernodes.
Influence-Matched Top-K Control: A baseline that selects the top- $K$ features by graph influence to match the cumulative influence budget of the labeled intervention.

Key Results
The method was evaluated across four factual domains (USA state capitals, book characters, company founders, and painting artists) using the Gemma-2-2B model with a public CLT dictionary, totaling 44,596 intervention runs.

Superiority over Baselines: The labeled supernodes significantly outperformed both random and influence-matched baselines. In the USA domain, the labeled approach achieved a 72.8% Hit% (successful redirection) compared to 0.7% for matched-random and 4.2% for the influence-matched Top-K baseline.
Efficiency vs. Human Annotation: In a case study comparing the automated pipeline to a human-curated graph for the Dallas/Austin prompt, the automated method achieved comparable success rates (40/49 sources vs. 38/49 for human) but required a larger feature set (96.5 features vs. 4 features per call). However, the automated method successfully identified redirection paths that human curation missed for specific source states, demonstrating non-redundancy.
Field Additivity: The study found that "less is more." Interventions focusing on intermediate and answer fields (e.g., the state and the capital) were significantly more effective than including the input field (the city), which often encodes what the model reads rather than what it should produce.
Steering Strength: The optimal amplification multiplier ( $M$ ) was found to be bimodal (peaking around $M \approx 2.4$ and $M \approx 6.9$ ), challenging the standard $M=20$ default used in single-feature steering.

Significance and Claims
The paper claims that probe prompting demonstrates that circuit-level interpretability can be partially automated via a transparent, rule-based pipeline. Its primary contributions are:

Automation of Circuit Grouping: It provides a scalable method to group thousands of features into concept-aligned supernodes without manual intervention.
Causal Validation Framework: It establishes a robust, method-agnostic harness for evaluating grouping quality through matched-random and influence-matched controls, addressing the critique that interpretability scores can be achieved by randomly initialized models.
Operational Utility: The resulting labels are not just descriptive; they are operationally useful for performing entity swaps, serving as a rapid test for whether features are correctly clustered by their actual behavior in the computational graph.

The authors modestly note that their claims currently cover Levels 1 and 2 of causal abstraction (identifying features and their functional roles) but do not yet claim Level 3 (full mechanistic identification of latent variables). They acknowledge limitations, including the current restriction to two-hop factual recall, the freezing of attention during graph generation (though not during intervention), and the reliance on English functional token vocabularies. The work is presented as a reusable harness for calibrating future grouping methods against causal interventions.