💬 NLP

ADAG: Automatically Describing Attribution Graphs

The paper introduces ADAG, an automated end-to-end pipeline that uses attribution profiles, novel clustering, and an LLM explainer-simulator to generate natural-language descriptions of attribution graphs, successfully recovering interpretable circuits and identifying harmful feature clusters in language models without relying on manual human inspection.

Original authors: Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, Sarah Schwettmann

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt, Sarah Schwettmann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (like the one powering this chat) as a giant, bustling city. Inside this city, millions of tiny workers (neurons) are constantly passing notes, shouting instructions, and making decisions to produce the final answer you see on your screen.

For a long time, researchers knew which workers were involved in a specific task (like answering a geography question), but they didn't know what those workers were actually thinking or saying. They had to manually inspect the workers, read their notes, and guess their roles. It was like trying to understand a complex play by watching the actors from the back of the theater and guessing their lines.

ADAG (Automatically Describing Attribution Graphs) is a new tool that acts like a super-smart, automated documentary crew. It doesn't just find the workers; it interviews them, groups them into teams, and writes a clear, plain-English script explaining exactly what each team does.

Here is how ADAG works, broken down into simple steps:

1. The "Who Did What?" Map (Circuit Tracing)

First, ADAG looks at a specific question (e.g., "What is the capital of Texas?") and traces the path of the "electricity" through the city. It identifies exactly which neurons fired to produce the answer "Austin."

The Old Way: Researchers would manually look at the list of fired neurons and try to guess, "Oh, this one probably knows about Texas."
The ADAG Way: It automatically draws a map showing exactly which neurons contributed and how strongly.

2. The "Resume" (Attribution Profiles)

Instead of just looking at when a neuron fired, ADAG looks at its resume. It asks two questions for every neuron:

Input: What kind of words in the question made this neuron wake up? (e.g., "Did it wake up because it saw the word 'Dallas'?")
Output: What kind of words did this neuron help push toward the final answer? (e.g., "Did it help push the word 'Austin' or 'Texas'?")

This creates a detailed "profile" for every neuron, showing its strengths and weaknesses.

3. The "Team Building" (Clustering)

In a city of millions, you can't talk to every worker individually. ADAG groups similar workers into teams (supernodes).

It uses a smart algorithm to say, "Hey, these 20 neurons all wake up when they see 'Dallas' and they all push toward 'Texas.' Let's put them in the same team."
This turns a chaotic list of 1,000 neurons into a neat list of 20 distinct teams, each with a specific job.

4. The "Translator" (LLM Explainer)

Now comes the magic. ADAG takes these teams and asks a second AI (a language model) to act as a translator.

The Explainer: "Based on this team's resume, what do you think their job is?" It guesses a label like "Texas City Detector."
The Simulator: "Okay, let's test that label. If this team is really a 'Texas City Detector,' what would happen if we showed them a city in France?" The simulator checks if the guess makes sense.
If the guess fails, the Explainer tries again. If it passes, ADAG keeps the label.

Why This Matters: Two Real-World Examples

Example 1: The Geography Quiz
When asked, "What is the capital of the state containing Dallas?", ADAG automatically found a team of neurons labeled "Dallas Texas."

The Discovery: When the researchers turned off this specific team, the model stopped saying "Austin" and started guessing "Oklahoma" or "Texas" instead.
The Result: ADAG proved, without human help, that this specific team of neurons is the "Dallas-to-Texas" bridge in the model's brain.

Example 2: The "Jailbreak" (Harmful Advice)
Researchers found a tricky prompt that tricks the model into giving dangerous medical advice (like "take all five pills at once").

The Discovery: ADAG scanned the model's brain while it was being tricked. It found a team of neurons labeled "Ridiculous-to-Introductory."
The Fix: When they boosted the activity of this specific team, the model suddenly started giving safe advice again. When they silenced it, the model became dangerous.
The Impact: This helps us understand exactly how models get tricked and how to patch the holes automatically.

The Big Picture

Before ADAG, understanding how AI thinks was like trying to fix a car engine by guessing which bolt to tighten. You had to be an expert, spend hours, and often guess wrong.

ADAG is like a mechanic's diagnostic computer. You plug it in, and it instantly tells you:

"The 'Fuel Injection Team' is working hard."
"The 'Brake Sensor Team' is confused."
"Here is a simple sentence explaining what's wrong."

By automating this process, ADAG allows us to scale up. We can now check the "brains" of massive AI models to ensure they are safe, honest, and doing what we want them to do, without needing a team of human detectives to stare at data for weeks.

1. Problem Statement

In the field of Large Language Model (LLM) interpretability, circuit tracing aims to identify the specific internal features (e.g., neurons, attention heads) that causally contribute to a model's output and map their interactions. While methods exist to extract these "circuits" (subgraphs of the computation graph), a critical bottleneck remains: human legibility.

Currently, interpreting these circuits is an ad-hoc, manual process. Researchers must inspect data artifacts (like maximum-activating examples) and perform targeted causal interventions to hypothesize what a feature does. This process is:

Non-scalable: It does not scale to large models or massive datasets.
Subjective: Relies heavily on researcher intuition.
Incomplete: Often fails to capture non-local dependencies (how a feature affects tokens far from its activation point).

The paper addresses the need for a fully automated, end-to-end pipeline that can trace circuits, group features into meaningful functional units (supernodes), and generate natural language descriptions of their roles without human intervention.

2. Methodology: The ADAG Pipeline

The authors propose ADAG (Automatically Describing Attribution Graphs), a four-stage automated pipeline:

A. Circuit Tracing Backbone

Goal: Construct an attribution graph $G=(V, E)$ identifying important features and their interaction weights.
Technique: Uses a locally linear replacement model combined with RelP (Relevance Patching).
- RelP freezes nonlinearities (like SiLU, softmax) to behave as constant multipliers based on a single input.
- It stops gradients through query/key paths in self-attention and halves gradients via element-wise multiplications.
- Attribution scores ( $\alpha$ ) are computed for MLP neurons by backpropagating from the sum of top- $k$ output logits.
- Features exceeding a threshold $\tau$ are retained to form the circuit.

B. Attribution Profiles (Quantifying Functional Roles)

To overcome the "locality bias" of previous methods (which only look at the token where a feature fires), ADAG introduces Attribution Profiles. For a feature $f$ and context $x$ , this consists of two vectors:

Input Attribution: The proportion of the feature's activation attributed to each preceding input token (gradient-based).
Output Contribution: The proportion of each future output logit's activation contributed by the feature.

Significance: This captures non-locality, revealing how a neuron influences tokens far in the future or depends on tokens far in the past.

C. Automatic Clustering (Supernode Formation)

Features are grouped into supernodes (clusters) that share functional roles.

Similarity Metric: Uses a multi-view spectral clustering approach.
- Computes pairwise cosine similarities for both Input Attribution and Output Contribution across all contexts.
- Aggregates these using a harmonic mean of clamped non-negative similarities. This penalizes anti-correlation (ensuring features with opposing output effects aren't clustered together).
Algorithm: Applies spectral clustering on the aggregated similarity matrix to partition features into $k$ clusters.
Optimization: The method is tuned to maximize the Silhouette Score (cohesion), minimize the Coefficient of Variation (balance), and minimize Opposing Sign Intra-cluster Contributions.

D. Automatic Natural Language Description

Once supernodes are formed, ADAG describes their roles using an Explainer-Simulator framework (inspired by Bills et al., 2023):

Explainer LM: Takes the aggregated attribution profiles (input and output) and generates candidate natural language labels.
- Input Attribution: Uses a fine-tuned Llama 3 explainer (highlighting tokens in {{}}).
- Output Contribution: Uses an API-based LLM (Claude) to generate labels based on logit contribution scores.
Simulator LM: Scores the candidate labels by predicting the feature's activation/contribution scores given the description.
Selection: The label with the highest Pearson correlation ( $r$ ) between predicted and true scores is selected.
Summarization: A final LLM summarizes the best input and output descriptions into a concise label.

3. Key Contributions

First End-to-End Automated Circuit Interpreter: ADAG is the first system to fully automate the pipeline from circuit tracing to natural language description, removing the need for manual human inspection.
Attribution Profiles: Introduces a novel representation that quantifies feature roles via gradient-based input/output dependencies, effectively solving the locality bias problem.
Novel Clustering Algorithm: A multi-view spectral clustering method specifically designed to group features by functional similarity while preventing the mixing of features with opposing causal effects.
Validation of Explainer-Simulator: Demonstrates that LLMs can effectively score and select descriptions for circuit components, achieving performance comparable to or better than human experts in specific tasks.

4. Experimental Results

The authors evaluated ADAG on two primary datasets using Llama 3.1 8B Instruct:

A. Multi-hop State Capitals (Capacitals Dataset)

Task: Answering questions like "What is the capital of the state containing Dallas?"
Results:
- ADAG successfully recovered the known human-verified circuit structure (City $\to$ State $\to$ Capital).
- Steering Experiments: Ablating specific clusters (e.g., "Dallas Texas" cluster) caused the model's output probability for "Austin" to drop significantly (from 96.5% to 52.7%) while "Oklahoma" rose, confirming the causal role of the identified clusters.
- Description Quality: LLM-generated descriptions for gold-standard supernodes scored nearly as high as human-generated descriptions for input attribution and significantly higher for output contribution.

B. Harmful Medical Advice Jailbreak (Pills Dataset)

Task: Analyzing a jailbreak prompt that tricks the model into giving dangerous medical advice.
Results:
- ADAG identified specific clusters responsible for the jailbreak success.
- Cluster C3 ("pills safety redirect"): Negatively steering this cluster (setting activation to 0) increased the Attack Success Rate (ASR) from ~28% to 88%.
- Cluster C9 ("ridiculous-to-introductory"): Positively steering this cluster increased ASR to 90%.
- This demonstrates ADAG's ability to find "steerable" clusters that control safety behaviors in real-world adversarial scenarios.

C. Non-Localality Findings

Experiments confirmed that MLP neurons exhibit significant non-locality. In deeper layers, input attribution is often distributed across many preceding tokens, and output contributions often originate from tokens far before the target, validating the necessity of the Attribution Profile approach.

5. Significance and Impact

Scalability: By automating the interpretation process, ADAG enables the analysis of circuits in much larger models and datasets, which is currently infeasible with manual methods.
Safety & Alignment: The ability to automatically identify and describe clusters responsible for harmful behaviors (like the medical jailbreak) provides a powerful tool for AI safety researchers to detect, understand, and mitigate risks in LLMs.
Bridging the Gap: The paper argues that even if we can reverse-engineer model internals into clean mathematical units, LLMs are essential to bridge the gap between these formal descriptions and human-understandable language. ADAG operationalizes this insight.
Reproducibility: The system is open-source, providing a standardized framework for future interpretability research.

In conclusion, ADAG represents a major step forward in mechanistic interpretability, moving from manual, case-by-case circuit analysis to a scalable, automated system capable of explaining the "why" and "how" of LLM computations.