🤖 AI

Query Circuits: Explaining How Language Models Answer User Prompts

This paper introduces "query circuits," a method for tracing input-specific information flow within language models to provide faithful, sparse, and computationally accessible explanations for individual outputs, validated by a new metric called Normalized Deviation Faithfulness (NDF) and demonstrated across multiple benchmarks.

Original authors: Tung-Yu Wu, Fazl Barez

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tung-Yu Wu, Fazl Barez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, bustling city with millions of roads, intersections, and traffic lights. When you ask the model a question (a "prompt"), it's like sending a delivery truck through this city to drop off an answer.

For a long time, researchers have tried to understand how this city works by mapping out general traffic patterns. They know, for example, that "Route A" is always used when the city needs to find a hidden object, or "Route B" is used for math problems. These are called capability circuits.

However, knowing the general route doesn't explain why the truck took a specific path for your specific delivery today. Maybe there was a construction zone, a sudden detour, or a unique traffic light that only mattered for this one trip.

This paper introduces a new way to look at the city: Query Circuits. Instead of looking at the general map, the authors want to trace the exact, tiny path the truck took for your specific question.

Here is a breakdown of their discovery using simple analogies:

1. The Problem: The "Surrogate" Map Was Wrong

Previously, to understand a specific trip, researchers tried to build a tiny, simplified model of the city (a "surrogate") and trace the path there.

The Analogy: Imagine trying to understand a complex traffic jam in New York City by building a toy model of the city out of LEGOs. You trace the path on the LEGOs, hoping it matches the real city.
The Issue: The paper argues that these LEGO models often don't match the real city perfectly. They miss details, and building them is slow and expensive. The authors wanted to trace the path directly on the real city (the actual model) without needing a toy version.

2. The Solution: The "Best-of-N" Lottery

The authors found that finding the exact path for one specific question is surprisingly hard. If you just look at the traffic lights for one question, the data is often "noisy" (like static on a radio), making it hard to see the true path.

The Analogy: Imagine you are trying to find a winning lottery ticket, but the ticket numbers are slightly blurry. If you try to pick the winner based on just one ticket, you might pick the wrong one.
The Fix: The authors propose a method called Best-of-N (BoN).
1. Take your original question.
2. Ask the model the same question but phrase it in 9 different ways (paraphrases). It's like asking, "What color is the sky?" vs. "Describe the color of the sky on a clear day."
3. Trace the path for all 10 versions.
4. Pick the winner: Choose the path that worked best.
The Result: Even though the original question was "blurry," one of the rephrased versions was clear enough to reveal the true, hidden path. By checking multiple versions, they found a "winning ticket" that accurately explains how the model answered.

3. The New Scorecard: NDF

To know if they found the right path, they needed a way to measure success. The old scorecard (called NFS) was broken for complex questions; it would give scores like "200%" or "-50%," which made no sense.

The Analogy: It's like a speedometer that sometimes says you are driving backward at 100 mph or forward at 500 mph when you are actually driving at 60.
The Fix: They invented a new score called NDF (Normalized Deviation Faithfulness). This is a reliable speedometer that always stays between 0 and 1.
- 1.0 means the path they found is perfect (it explains the model's answer exactly).
- 0.0 means the path is useless.
- This new scorecard allows them to trust their findings, even on difficult topics like medical exams or astronomy.

4. The Big Surprise: The City is Mostly Empty

The most exciting finding is that for any single question, the model doesn't use the whole city. It only uses a tiny, specific neighborhood.

The Analogy: You might think a delivery truck needs to use 50% of the city's roads to get from point A to point B. But the authors found that for a specific question, the truck only uses about 1.3% of the roads.
The Proof: They showed that if you block off 98.7% of the city and only let the truck use that tiny 1.3% path, it can still answer the question correctly about 60% of the time. This proves that the model's "brain" is incredibly sparse and efficient for individual tasks.

Summary

The paper introduces a new tool to explain exactly how an AI answers a specific question.

They stopped using "toy models" and looked directly inside the real AI.
They realized that asking the same question in different ways helps reveal the hidden "truth" of how the AI thinks.
They created a better way to measure if their explanation is correct.
They proved that for any single question, the AI only uses a tiny, specific fraction of its massive brain to get the job done.

This moves us from knowing "how the AI does math in general" to understanding "exactly how the AI solved this math problem."

Technical Summary: Query Circuits: Explaining How Language Models Answer User Prompts

1. Problem Definition

Current mechanistic interpretability research has largely focused on capability circuit discovery, which identifies global sub-networks responsible for specific algorithmic skills (e.g., indirect object identification or greater-than comparisons) across a dataset. While valuable, these methods do not explain why a Large Language Model (LLM) produces a specific output for a single, specific user input query (instance-level explanation).

Existing approaches to instance-level circuits rely heavily on surrogate models, such as Sparse Autoencoders (SAEs) or Cross-Layer Transcoders (CLTs). These surrogates are often used to approximate the LLM's behavior because they offer sparse activations that make circuit discovery easier. However, the paper argues that these surrogates frequently fail to faithfully reconstruct the original model's activations and may not capture the true computational mechanisms of the LLM. Furthermore, training these surrogates is computationally expensive, limiting their accessibility.

The paper introduces the task of Query Circuit Discovery: identifying a sparse, faithful sub-network directly within the original LLM (in-place) that drives the model's response to a single input query. This approach aims to provide local, input-level explanations without relying on potentially unfaithful surrogate models.

2. Methodology

The authors propose a framework to address two primary technical challenges in query circuit discovery: the instability of existing evaluation metrics and the inefficacy of standard discovery methods on single queries.

2.1. Evaluation Metric: Normalized Deviation Faithfulness (NDF)

The standard metric for circuit discovery, the Normalized Faithfulness Score (NFS), is shown to be unstable on general datasets (e.g., MMLU). NFS values often fluctuate wildly, exceeding 1 or dropping below 0, making it impossible to reliably track discovery progress or compare circuit quality.

To resolve this, the authors introduce Normalized Deviation Faithfulness (NDF):
$\text{NDF}(C_q) = 1 - \min\left( \left| \frac{L(M(q)) - L(C_q(q))}{L(M(q)) - L(M(q'))} \right|, 1 \right)$
Where:

$M$ is the target LLM.
$C_q$ is the discovered query circuit.
$q$ is the original query, and $q'$ is a corrupted query (e.g., with key factual cues removed).
$L(\cdot)$ is a performance metric (e.g., logit difference).

Key Properties of NDF:

Bounded: It is strictly bounded within $[0, 1]$ .
Symmetric: It penalizes deviations above and below the model's performance equally.
Interpretability: A score of 1 indicates the circuit perfectly recovers the model's performance; 0 indicates the deviation is as large as the gap between the original and corrupted queries.

2.2. Discovery Method: Best-of-N (BoN) Sampling

Standard edge-scoring methods (like Edge Attribution Patching, EAP) often fail to find compact, faithful circuits for single queries due to gradient noise and the neglect of combinatorial effects among edges. The authors observe that while a circuit discovered on a specific query might fail, circuits discovered on paraphrases of that query often succeed.

Based on this, they propose Best-of-N (BoN) Sampling:

Paraphrasing: Generate $p$ paraphrases of the original query $q$ .
Scoring: Compute edge importance scores (using EAP-IG) for the original query and all paraphrases, resulting in $p+1$ edge score matrices.
Selection: Construct $p+1$ candidate circuits (one for each score matrix) and select the one with the highest NDF score.

To improve efficiency and avoid running the LLM $p+1$ times for every edge budget, the authors propose two variants:

Interpolated BoN (iBoN): Interpolates between two previously discovered faithful circuits of different sizes to form a new circuit without additional forward passes.
BoN with Constraint-Adaptive Score Matrix (BoN-CSM): Aggregates score matrices from all previously discovered circuits into a unified score matrix and tier matrix, allowing for the efficient construction of new circuits of varying sizes.

3. Key Contributions

Task Formulation: The paper formally defines Query Circuit Discovery, distinguishing it from global capability circuit discovery and surrogate-based instance-level approaches. It emphasizes tracing information flow directly within the original model.
Robust Evaluation (NDF): The authors identify the instability of NFS on general datasets and propose NDF as a more reliable, bounded, and symmetric metric for evaluating query circuits.
Efficient Discovery (BoN): They introduce Best-of-N sampling and its variants (iBoN, BoN-CSM) to overcome the limitations of gradient noise and combinatorial effects, enabling the discovery of compact, faithful circuits for complex natural language queries.
Empirical Validation: The paper demonstrates that sparse query circuits exist within LLMs that recover a significant portion of model performance on individual queries, even for complex tasks like MMLU and ARC Challenge.

4. Experimental Results

The authors evaluated their methods across multiple benchmarks: Indirect Object Identification (IOI), arithmetic tasks, ARC Challenge, and nine categories of MMLU.

Performance: BoN sampling consistently outperformed baselines (single-query scoring and averaging scores over paraphrases).
Sparsity: On MMLU, a query circuit containing only 1.3% of the model's total edges (approx. 5,000 edges in Llama-3.2-1B-Instruct) achieved an average NDF of 0.6, recovering roughly 60% of the model's behavior on that query. In contrast, standard EAP-IG required significantly more edges (up to 51.7% of the model) to reach similar performance levels.
Shared Sub-circuits: Analysis revealed that while query circuits vary based on paraphrasing, they share a core set of edges (a "shared sub-circuit") that corresponds to the underlying capability, supporting the hypothesis that these circuits are not random "lottery tickets" but variants of a common mechanism.
Actionability: In experiments involving gender bias, circuits with higher NDF scores were found to be more actionable. Zeroing out features identified by high-NDF circuits resulted in significantly greater bias reduction compared to low-NDF circuits, suggesting NDF is a reliable proxy for the circuit's ability to capture meaningful computations.

5. Significance and Claims

The paper claims that Query Circuit Discovery represents a significant step toward faithful and scalable explanations of how language models process individual inputs.

Faithfulness: By operating directly within the model (in-place) rather than relying on surrogates, the explanations are more faithful to the model's actual computational mechanisms.
Scalability: The proposed BoN sampling methods allow for the discovery of these circuits without the prohibitive computational cost of training surrogate models or running excessive forward passes.
Practicality: The discovery of sparse sub-networks (e.g., 1.3% of connections) that explain a majority of the model's behavior on specific queries suggests that LLMs process individual inputs through highly localized pathways, making them amenable to detailed mechanistic analysis.

The authors conclude that while challenges remain (such as handling multi-token generation and fully automating human-readable labeling), their work establishes a practical path for explaining LLM decisions at the prompt level, which is essential for deployment in high-stakes domains like medicine and autonomous systems.