💬 NLP

Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams

This paper demonstrates that the directed dependency graph of tool-call trajectories in LLM agents is linearly decodable from the model's residual streams, revealing that the network actively represents abstract execution topology rather than merely tracking positional order or identifier values.

Original authors: Tianda Sun, Dimitar Kazakov

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tianda Sun, Dimitar Kazakov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Reading the "Ghost" of a Plan

Imagine a large language model (LLM) as a very smart, but slightly forgetful, assistant. When you ask this assistant to do a complex task—like "Find a user, check their order, and then exchange the product"—it doesn't just do it in one giant leap. It has to take several steps, calling different tools along the way.

The paper asks a simple question: Does the model "know" how these steps connect to each other while it is thinking?

Specifically, if Step A gives information that Step B needs, does the model's internal brain (its "residual stream") actually hold a map showing that connection? Or is the model just guessing the next step based on what it said last?

The researchers found that yes, the model does hold this map. They built a tiny, simple "decoder" that can read the model's internal thoughts and reveal the hidden dependency graph (the map of who needs what from whom) with high accuracy.

The Analogy: The Construction Site Blueprint

Think of the model as a construction site.

The Tools: The model uses tools like "Get User" or "Get Order."
The Trajectory: The sequence of actions the model takes.
The Dependency Graph: This is the blueprint. It says, "You can't pour the concrete (Step B) until you have the cement delivery receipt (Step A)."

Usually, we only see the finished building (the final answer). We don't see the blueprint inside the workers' heads. This paper is like putting on special X-ray glasses that let us see the blueprint floating inside the workers' minds as they work.

How They Did It (The "Decoder")

The researchers used a model called Qwen3-32B. They watched it solve problems and recorded its internal "residual stream" (a continuous stream of data representing its current state of mind).

They trained a very small, simple tool (a "probe") to look at this stream and answer a binary question for any two steps in the process:

"Does the output of Step A feed into the input of Step B?"

The Results:

It works: The probe could predict these connections with about 87% accuracy.
It's not a trick: They tested against random guesses and simple patterns (like "Step 2 always follows Step 1"). The probe did much better than those, proving it was actually reading the logic of the connection, not just the order of events.
It's abstract: When they changed the specific numbers in the data (e.g., changing a user ID from "123" to "456") but kept the structure the same, the probe still worked. This means the model understands the relationship (A depends on B), not just the specific words.

The "Ghost" in the Machine (Propagation)

One of the coolest findings is about how this information travels.

Imagine you whisper a secret to the first worker (Step A). The paper shows that this secret doesn't just disappear after the worker finishes. It travels through the entire construction site, lingering in the "air" (the residual stream) all the way to the last worker (Step Z).

They proved this by "patching" (swapping) the internal state of the first worker with a different scenario. Even though the model didn't change its final behavior (it still built the same house), the "blueprint" inside the later workers changed to match the new scenario. This suggests the model is actively carrying the structural plan forward, not just reacting to the immediate prompt.

When Does This Map Disappear?

The researchers tested this on different types of tasks to see if the map is always there. They found a pattern:

Complex Chains (Multi-hop): If the task requires a long chain of dependencies (A $\to$ B $\to$ C $\to$ D), the map is very clear and easy to read.
Simple Lists: If the task is just a list of independent steps (A, then B, then C, with no connections), the map vanishes.
The "Order" Clue: If the steps are so simple that just knowing the order tells you everything (e.g., "First do A, then B"), the model doesn't need to build a complex internal map. The "extra" signal disappears because the position alone is enough.

What This Means (and What It Doesn't)

What it means:
We have found a new way to look inside AI agents. We can see that they aren't just blindly guessing the next word; they are maintaining a structured, abstract map of how their actions depend on one another. This map is linear (easy to read) and travels through the model's layers.

What it does NOT mean (based strictly on this paper):

It does not mean we can now easily control the model to make better decisions (the paper explicitly says the patching changed the internal map but not the final behavior).
It does not mean this works on every single AI model or every tiny model (they only tested specific large models).
It does not mean we can use this to fix "hallucinations" or make the model safer yet; that is future work.

Summary

The paper is like discovering that a magician's assistant isn't just memorizing a script, but is actually holding a mental map of the entire trick's logic. By using a simple decoder, the researchers proved that this map exists, travels through the model's brain, and is essential for solving complex, multi-step problems.

Technical Summary: Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams

Problem Statement

While mechanistic interpretability has successfully characterized how Large Language Models (LLMs) represent static structures (e.g., syntactic trees, board states, factual knowledge), it remains unclear whether the dynamic execution structures of tool-using agents are encoded within the model's internal states. Specifically, when an agent interleaves generation with external function calls, earlier tool outputs often supply arguments to later calls, forming a directed acyclic graph (DAG) of dependencies. It is unknown whether this runtime dependency graph is explicitly represented in the agent's residual stream, whether such a representation is causal, or if it is a general property of agents or specific to certain benchmarks.

Methodology

The authors investigate this question using a structural probing approach on the Qwen3-32B model (and replicated on Llama-3.3-70B) within the $\tau$ -bench retail domain.

Oracle Construction: For trajectories containing $\ge 2$ tool calls, the authors construct an "oracle" tool-call DAG. An edge $i \to j$ is defined if a contiguous, case-sensitive, whitespace-normalized substring (length $\ge 4$ ) of call $i$ 's output appears verbatim in the serialized JSON arguments of call $j$ . This defines a binary edge label for the probe.
Probe Architecture: A low-capacity, L2-regularized logistic regression probe is trained to predict the existence of an edge $(i, j)$ given the concatenation of residual stream vectors from the terminating boundary tokens of calls $i$ and $j$ . The vectors are mean-pooled across a fixed set of layers $\{0, 14, 28, 41, 50, 57, 64\}$ .
Evaluation Protocol:
- Cross-Validation: Per-trajectory Leave-One-Group-Out (LOGO) cross-validation ensures no token from a held-out trajectory enters its training fold.
- Controls: The study employs a comprehensive battery of controls:
  - Hewitt–Liang Random-Label Control: Permuting edge labels to rule out memorization.
  - Positional Baseline: A 5-feature baseline ( $[i, j, j-i, n_{agent}, j/n_{agent}]$ ) to separate structural encoding from simple call ordering.
  - Surface-Form Ceiling: A decoder based on n-gram overlap and tool names to ensure the signal is not merely textual.
  - Random-Init Control: Using a randomly initialized model to isolate training-attributable signals.
- Counterfactuals: The authors perform value corruption (swapping IDs in outputs) and structural perturbation (rewriting outputs to empty results) to test if the probe tracks abstract topology or specific identifier values.
- Activation Patching: To test causality, the authors patch the residual stream of a source call with features from a "donor" trajectory and measure the shift in the probe's readout at a later, non-patched boundary.

Key Contributions and Results

1. Linear Decodability of the Dependency Graph

The tool-call dependency graph is linearly decodable from the frozen residual stream.

Performance: The multi-layer probe achieves an AUROC of 0.869 on held-out pairs.
Significance: This significantly outperforms the Hewitt–Liang random-label control (max AUROC 0.565, $p < 10^{-4}$ ) and the positional baseline (AUROC 0.792). The residual stream contributes a conditional AUROC gap of +0.0775 over the positional baseline, confirming the signal is not merely a proxy for call order.
Emergence: The signal emerges sharply by Layer 14 (~22% depth) and saturates around Layer 41, matching patterns of mid-stack semantic emergence.

2. Topology Sensitivity vs. Value Specificity

The probe tracks abstract dependency topology rather than specific identifier values.

Value Corruption: Swapping ID values in tool outputs (while keeping the plan fixed) results in no statistically significant shift in the decoded DAG structure (Cohen's $d = 0.06$ ).
Structural Perturbation: Rewriting a tool response to an empty result (forcing a plan change) causes a large shift in the decoded structure (Cohen's $d = 0.87$ ).
Replication: This dissociation holds across domains, including ComplexFuncBench, and is reproduced using an independent, schema-typed value-equality oracle (no substring matching), confirming the signal is not an artifact of the substring heuristic.

3. Representational Propagation

Activation patching provides evidence that the dependency representation propagates through the network rather than being a passive readout.

Patching the residual stream at a source call shifts the probe's prediction at a strictly later, non-patched boundary (e.g., at Layer 57).
Crucially, while the representation shifts, the agent's realized tool choice does not move (0% shift in behavior). This suggests the representation is load-bearing for the internal state but not sufficient for behavioral control in this context.

4. Cross-Domain Generality and Modulation

The non-positional contribution generalizes across interactive multi-hop benchmarks but attenuates where call order alone suffices.

Positive Replications: The signal is strong in ToolHop (+0.161 contribution) and ComplexFuncBench (+0.060 contribution).
Transitive Edges: The probe decodes transitive (multi-hop) dependencies even more strongly (AUROC 0.986) than direct edges, with the transitive signal rising faster in early layers.
Attenuation: The contribution vanishes in TaskBench (single-shot planning, contribution -0.02) and is untestable in BFCL (direct-only tool use), supporting the hypothesis that the signal exists where call order is a weak proxy for dependency.

Significance and Claims

The authors claim this is the first structural probe of an LLM agent's runtime tool-call dependency graph.

Representation vs. Control: The claims are strictly about representation. The paper demonstrates that the dependency graph is linearly readable and causally propagated within the residual stream, but explicitly states that activation patching does not currently yield behavioral control (the agent's tool choices remain unchanged).
New Target Class: The results suggest that runtime execution structures (DAGs of tool calls) constitute a new class of mechanistic interpretability targets, distinct from token-level decisions or static reasoning traces.
Modesty: The authors are modest about the scope of propagation, noting that cross-domain causal replication and behaviorally-sufficient interventions remain future work. They also acknowledge that the dependency definition relies on value-reuse (substring or schema matching), excluding dependencies mediated purely by control flow without value transfer.

In summary, the paper establishes that LLM agents encode the abstract topology of their tool-call execution graphs in their residual streams, a representation that is distinct from positional priors, robust to value changes, and propagates through the network layers.