🤖 AI

Belief or Circuitry? Causal Evidence for In-Context Graph Learning

This paper provides causal evidence that large language models learn in-context graph structures through a dual mechanism where genuine global topology inference and local pattern matching operate simultaneously, rather than relying on a single strategy.

Original authors: Katharine Kowalyshyn, Timothy Duggan, Daniel Little, Michael C Hughes

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Katharine Kowalyshyn, Timothy Duggan, Daniel Little, Michael C Hughes

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, but slightly mysterious, robot how to navigate a maze. You show it a few steps of a path, and then ask it to guess the next step. The big question researchers are asking is: How does the robot figure out the path?

Is it just memorizing the last few steps it saw (like a parrot repeating words)? Or is it actually building a mental map of the whole maze in its head (like a human understanding the layout)?

This paper, titled "Belief or Circuitry?", tries to answer that question using a Large Language Model (like the ones that power chatbots) and a game of "random walks" on two different types of maps: a Grid (like a city block with many intersections) and a Ring (like a circular track).

Here is what the researchers found, explained simply:

1. The "Belief" vs. "Copying" Debate

For a long time, people argued about how these AI models learn from context.

The "Copying" Theory: The model just looks at the last few words and says, "Oh, I've seen this word followed by that word before, so I'll guess that again." It's like a student who only studies the last page of the textbook before a test.
The "Belief" Theory: The model is actually inferring a hidden structure. It's thinking, "Based on these clues, I believe I am on a Ring track, not a Grid," and then uses that mental map to make predictions.

2. The Experiment: Mixing the Maps

The researchers didn't just give the model one map; they gave it a mixture. Imagine showing the robot a path that jumps back and forth between a Grid and a Ring.

If the robot were just a copier, it wouldn't care about the difference between the two shapes. It would just copy whatever it saw most recently.
If the robot were a believer, it would treat the two shapes differently. It would realize that the "Ring" is simpler and easier to learn, while the "Grid" is more complex and takes more evidence to figure out.

The Result: The robot acted like a believer. It took longer to "commit" to the complex Grid than the simple Ring. It wasn't just copying; it was weighing the complexity of the map it was on.

3. The "X-Ray" Test (Looking Inside the Brain)

To see how the robot was doing this, the researchers looked at the model's internal "brain waves" (mathematically, this is called the residual stream). They used a technique called PCA (think of it as an X-ray that flattens a 3D object into a 2D shadow to see its shape).

What they expected if it was just copying: The robot's brain would look like a blurry mix of the Grid and the Ring, mashed together.
What they actually saw: At the middle of the experiment, the robot's brain held two distinct, separate maps at the same time. One part of its brain held the "Grid" shape, and a completely different part held the "Ring" shape. They were like two different radio stations playing in separate rooms, not a static-filled mess.

This is strong proof that the model isn't just copying; it is maintaining a structured understanding of the world.

4. The "Remote Control" Test (Causal Proof)

Finally, the researchers wanted to know: "Does this mental map actually cause the robot to make the right guess, or is it just a side effect?"

They used two tricks:

The "Brain Swap" (Patching): They took the "brain state" from a robot that knew the answer and swapped it into a robot that was confused.
- Result: The confused robot suddenly knew the answer. This proves the information in the brain is what drives the prediction.
The "Nudge" (Steering): They gave the robot a tiny mathematical push in the direction of "Grid" or "Ring."
- Result: The robot's guesses shifted exactly in the direction they pushed. If they pushed it toward "Ring," it guessed Ring neighbors.

The Big Conclusion

The paper concludes that the answer isn't "Belief OR Circuitry." It's both.

Think of the AI like a detective solving a crime:

Circuitry (Copying): It looks at the immediate clues (the last few footprints).
Belief (Structure): It builds a theory about the suspect's profile (the map).

The model does both at the same time. It uses its "induction circuits" to copy recent patterns, but it also builds a genuine, internal model of the structure. These two systems work in parallel, and the final answer comes from combining the immediate clues with the big-picture theory.

In short: The AI isn't just a parrot repeating what it hears; it's a builder constructing a mental model of the world, even when that world is just a simple game of dots and lines.

Technical Summary: Belief or Circuitry? Causal Evidence for In-Context Graph Learning

1. Problem Statement

The paper addresses a fundamental debate in mechanistic interpretability regarding how Large Language Models (LLMs) perform in-context learning (ICL). Specifically, it investigates whether LLMs infer latent structural relationships (a "belief" mechanism akin to Bayesian inference over world models) or simply copy recent local transitions via induction circuits (a "circuitry" mechanism).

While recent work (Park et al., 2025) suggested LLMs maintain implicit probabilistic world models over graph structures, counter-arguments (Arditi, 2026; Ransome, 2026) posit that observed behaviors can be fully explained by local induction heads accumulating transition statistics without global structure inference. The authors aim to triangulate this debate using a controlled toy task involving random walks on competing graph structures.

2. Methodology

The study employs a multi-faceted approach combining behavioral modeling, representational analysis, and causal interventions on the Llama-3.1-8B model.

2.1. Task Design

The authors utilize a graph random-walk task where the model predicts the next node in a sequence generated by an unknown graph. Two competing hypotheses are tested:

$H_{grid}$ : A $4\times4$ grid graph (16 nodes, 24 edges, variable degree).
$H_{ring}$ : A 16-node ring graph (16 edges, uniform degree).

To create a genuine competition, the authors interleave random walks from both graphs at controlled mixture ratios ( $\rho$ ), where $\rho$ represents the probability of a segment being drawn from the ring.

2.2. Behavioral Modeling (Experiment 1)

The authors fit a belief-dynamics model to observed accuracy curves. Unlike previous flat log-prior approaches, they introduce a complexity-weighted prior:
$b_k = b_0 - \lambda \cdot C(H_k)$
Where $C(H_k)$ is the Minimum Description Length (MDL) complexity of the graph hypothesis (based on edge-list encoding).

Hypothesis: If the model performs structural inference, it should require more evidence to commit to the more complex graph ( $H_{grid}$ ), resulting in $\hat{\lambda} > 0$ .
Contrast: Pure induction circuits should accumulate transition statistics uniformly regardless of global topology, predicting $\hat{\lambda} \approx 0$ .

2.3. Representational Analysis (Experiment 2)

The authors analyze the residual stream activations (specifically layer 26) using Principal Component Analysis (PCA).

Metric: They measure degree-normalized Dirichlet energy ( $E_{norm}$ ) to quantify how well the low-dimensional geometry of node-token representations aligns with the true graph adjacency structure.
Goal: To determine if both graph topologies are simultaneously recoverable in orthogonal subspaces at intermediate mixture ratios, a signature inconsistent with a single blended representation from local copying.

2.4. Causal Interventions (Experiment 3)

To establish causality between internal representations and predictions, the authors perform two interventions:

Activation Patching: Replacing residual stream activations from a "corrupt" (mixed) prompt with those from a "clean" (single-graph) prompt at specific layers. They measure the recovery of the clean graph preference using a logit contrast score.
Activation Steering: Adding a learned "graph-difference vector" (derived from clean vs. ring contexts) to the residual stream to shift predictions toward a specific graph family.

3. Key Results

3.1. Behavioral Evidence

Complexity Sensitivity: The fitted model shows $\hat{\lambda} > 0$ and $\hat{b}_{grid} < \hat{b}_{ring}$ . The model requires disproportionately more context to commit to the denser grid topology compared to the ring.
Phase Transition: The inflection point for the grid shifts monotonically later as ring evidence increases.
Model Selection: A per-graph parameterization (allowing distinct dynamics for each graph) significantly outperforms a mixture-bias ablation (which only adjusts the prior) on AIC and BIC metrics. This suggests the model processes structural evidence differently for each topology, not just via a static prior.

3.2. Representational Evidence

Orthogonal Subspaces: At intermediate mixture ratios ( $\rho = 0.5$ ), PCA reveals that both the grid and ring topologies are simultaneously encoded in orthogonal principal subspaces.
Implication: A purely local induction mechanism would produce a single blended representation of co-occurrence statistics. The existence of two separable global structures contradicts the "copy-only" hypothesis.
Geometry Alignment: As context length increases past the inflection point, the internal geometry progressively mirrors the true topology, with $E_{norm}$ shifting from 0.785 to 0.828.

3.3. Causal Evidence

Patch Transfer: Late-layer patching (around layer 26–30) almost fully transfers the clean graph preference to the corrupt prompt. Crucially, this effect extends to held-out edges (neighbors never seen in the corrupt context), indicating the model is not merely replaying observed transitions but utilizing a structural representation.
Steering: Linear steering using the graph-difference vector successfully moves predictions in the intended direction. This effect is robust against norm-matched random vectors and shuffled-label controls, confirming the directionality is specific to the graph family signal.

4. Contributions

Complexity-Weighted Prior: Replaced flat priors with an MDL-inspired complexity measure, recovering a quantitative signature of topology-sensitive structural bias ( $\hat{\lambda} > 0$ ).
Interleaved Competition: Demonstrated that topology-biased evidence accumulation holds against the induction circuit prediction of symmetric behavior in a mixed-hypothesis regime.
Simultaneous Encoding: Provided PCA evidence that distinct global graph topologies are recoverable in orthogonal subspaces simultaneously, a pattern difficult to reconcile with local transition copying.
Causal Link: Established via patching and steering that these representational structures causally control next-token predictions, including for unseen transitions.

5. Significance and Conclusion

The paper concludes that the dichotomy between "belief" (global structure inference) and "circuitry" (local induction) is likely a false dichotomy. The evidence supports a dual-mechanism account:

Induction-like caches remain important, as evidenced by delayed effects on held-out edges and incomplete recovery under steering.
Latent-structure representations operate in parallel, integrating evidence before the final prediction.

The authors assert that the architecture of this coexistence—where genuine structure inference and induction circuits operate simultaneously—is the primary challenge for mechanistic interpretability moving forward. They do not claim that LLMs possess full human-like reasoning, but rather that they maintain implicit, manipulable probabilistic world models alongside local pattern-matching mechanisms.