💬 NLP

GraphGhost: Tracing Structures Behind Large Language Models

This paper introduces GraphGhost, a novel graph-based framework that models internal token interactions and neuron activations in Large Language Models to reveal the global information flow and structural patterns underlying their multi-step reasoning capabilities.

Original authors: Xinnan Dai, Xianxuan Long, Chung-Hsiang Lo, Kai Guo, Shenglai Zeng, Dongsheng Luo, Jiliang Tang

Published 2026-02-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinnan Dai, Xianxuan Long, Chung-Hsiang Lo, Kai Guo, Shenglai Zeng, Dongsheng Luo, Jiliang Tang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, bustling city where millions of tiny workers (neurons) pass notes (tokens) to each other to solve a problem. Usually, when we try to understand how this city works, we only look at the final note handed to the customer. We might say, "Ah, the word 'therefore' was important," or "The number '5' mattered." But this is like trying to understand a complex play by only reading the final line of the script. It tells you the ending, but not how the actors got there.

The paper GraphGhost introduces a new way to peek behind the curtain. Instead of just looking at the final note, it maps out the entire secret network of connections the city uses to think.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: Looking at the Wrong Map

Current methods are like taking a photo of a single brick in a wall and saying, "This brick holds up the house." It's true, but it misses the whole structure. The paper argues that reasoning isn't just about one word; it's about how words and neurons dance together in a specific pattern.

2. The Solution: GraphGhost

The authors built a tool called GraphGhost. Think of it as a "ghostly X-ray" that turns the invisible flow of information inside the AI into a visible map (or graph).

The Nodes: These are the specific words (tokens) and the neurons (workers) that light up.
The Edges: These are the invisible wires connecting them, showing who talked to whom and in what order.

3. Two Ways to Look at the Ghost

The paper says we can look at this map in two different ways:

The "Sample View" (The Single Story):
Imagine you are watching a single person solve a math problem. The Sample View draws a map of exactly how that specific person's brain connected the dots for that one problem. It traces the path from the question to the answer, showing which words were merged and which neurons fired to get the result.
- Analogy: It's like drawing the exact route a delivery driver took to get a package to your door, noting every turn and traffic light.
The "Dataset View" (The City's Habit):
Now, imagine watching thousands of people solve thousands of problems. The Dataset View looks for recurring patterns. It asks, "Do people always use the same shortcut for math problems? Is there a specific 'thinking corner' in the city that everyone visits when they see the word 'Therefore'?"
- Analogy: It's like a city planner looking at traffic data to see that every morning, 90% of cars take the same highway to get to work. This reveals the "habitual" structure of the AI's brain.

4. What They Discovered

By analyzing these maps, the researchers found some surprising things:

The "Merge" Points: Just like a river where many small streams come together to form a big one, the AI has specific layers where information from different words merges into a single idea. For example, the words "1" and "0" might merge in the middle layers to become the concept of "10."
The "Trigger" Points: There are specific neurons that act like light switches. If you flip them, the whole reasoning process changes.
- The Experiment: The researchers tested this by "muting" (turning off) specific neurons in the map.
- Result 1 (Language Switch): When they turned off a neuron associated with the word "the" in the middle of the process, the AI suddenly stopped speaking English and started speaking Chinese! It was as if they cut the wire to the English department and the Chinese department took over.
- Result 2 (Logic Fix): In another case, turning off a specific "period" (.) neuron caused the AI to insert a new logical step ("So...") that it had missed before, actually helping it get the right answer.

5. The Big Takeaway

The paper concludes that the AI isn't just a black box guessing words. It has a real, structural skeleton. The way the AI reasons is tightly bound to the shape of these internal maps. If you change the structure (by messing with the key nodes), you change the logic.

In short: GraphGhost is a tool that turns the AI's invisible, chaotic thinking process into a clear, visual map, showing us that the AI's "intelligence" is built on specific, repeatable structural patterns, not just random word associations.

Technical Summary: GraphGhost: Tracing Structures Behind Large Language Models

Problem Statement

Large Language Models (LLMs) demonstrate strong capabilities in structured reasoning tasks (e.g., mathematical problem solving, logical inference), yet the internal mechanisms driving these behaviors remain poorly understood. Existing interpretability methods primarily focus on token-level attributions, which trace individual predictions back to specific input tokens or activations. While useful for localized explanations, these approaches fail to capture the multi-step reasoning processes and the global information flow within the model. Consequently, there is a lack of insight into how representations are composed, merged, and reused across reasoning steps, creating a gap between faithful internal mechanism understanding and human-interpretable reasoning logic.

Methodology: GraphGhost

To address these limitations, the authors propose GraphGhost, a graph-based framework that models internal token interactions and neuron activations in LLMs as structured graphs. The framework operates from two complementary perspectives:

1. Sample View (Instance-Specific)

This view traces internal token dependencies within a single input–output pair to explain a specific prediction.

Mechanism: Starting from the final answer token, the method iteratively expands backward through the model's internal computation using circuit-tracing techniques (specifically utilizing transcoders to decompose dense MLP activations into sparse, interpretable features).
Construction: It aggregates local logit-based attribution graphs. Nodes represent contributing tokens (or transcoder features), and directed edges represent dependency relations.
Output: An unweighted structural graph ( $G_{sample}$ ) that captures the existence of dependencies, alongside a local edge-weight map ( $W_{local}$ ) that accumulates contribution strengths for later aggregation. This reveals how intermediate tokens are merged and propagated during sequential generation.

2. Dataset View (Generalizable Patterns)

This view aggregates structures across multiple samples to identify recurring, shared token-level patterns learned during training.

Mechanism: Sample-level GraphGhosts from a dataset are merged via set union to form a unified graph ( $G_{data}$ ).
Aggregation: Edge weights are accumulated across samples to quantify the frequency and strength of recurring dependencies. A row-stochastic normalization is applied to determine relative edge importance at the dataset level.
Output: A graph capturing reusable reasoning primitives and task-specific structures, revealing how the model generalizes across different domains (e.g., mathematical vs. logical reasoning).

Graph Analytics and Intervention

The framework employs graph analytics to characterize the model's internal state:

Semantic Merging: Analyzed via node in-degree, identifying tokens where information from multiple sources converges (e.g., merging "1" and "0" into a unified "10" representation).
Reasoning Triggers: Identified using PageRank on reversed edges to find influential nodes that initiate reasoning steps (e.g., tokens like "So" or "Step").
Intervention: The authors perform targeted perturbations (muting specific neuron nodes associated with critical tokens) to observe causal changes in reasoning trajectories and output.

Key Contributions

Unified Framework: The proposal of GraphGhost, which models internal token interactions and neuron activations at both the sample and dataset levels, moving beyond isolated token attributions.
Structural Insights: The revelation that internal graphs capture structural constraints governing model behavior, specifically shaping information composition and propagation during reasoning.
Empirical Validation: Quantitative experiments demonstrating that structurally important token-related neuron nodes strongly influence reasoning trajectories. Interventions on these nodes result in measurable changes in both language expression (e.g., language switching) and logical flow.

Experimental Results

The authors evaluated GraphGhost on multiple models (Qwen3-0.6B, Qwen2.5-1.5B, Llama3.1-8B-Instruct, and DeepSeek-distilled variants) across three reasoning domains: mathematical (GSM8K, MAWPS), logical (ProntoQA, BoolQA), and scientific (ARC-Easy, QASC).

Structural Properties: Graph structural properties were found to be closely associated with influential tokens and neuron nodes. High in-degree nodes (merging points) and high PageRank nodes (triggers) were identified as structurally critical.
Perturbation Effects:
- Semantic Control: Muting a neuron node associated with the token "the" at a middle layer (Layer 20) disrupted the English semantic pathway, causing the model to generate Chinese text (a direct translation), demonstrating the node's role in language maintenance.
- Logic Alteration: Muting a neuron node associated with the token "." at a higher layer (Layer 25) altered the reasoning flow, introducing a new trigger token ("So") that led the model to produce a correct logical conclusion where it previously failed.
Generalization: The dataset view revealed that while certain tokens (e.g., ".", "the") are shared across domains as universal reasoning primitives, others are domain-specific (e.g., arithmetic tokens in GSM8K/MAWPS vs. choice tokens in ARC).

Significance and Claims

The paper claims that the structural patterns captured by GraphGhost reflect the meaningful internal organization of LLM reasoning. By formalizing reasoning as a graph of token dependencies, the framework provides a principled way to understand the source of intelligence in LLMs. The results suggest that:

Reasoning is not merely a sequence of isolated token predictions but is driven by structured interactions and global information flow.
Internal graph structures fundamentally shape how individual tokens influence reasoning outcomes.
Faithful interpretability requires aligning internal reasoning mechanisms with human-interpretable logic, which GraphGhost facilitates by exposing the "mechanistic" process of how intermediate tokens are merged and reused.

The authors conclude that GraphGhost offers a robust method for mechanistic analysis, bridging the gap between local token explanations and the global structural constraints that govern LLM behavior.