Technical Summary: Probing Neural Topology of Large Language Models

Problem Statement

While probing large language models (LLMs) by linking neural activations to interpretable semantics has yielded insights into internal mechanisms, the complex relationship between neural functional co-activation and emergent model capabilities remains largely unknown. Existing approaches primarily focus on static neural activations, overlooking the temporal and functional neural topology—the dynamic connectivity patterns between neurons—which is believed to be essential for the emergence of intelligence. Furthermore, few works explicitly connect these topological findings to language generation performance, a primary indicator of LLM intelligence.

Methodology: Graph Probing

The authors introduce graph probing, a novel framework for investigating the functional connectivity of LLM neurons and its relationship to language performance.

1. Neural Topology Construction:

   • Inspired by neuroscience (constructing functional brain networks from fMRI/EEG), the method treats LLM neurons as nodes in a graph.
   • As an LLM processes a token sequence $X = [x_0, \dots, x_t]$, it produces a time series of hidden states $H \in \mathbb{R}^{n \times t}$ for $n$ neurons.
   • A weighted connectivity matrix $A \in \mathbb{R}^{n \times n}$ is constructed where each edge weight $a_{ij}$ represents the Pearson correlation coefficient between the activation time series of neuron $i$ and neuron $j$. This captures temporal co-activation patterns.

2. Probing Mechanism:

   • Unlike traditional probing which takes flattened neural activations as input, graph probing uses the flattened connectivity matrix $A$ as input.
   • Simple linear or Multi-Layer Perceptron (MLP) probes are trained to predict language performance metrics (e.g., perplexity, semantic regression) directly from this topology.
   • The core hypothesis is that the structure of computation (connectivity) contains richer information about model intelligence than the magnitude of computation (activations).

3. Scalability:

   • To address the quadratic computational cost of dense graphs ($O(n^2)$), the authors employ sparsification (pruning weak edges) and Graph Neural Network (GNN) probes that reduce parameters via weight sharing.

Key Contributions and Results

1. Universal Predictability of Performance

The study demonstrates that language generation and understanding performance can be reliably predicted using only neural topology.

• Models Tested: Experiments spanned three families (GPT-2, Pythia, Qwen2.5) ranging from 160M to 14B parameters.
• Performance Gain: Graph probing significantly outperformed activation-based probing.
  • Perplexity Prediction: Topology-based probes achieved an $R^2$ near or above 0.90, whereas activation-based probes struggled below 0.45. This represents a performance improvement of up to 130.4%.
  • Semantic Regression: For predicting space (longitude/latitude) and time (release years), topology-based probes outperformed activation probes by up to 67.7%.
• Robustness: The predictive power persisted even when retaining only 1% of the neuron connections, suggesting that a small subset of strong connections carries the critical topological signal.

2. Causal Evidence of Topological Reliance

The authors provide causal evidence that LLMs actively utilize their internal topological structure:

• Default Networks: Analysis revealed a stable "default network" where a core set of ~40 "hub neurons" (highest degree) consistently function as hubs across diverse inputs and datasets.
• Interventional Experiments: Disabling the top 1% of hub neurons (based on topological degree) caused catastrophic performance drops on the MMLU benchmark (e.g., 69.8% drop for the 14B model), far exceeding the impact of disabling neurons based on activation magnitude or random selection. This confirms that hub neurons are functionally dominant.

3. Proof-of-Concept Applications

The paper demonstrates two practical applications of graph probing:

• Model Pruning: A pruning strategy based on removing low-degree neurons (based on topology) proved more robust and effective than the state-of-the-art WANDA method (based on activation). Models retained above-random accuracy even after pruning 50% of low-degree neurons.
• Hallucination Detection: Graph probing was adapted for binary classification to detect hallucinations on the TruthfulQA dataset. Topology-based probes outperformed all baselines (including HaMI and CCS) by up to 9.73%, suggesting distinct topological signatures exist for truthful vs. hallucinated generation.

4. Extended Analyses

• Layer Specificity: Middle layers of LLMs were found to be the most informative for predicting performance, aligning with prior findings that these layers handle semantic processing.
• Functional Specialization: Distinct topological patterns were observed for different subjects (e.g., STEM vs. Social Sciences), with specific neurons acting as "STEM neurons" that disproportionately affect STEM-related outputs.
• Cross-Model Matching: Graph matching via contrastive learning successfully detected genetic relationships between models (e.g., same family vs. different families) and served as a robust fingerprinting tool, remaining stable even after significant weight drift during fine-tuning.

Significance and Claims

The paper claims that neural topology contains orders of magnitude richer information regarding LLM performance than neural activation alone. By shifting the focus from "what neurons are doing" (activation) to "how neurons are connected" (topology), the authors offer a new lens for understanding emergent capabilities.

The authors position graph probing as a tool to:

1. Demystify Internal Mechanisms: Reveal that LLMs develop intricate, consistent topological structures fundamental to their linguistic abilities.
2. Guide Development: Provide a pathway for improving LLM efficiency and reliability through topology-aware pruning and hallucination detection.
3. Bridge Disciplines: Strengthen the parallel between LLMs and human brain networks, suggesting shared organizational principles (e.g., hub neurons, default networks).

The authors remain modest, noting that while they identified stable default networks and hub neurons, more nuanced structures (like motifs or small-worldness) and their causal roles remain open questions. They also acknowledge computational limits, having evaluated models up to 14B parameters, leaving larger models for future work. The code and data are released to facilitate further research in this direction.