💬 NLP

Integrated and Cross-Architecture Interpretation of LLM Reasoning

This paper introduces the Integrated, cross-Architecture Reasoning (IAR) framework, which combines bandwidth-calibrated Mutual Information Peak analysis with Deep-Thinking Ratio overlap and Jaccard stability metrics to provide a unified, generalizable method for interpreting reasoning patterns across different large language model architectures and domains.

Original authors: Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu Yang

Published 2026-05-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonardo Matthew Yauw, Wei-Bin Kou, Yujiu Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician perform a complex trick. You can see the final result (the rabbit in the hat) and hear the magician's spoken explanation ("Now, watch closely as I pull the rabbit out..."). But you cannot see the secret moves happening inside the hat or the magician's internal thought process.

For a long time, researchers have been trying to figure out how Large Language Models (LLMs) "think" to solve hard problems like math equations or coding puzzles. They can see the words the AI writes down (the "reasoning chain"), but the actual mental work happening inside the computer's brain remains a black box.

This paper, titled "Integrated and Cross-Architecture Interpretation of LLM Reasoning," proposes a new way to peek inside that black box. The authors, Leonardo Matthew Yauw, Wei-Bin Kou, and Yujiu Yang, argue that looking at just one clue isn't enough. Instead, they built a three-part detective kit called IAR (Integrated, Architecture-agnostic Reasoning) to understand how different AI models reason.

Here is how their method works, explained with simple analogies:

The Three Clues of the IAR Detective Kit

The researchers realized that previous methods were like trying to solve a mystery with only half the evidence. They combined three different "probes" to get the full picture:

1. The "Aha!" Moment Detector (MIP)

The Analogy: Imagine you are listening to a lecture. Most of the time, the speaker is just filling space. But occasionally, they say something that perfectly connects to the answer key. That specific word is an "Aha!" moment.
The Science: The team uses a tool called Mutual Information Peak (MIP). It scans the AI's output to find specific words (tokens) that carry the most information about the correct answer.
The Catch: To make this work on different types of AI models, they had to "calibrate" the tool, like adjusting the focus on a camera lens, so it doesn't get blurry on some models but too sharp on others.

2. The "Deep Thought" Tracker (DTR)

The Analogy: Imagine a student solving a math problem. Some steps are easy and happen instantly (shallow thinking). Other steps require the student to pause, scratch their head, and think deeply before writing anything down (deep thinking).
The Science: They use a tool called Deep-Thinking Ratio (DTR). This measures how much "computational work" happens inside the model's layers before a word is finally chosen. If a word is only decided after the model has processed it through many deep layers, it's a "deep thought" token.

3. The "Consistency" Check (Jaccard Stability)

The Analogy: If you ask a student the same hard question three times, will they use the exact same key steps to solve it? If they do, they likely truly understand the concept. If they guess differently every time, they might just be lucky.
The Science: The researchers ask the AI the same problem three times with slight variations. They check if the "Aha!" words (from step 1) appear in the same spots every time. If they do, the reasoning is stable and genuine. If they change wildly, the reasoning might be a fluke.

The Big Discovery: Putting the Clues Together

The most exciting finding of the paper is what happens when you combine these clues.

The "Filter" Effect:
The researchers found that the "Aha!" words (MIP) are almost always a small, special subset of the "Deep Thought" words (DTR).

Think of it this way: Imagine a factory producing 1,000 widgets (deep thoughts). Most are just standard parts. But only 50 of them are the critical gears that actually make the machine work.
The MIP tool acts like a filter that finds those 50 critical gears inside the pile of 1,000 deep thoughts.
Why this matters: Before, people thought "Deep Thought" meant "Good Reasoning." This paper shows that just because a model is thinking hard doesn't mean it's thinking correctly. You need the "Aha!" filter to find the right hard thoughts.

Testing Across Different Models

The team didn't just test this on one AI. They tested it on three different models (Qwen-7B, Qwen-14B, and Llama-8B) across four types of tasks: Math, Coding, Logic, and Common Sense.

The Results:

Universal Pattern: The "Filter" effect (where the critical "Aha!" words are a small part of the deep thoughts) worked the same way across all three models, even though they were built differently. This suggests a universal way that AI models reason.
Lucky vs. Genuine: They could tell the difference between a model that truly solved a problem and one that got lucky.
- Genuine Reasoning: The model found a few, very specific "Aha!" words and stuck to them consistently.
- Lucky Guessing: The model produced a lot of "Aha!" words, but they were scattered, inconsistent, and didn't match up across different attempts.
The "Silent" Failure: Sometimes the model gets the answer wrong but still produces a reasoning chain. The team found that in these cases, the "Aha!" words were either missing or unstable.

What This Means (and Doesn't Mean)

What the paper claims:

We can now see where and how AI models reason by looking at specific words and how deep the model thinks about them.
This method works across different types of AI models, not just one specific brand.
We can distinguish between a model that truly understands a problem and one that is just guessing or hallucinating.

What the paper does NOT claim:

It does not say this will make AI smarter in the future.
It does not claim this can be used to diagnose human mental health or clinical issues.
It does not promise that we can now "fix" the AI's reasoning errors immediately; it just gives us a better way to see them.

Summary

Think of this paper as inventing a new pair of glasses. Before, when we looked at an AI solving a math problem, we saw a blur of words. With these new glasses (the IAR framework), we can clearly see the specific moments where the AI is doing its real work, distinguish between a brilliant insight and a lucky guess, and confirm that this "thinking style" is a fundamental part of how these machines work, regardless of which brand they are.

Technical Summary: Integrated and Cross-Architecture Interpretation of LLM Reasoning

1. Problem Statement

The interpretability of Large Language Model (LLM) reasoning is hindered by a fundamental asymmetry: while generated reasoning chains (outputs) are observable, the internal computational trajectories producing them remain opaque. Existing interpretability methods suffer from two primary limitations:

Isolation of Probes: Current methods rely on single probes that provide partial views. Mutual Information Peaks (MIP) identify tokens with high information content regarding the gold answer at the output layer but ignore the depth of computation required to produce them. Conversely, Deep-Thinking Ratio (DTR) measures computational depth via cross-layer divergence but fires on too many tokens to serve as a discriminative signal for reasoning relevance on its own.
Lack of Cross-Architecture Validation: Prior analyses are often confined to a single model family, making it difficult to distinguish fundamental reasoning properties from architectural artifacts.

2. Methodology: The IAR Framework

The authors propose the Integrated, Architecture-agnostic Reasoning (IAR) framework, a unified pipeline that analyzes reasoning along three axes: reasoning-relevance, computational depth, and noise-robustness.

2.1 Reasoning-Crucial Token Isolation (Output Layer)

The framework first isolates tokens critical for reasoning using a Bandwidth-Calibrated MIP:

HSIC Estimation: Mutual Information (MI) between a token's output representation and the gold answer embedding is estimated using the Hilbert–Schmidt Independence Criterion (HSIC) with a Gaussian RBF kernel.
Bandwidth Calibration: To prevent kernel collapse (saturation of similarities), the kernel bandwidth ( $\sigma$ ) is calibrated per architecture. For Qwen models, a fixed $\sigma=50$ is used; for Llama-8B, a per-problem median heuristic based on hidden state distances is applied.
Peak Detection: A non-parametric Tukey IQR rule identifies "reasoning-crucial" tokens ( $P_q$ ) where MI exceeds $Q3 + 1.5 \times IQR$ .

2.2 Cross-Layer Trajectory Analysis (Computational Depth)

The framework then traces the computational depth of these tokens using DTR:

Jensen–Shannon Divergence (JSD): The divergence between the predictive distribution of each hidden layer and the final layer is computed.
Settling Layer: A token is defined as DTR-deep if its predictive distribution only stabilizes (JSD < threshold) in the upper layers of the network.
Overlap Analysis: The framework performs a set-level overlap analysis between the MIP set ( $P_q$ $P_{q}$ ) and the DTR-deep set ( $D_q$ $D_{q}$ ). It tests two hypotheses:
- Independence ( $H_1$ ): MIPs are scattered across deep and shallow tokens.
- Containment ( $H_2$ ): $P_q \subseteq D_q$ , implying MIPs are a sparse, reasoning-relevant subset of computationally deep tokens.

2.3 Reasoning-Quality Validation (Noise-Robustness)

To verify that identified tokens reflect genuine reasoning rather than stochastic artifacts, the framework employs a Three-Way Jaccard Stability metric:

Stochastic Generation: Three independently seeded reasoning chains are generated for each problem.
Stability Metric ( $J_3$ ): The intersection of MIP sets across the three generations is divided by their union.
Taxonomy: Problems are categorized into Genuine (correct and stable MIP structure), Lucky (correct but unstable MIP structure), and Silent (incorrect) based on correctness and $J_3$ .
Discrimination: Aggregate MIP statistics (peak count, ratio, intensity) are tested to see if they discriminate Genuine problems from Lucky/Silent ones using Mann–Whitney U tests.

3. Experimental Setup

Models: Three reasoning models from the DeepSeek-R1-Distill family: Qwen-7B, Qwen-14B, and Llama-8B.
Domains: Mathematics (GSM8K), Code (MBPP), Logic (BBH Boolean Expressions), and Commonsense (CommonsenseQA).
Decoding: Deterministic (greedy) for detection/overlap analysis; Stochastic (sampling, $T=0.7$ ) for stability and quality discrimination.

4. Key Results

The experiments yield four primary findings across the three architectures and four domains:

Detectability (RQ1): Bandwidth-calibrated MIP detection is viable across architectures but shows conditional sensitivity. Llama-8B exhibits uniform detection across all domains. Qwen models show a split: high detection on symbolic tasks (math, code) but near-zero detection on natural-language tasks (logic, commonsense) for Qwen-7B, attributed to embedding dimensionality and kernel bandwidth interactions.
Containment (RQ2): The Containment Hypothesis ( $H_2$ ) is strongly supported. MIP-identified tokens are a high-precision subset (Token-Pool Precision $\approx 0.82$ ) of DTR-deep tokens. This confirms that reasoning-crucial tokens are computationally intensive, and MIP acts as an interpretable filter on the broader set of deep tokens.
Stability (RQ3): While MIP existence is sample-stable (the binary event of detecting peaks is reproducible), the specific identity of MIP tokens is sample-fragile (low Jaccard stability, $J_3 \approx 0.09–0.28$ ). This suggests reasoning quality should be assessed via aggregate statistics rather than specific token identities.
Quality Discrimination (RQ4): Aggregate MIP statistics discriminate Genuine reasoning from Lucky guesses. Counter-intuitively, Genuine problems exhibit fewer, sparser, and weaker MIPs than Lucky ones. This supports a "reasoning economy" view where true solutions rely on a few sharp anchors, while lucky trajectories produce diffuse, high-volume MI activity. This discrimination is portable across architectures, though effect sizes vary (strongest in Qwen-14B).

5. Significance and Contributions

The paper claims the following contributions:

Unified Framework: It introduces IAR, a paradigm that jointly reveals reasoning via relevance, depth, and robustness, moving beyond isolated probes.
Compositional Insight: It establishes that MIP and DTR are not competing metrics but compositional; MIP filters the computationally deep tokens identified by DTR to find the reasoning-relevant subset.
Cross-Architecture Generalization: It provides the first evidence of a transferable reasoning pattern (MIP containment within DTR-deep tokens) across different model families (Qwen and Llama) and scales.
Methodological Rigor: It demonstrates the necessity of multi-seed stochastic evaluation and bandwidth calibration to avoid architectural artifacts, refining the understanding of "thinking tokens" in LLMs.

The authors conclude that while individual token identities are unstable, the aggregate statistical properties of reasoning traces offer a robust, architecture-invariant signal for distinguishing genuine reasoning from stochastic success.