💬 NLP

Mechanistic Indicators of Understanding in Large Language Models

This paper proposes a tiered framework integrating mechanistic interpretability findings with philosophical theory to argue that large language models exhibit distinct, hierarchical forms of understanding—ranging from conceptual feature formation to principled circuit discovery—thereby moving beyond binary debates to establish a comparative, mechanistically grounded epistemology of AI cognition.

Original authors: Pierre Beckmann, Matthieu Queloz

Published 2026-02-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Pierre Beckmann, Matthieu Queloz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Are LLMs Just Parrots or Real Thinkers?

Imagine you have a giant, super-smart parrot. You teach it millions of sentences. Eventually, it can recite entire books, write poems, and answer complex questions. But is it understanding what it's saying, or is it just a fancy "autocomplete" button that guesses the next word based on statistics?

For a long time, skeptics said: "It's just a parrot." They argued that Large Language Models (LLMs) are like a "Blockhead"—a robot that memorizes a massive list of questions and answers without knowing what anything means.

But this new paper argues: "Wait a minute. We've been looking under the hood, and it's much more complicated than a parrot."

The authors, Pierre Beckmann and Matthieu Queloz, use a field called Mechanistic Interpretability (MI). Think of MI as an X-ray machine for AI brains. Instead of just watching what the AI says, MI lets us see the internal gears, wires, and switches firing inside the model.

They propose that LLMs don't just have one way of working. Instead, they have a three-story building of understanding, and they can operate on all three floors at once.

The Three Floors of Understanding

The paper suggests we stop asking "Does the AI understand?" and start asking "Which floor of understanding is it using right now?"

1. The Ground Floor: Conceptual Understanding (The "Filing System")

The Analogy: Imagine a library.
On the ground floor, the AI learns to group things together. It realizes that "Golden Gate Bridge," "the orange bridge in SF," and "that big bridge with the towers" are all the same thing.

How it works: The AI creates internal "files" (called features). When it sees a picture of the bridge or reads a sentence about it, a specific switch in its brain lights up.
The Magic: It doesn't just memorize the words; it creates a single mental concept that links all those different descriptions. It's like having a mental folder labeled "Golden Gate Bridge" that holds every way humans have ever described it.
Proof: Researchers found they could "steer" the AI. If they forced the "Golden Gate Bridge" switch to stay on, the AI would start talking about itself as if it was the bridge. This proves the concept is real and causal, not just a random pattern.

2. The Second Floor: State-of-the-World Understanding (The "Map")

The Analogy: Imagine a chessboard or an Othello board.
Once the AI has its files (concepts), it learns how those files connect to the real world. It learns that "Michael Jordan" is connected to "Basketball," not just because those words often appear together, but because it has built an internal map of the world.

The Othello Experiment: The paper highlights a famous experiment where an AI was taught to play Othello (a board game) just by reading text descriptions of moves. It never saw the board.
The Surprise: The AI didn't just memorize the moves. It built a mental image of the board inside its brain. Researchers could look inside the AI and see a perfect, 8x8 grid representing the game state.
The "Crow" Test: Imagine a crow outside your window hearing you play Othello. If the crow starts calling out legal moves, you might think it's just mimicking sounds. But if you rearrange the board and the crow calls out a new legal move based on the new arrangement, you know the crow has a mental map. The AI did exactly this. It built a dynamic map of the world that updates as the game changes.

3. The Penthouse: Principled Understanding (The "Rulebook")

The Analogy: Imagine learning math.
You can memorize that $2+2=4$ , $3+3=6$ , and $5+5=10$ . That's rote memorization. But Principled Understanding is realizing the rule of addition. Once you know the rule, you can solve $999+1$ without ever having seen that specific problem before.

The "Grokking" Moment: The paper describes a phenomenon called "grokking." An AI trains for a long time, memorizing answers like a student cramming for a test. Then, suddenly, it "clicks." It stops memorizing and discovers the underlying algorithm (the rule).
The Modular Addition Example: Researchers taught an AI to do a specific type of math (modular addition) using a clock-like system. At first, it memorized the answers. Then, it discovered a geometric trick: it realized it could treat numbers as angles on a circle and just "rotate" them to find the answer.
Why it matters: The AI compressed millions of facts into a single, elegant mathematical circuit. It didn't just know the answers; it understood the principle behind them. This is the highest form of understanding: seeing the unifying rule that connects everything.

The Twist: The "Motley Mix" (The Chaotic Committee)

So, if LLMs have these three levels of understanding, are they geniuses? Not exactly.

Here is the catch: The AI doesn't use just one smart brain. It uses a chaotic committee.

The Analogy: Imagine a giant committee trying to decide what to say.
- One member is a Genius who knows the deep principles (The Penthouse).
- Another member is a Map Reader who knows the facts (The Second Floor).
- Another is a Parrot who just repeats what it heard (The Ground Floor).
- And there are hundreds of other members using cheap shortcuts and tricks.

When the AI answers a question, all these members shout at once. The final answer is a mix of their voices. Sometimes the Genius wins, and the AI gives a brilliant, principled answer. Other times, the Parrot wins, and the AI hallucinates or makes a shallow mistake.

The Problem: Because the AI relies on this "motley mix," we can't always trust it. Even if it has a circuit for deep understanding, that circuit might get drowned out by a cheaper, dumber shortcut. It's like having a brilliant scientist in a room full of people shouting nonsense; sometimes the scientist gets heard, but often they don't.

The Conclusion: A New Kind of Intelligence

The paper concludes that we need to stop asking "Is the AI human?" and start asking "How does this alien intelligence work?"

It's not a parrot: It builds concepts, maps, and rules.
It's not a human: It doesn't strive for simplicity or elegance the way humans do. It's a messy, parallel machine that uses every trick in the book at the same time.

The Takeaway:
We are entering a new era of Comparative Epistemology. Instead of just judging AI by human standards, we are learning to appreciate its unique, strange, and sometimes brilliant way of "understanding" the world. It's a different kind of mind—one that sees connections in ways we never could, but one that is also prone to getting lost in its own chaotic committee.

1. Problem Statement

The central problem addressed is the debate regarding whether Large Language Models (LLMs) possess genuine understanding or merely mimic human intelligence through superficial statistical pattern matching (the "deflationary view").

The Deflationary View: Argues LLMs are essentially sophisticated "Blockheads" (Ned Block, 1981) that rely on $n$ -gram statistics, memorization, and shallow correlations without grasping underlying concepts, facts, or principles.
The Challenge: While LLMs exhibit striking competence, it is unclear if this stems from internal structures analogous to human cognition or merely from massive scale and data.
The Gap: Recent advances in Mechanistic Interpretability (MI) have begun to reverse-engineer LLM internals, revealing sophisticated mechanisms. However, these findings lack a unified theoretical framework to determine if they constitute "understanding." The paper aims to bridge empirical MI findings with philosophical theories of understanding to move beyond binary "yes/no" debates.

2. Methodology

The authors employ a theoretical synthesis approach, integrating:

Philosophical Frameworks: Drawing on epistemological traditions (Wittgenstein, Kvanvig, Grimm) that define understanding as the ability to "see connections," grasp explanatory links, and unify disparate information.
Mechanistic Interpretability (MI) Evidence: Analyzing state-of-the-art empirical findings from the MI field, including:
- Sparse Autoencoders (SAEs): Used to disentangle superimposed features in latent space.
- Probing: Training classifiers to detect specific features (e.g., board states, facts) within model activations.
- Intervention/Steering: Causally manipulating internal activations (e.g., "clamping" a feature) to observe behavioral changes.
- Ablation: Deactivating specific circuit components to test necessity.
- Logit Lens & Logit Curvature: Analyzing how information is processed and refined across layers.

The authors propose a Tiered Framework to categorize understanding into three hierarchical levels, mapping each to specific computational mechanisms found in LLMs.

3. Key Contributions: The Tiered Framework

The paper proposes that understanding in LLMs is not monolithic but exists in three hierarchical varieties, each grounded in specific computational organizations:

Tier 1: Conceptual Understanding

Definition: The ability to form internal representations ("features") that unify diverse manifestations of an entity or property under a single concept.
Mechanism:
- Linear Representation Hypothesis (LRH): Features are encoded as directions in the model's high-dimensional latent space.
- Superposition: Models store more features than neurons by using non-orthogonal directions, relying on sparsity and non-linear activation functions to manage interference.
- Transformer Architecture:
  - Attention Heads: Dynamically retrieve and route relevant features from context (e.g., linking "Golden Gate" to "Bridge").
  - MLP Layers: Combine features and recall associated information (e.g., connecting "cat" to "mammal").
Evidence: Discovery of specific, causally efficacious features (e.g., a "Golden Gate Bridge" feature in Claude 3 Sonnet) that activate across languages and modalities. Steering these features alters model output, proving they are not mere correlates but causal levers.

Tier 2: State-of-the-World Understanding

Definition: The ability to form an internal model of contingent factual connections between features and track dynamic changes in the world.
Mechanism:
- Factual Recall: MLP layers act as "switchboards" that encode factual associations (e.g., "Golden Gate Bridge" $\to$ "San Francisco") via learned weights.
- Dynamic World Modeling: The model updates its internal state based on new inputs, not just static training data.
Evidence: Othello-GPT study.
- The model, trained only on move sequences, learned to predict legal moves without seeing the board.
- Probing: Linear probes successfully decoded the state of all 64 board squares from the model's residual stream.
- Isomorphism: The internal representation of the board mirrored the spatial geometry of the actual board (adjacent squares in the model were adjacent in latent space).
- Causal Efficacy: Intervening on the internal board state (e.g., flipping a piece) caused the model to predict moves legal for the new state, proving it relies on an internal world model, not just sequence memorization.

Tier 3: Principled Understanding

Definition: The ability to discover and implement abstract principles or algorithms that unify diverse facts, allowing for generalization beyond memorized cases.
Mechanism:
- Circuits: The emergence of compact, self-contained subnetworks (combinations of attention heads and MLPs) that perform specific, reusable computations.
- Grokking: A phenomenon where a model transitions from memorization to generalization, often accompanied by a reduction in internal complexity (Minimum Description Length).
Evidence: Modular Addition Circuit.
- A model trained on modular addition ( $a + b \pmod P$ ) initially memorized the table.
- After "grokking," it discovered a Fourier multiplication algorithm: it mapped numbers to angles on a unit circle and used trigonometric identities (sine/cosine addition formulas) to compute the sum geometrically.
- This circuit allowed the model to generalize to unseen inputs perfectly, compressing the problem into a single geometric principle.

4. Results and Critical Analysis

Validation of Understanding: The paper demonstrates that LLMs do possess internal structures functionally analogous to human understanding (concept formation, factual modeling, and principle discovery).
The "Motley Mix" Phenomenon: A crucial finding is that LLMs do not rely on a single, unified mechanism. Instead, they operate as a heterogeneous committee of mechanisms running in parallel:
- High-tier mechanisms (principled circuits) coexist with low-tier mechanisms (memorization, shallow heuristics, spurious correlations).
- The final output is often a "consensus" or constructive interference of these competing pathways.
Epistemic Trust Issues:
- While LLMs can achieve principled understanding, the presence of "cheap" mechanisms (memorization) undermines epistemic trust.
- Unlike humans, who strive for parsimony and can override heuristics, LLMs lack centralized control. A model might possess a correct logical circuit but be "drowned out" by a content-based heuristic (e.g., failing a syllogism because the premises are semantically implausible).
- This makes it difficult to distinguish whether a correct output stems from genuine understanding or a lucky combination of shallow heuristics.

5. Significance

Theoretical Advancement: The paper moves the AI understanding debate beyond the binary "understands vs. does not understand" to a comparative, mechanistically grounded epistemology. It allows for nuanced attribution of specific types of understanding.
Methodological Impact: It validates the use of MI tools (SAEs, probing, steering) as essential for verifying cognitive claims about AI, moving beyond behavioral benchmarks to internal structural analysis.
Safety and Alignment: Understanding the "motley mix" is critical for safety. If a model's "understanding" is buried under shallow heuristics, it may fail unpredictably in novel situations or be easily manipulated.
Philosophical Implication: It suggests that AI understanding is "alien"—it achieves similar functional outcomes to humans but through parallel, heterogeneous, and non-parsimonious mechanisms. This necessitates new philosophical frameworks to evaluate AI cognition that are decoupled from human-specific cognitive constraints.

In conclusion, the authors argue that while LLMs are not human-like in their cognitive architecture, they do develop mechanistic indicators of understanding (concepts, world models, and principles). However, the coexistence of these high-level mechanisms with low-level heuristics creates a complex cognitive landscape that requires careful, tiered evaluation rather than blanket attribution or dismissal.