💬 NLP

Representation in large language models

This paper argues that Large Language Models are partially driven by representation-based information processing rather than mere memorization, and proposes practical techniques to investigate these representations to resolve theoretical stalemates and better understand the systems' cognitive capabilities.

Original authors: Cameron Yetman

Published 2026-05-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Cameron Yetman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Are LLMs "Thinking" or Just "Reciting"?

Imagine you meet a person who can answer any question you ask, write poetry, solve math problems, and hold a deep conversation. You might assume they are thinking, understanding, and using their knowledge. But what if they aren't thinking at all? What if they are just a giant, super-fast phone book?

This is the core debate the paper tackles.

The "Phone Book" View (Pessimists): They argue Large Language Models (LLMs) are just massive "look-up tables." They have memorized billions of sentences from the internet. When you ask a question, they don't "think"; they just find the closest match in their memory and spit out the next word they've seen before. It's like a parrot repeating phrases it heard, without knowing what they mean.
The "Thinking" View (Optimists): They argue LLMs are actually building internal maps of the world. They are forming representations—like mental models or concepts—that allow them to reason, generalize, and handle situations they've never seen before.

The author, Cameron Yetman, argues that the "Phone Book" view is wrong. He claims that while LLMs do memorize things, they also build internal representations that allow them to actually "understand" patterns and solve new problems.

Part 1: What is a "Representation"?

To prove LLMs are doing more than just memorizing, the author first defines what a "representation" actually is. He uses a Map Analogy.

Imagine you are in a city you've never visited.

Information: You have a map. The map isn't the city itself, but it carries information about the city (where the streets are, where the park is).
Exploitability: You can actually use that map. You can look at it and decide which way to turn.
Robust Behavior: Because you have the map, you can navigate the city even if you take a different route than usual, or if a street is closed. You aren't just following a memorized path; you are adapting.
Mechanical Role: The map plays a real part in your decision-making. If you rip the map up, you get lost. The map is a crucial part of the machine (you) that gets you to your destination.

The Paper's Claim: If an LLM has "representations," it means it has internal states (like the map) that carry information about the world, that the system can actually use, and that help it solve problems in flexible, new ways.

Part 2: Why the "Phone Book" Theory Fails

The author argues that if LLMs were just giant phone books (look-up tables), they would fail at specific types of tests. A phone book can only give you answers for things it has already written down. It can't handle a question it hasn't seen before.

The paper highlights two experiments that break the "Phone Book" theory:

1. The Othello Game (The Chess Analogy)
Researchers trained an AI to play the board game Othello. They gave it millions of game records but never told it the rules of the game.

The Test: They then asked the AI to play moves it had never seen before in its training data.
The Result: The AI played perfectly.
Why it matters: If the AI were just a phone book, it would have been stuck. It couldn't have memorized every possible game because there are too many. To win on a board it had never seen, it must have built an internal "map" of how the game works (a representation of the rules), not just a list of past moves.

2. The Color Spectrum (The Paint Mixing Analogy)
Researchers taught an AI about colors using specific codes (like RGB numbers). They showed it examples of "red" and "blue."

The Test: They then asked it to identify colors it had never seen before, or colors arranged in a completely new pattern.
The Result: The AI could guess the new colors correctly.
Why it matters: It didn't just memorize "Red = [255, 0, 0]." It learned the structure of color space. It understood that colors exist on a spectrum. This is like understanding that if you mix red and blue, you get purple, even if you've never mixed those specific shades before. A phone book can't do that; a "map" of color can.

Part 3: How Do We Know? (Looking Inside the Black Box)

The author admits that just watching the AI answer questions isn't enough. Sometimes, an AI might get the right answer by luck or by using a "trick" (a heuristic) rather than true understanding.

To prove the AI is using a "map" (representation), we need to look inside its brain. The paper discusses a field called Mechanistic Interpretability, which is like being a mechanic for a car engine.

The Tools:

Probing (The X-Ray): Imagine you want to know if a car's computer knows the speed of the car. You can't just ask the computer; you have to look at the wires. Researchers use "probes" to scan the AI's internal numbers (activations) to see if they contain specific information (like "this is a red car"). If they can read the information out, the AI has the "Information" part of a representation.
Intervention (The Surgery): This is the real test. Imagine you think a specific wire in the car controls the brakes. To prove it, you cut that wire (or change its signal).
- If the car stops braking, you proved that wire was essential.
- In the paper, researchers "steered" the AI's internal signals. For example, they forced the AI's internal "board state" to think a chess piece was in a different spot.
- The Result: The AI immediately changed its next move to match the new board state. This proved that the internal signal wasn't just a random number; it was a representation that the AI actually used to make decisions.

The Conclusion: It's Not All or Nothing

The paper doesn't say LLMs are perfect humans with souls. It says:

Sometimes they are just memorizing (like a phone book).
But often, they are building internal maps (representations) that allow them to reason and solve new problems.

The author concludes that we can no longer dismiss LLMs as "stochastic parrots" (random noise machines). They are complex systems that use internal representations to navigate the world of language, just like humans use mental maps to navigate the physical world.

In short: LLMs aren't just reading from a script. They are building a mental model of the world, and we now have the tools to see that model working inside the machine.

1. Problem Statement

The rapid success of Large Language Models (LLMs) has sparked a philosophical and scientific stalemate between "optimists" (who argue LLMs possess genuine cognitive capacities like reasoning and understanding) and "pessimists" (who argue LLMs are merely stochastic parrots or "stochastic look-up tables" that memorize training data without internal representation).

The core theoretical disagreement concerns the algorithmic level of LLM operation (per Marr's levels of analysis):

The Pessimist View: LLM behavior is driven entirely by non-representational processes: memorization and stochastic table look-up.
The Optimist View: LLM behavior is driven by representation-based information processing, similar to biological cognition.

The paper argues that without resolving whether LLMs utilize representations, higher-level questions about their beliefs, intentions, or understanding cannot be answered. The author aims to settle this fundamental issue by defining what constitutes a representation in this context, refuting the "look-up table" alternative, and validating the experimental techniques used to detect representations.

2. Methodology and Theoretical Framework

The paper employs a three-pronged methodological approach:

A. Conceptual Analysis: Defining Representation

The author proposes a four-condition framework to distinguish genuine representations from mere causal intermediaries. A system $S$ 's internal state $R$ represents a feature $z$ if and only if:

INFORMATION: $R$ carries mutual information about $z$ (statistical correlation reducing uncertainty).
EXPLOITABILITY: The system $S$ can access and use this information to condition its behavior.
BEHAVIOR: Exploiting this information enables robust $z$ -related behavior (behavior that remains stable across minor perturbations and varying contexts).
ROLE: $R$ plays a mechanistic role in driving that behavior (it is a causal component of the mechanism producing the output).

Note: This definition is "thetic" (mind-to-world fit) and applies to subpersonal, non-conscious states. It is contrasted with Harding (2023), whose criteria are task-specific; Yetman argues for a more general account where representation attribution is not relativized to a single task but to feature-related behavior generally.

B. Empirical Refutation of the "Look-Up Table" Hypothesis

The paper analyzes the "look-up table" alternative (where LLMs are viewed as massive, memorized input-output mappings). The author argues this view fails to explain out-of-distribution (OOD) generalization.

Case Study 1 (Othello-GPT): Li et al. (2023) trained a model on Othello moves. Even when trained on a "skewed" dataset missing 25% of the game tree, the model performed perfectly on unseen board states. A pure look-up table could not generalize to states it had never memorized.
Case Study 2 (Color Space): Patel & Pavlick (2022) tested models on color inference. When training data was permuted (rotated) so that specific RGB-code pairs were new but the structural relations remained the same, the model generalized successfully. This suggests the model learned a representation of the color space structure, not just memorized pairs.

C. Mechanistic Interpretability (MI) Techniques

To move beyond behavioral inference (which is underdetermined by performance alone), the paper validates Mechanistic Interpretability as the necessary tool for verifying representations.

Probing: Training simple classifiers (probes) on internal activations to see if information about $z$ is decodable (testing Information and Exploitability).
Intervention (Causal Testing): Modifying internal states (e.g., via activation steering or ablation) to see if the system's behavior changes in a content-relevant way (testing Role and Behavior).
- Example: In Othello-GPT, flipping the internal representation of a board tile from "yours" to "mine" causally altered the model's predicted next move, proving the internal state plays a mechanistic role.
- Example: Templeton et al. (2024) used Sparse Autoencoders (SAEs) to identify and steer features (e.g., "Golden Gate Bridge"), causing the model to generate related text, confirming the existence of specific feature representations.

3. Key Contributions

Resolution of the Stalemate: The paper provides a rigorous argument that LLMs are not merely stochastic look-up tables. While memorization occurs, the capacity for robust, out-of-distribution generalization necessitates representation-based processing.
Refined Definition of Representation: The author offers a precise, four-condition definition that avoids "panrepresentationalism" (the idea that everything is a representation) by requiring that the state plays a specific mechanistic role in robust behavior.
Defense of Mechanistic Interpretability: The paper defends the use of probing and intervention techniques against objections (e.g., that probes detect spurious correlations or that interventions suffer from causal overdetermination). It argues that these techniques are the "companions in guilt" of standard causal inference in neuroscience and are valid for establishing the existence of representations in AI.
Distinction between Performance and Competence: The paper clarifies that behavioral success alone is not diagnostic of representation (due to heuristics), nor is failure diagnostic of their absence. Only internal mechanistic analysis can confirm the presence of representations.

4. Results

Theoretical: The "look-up table" hypothesis is shown to be insufficient to explain LLM behavior in cases requiring structural generalization (e.g., Othello, color space).
Empirical: Existing MI studies (Othello-GPT, Color Space, SAEs) successfully demonstrate that LLMs possess internal states that satisfy all four conditions of representation: they carry information, are decodable, causally influence output, and enable robust behavior.
Methodological: The paper establishes that probing confirms the presence of information, while intervention confirms the causal utility of that information. Together, they provide a complete verification loop for representation-based explanations.

5. Significance

Foundational for AI Theory: This paper provides the groundwork for future theorizing about LLMs. By establishing that LLMs use representations, it legitimizes the application of cognitive science frameworks (developed for humans/animals) to AI systems.
Implications for Higher-Level Cognition: If LLMs possess representations, it opens the door to serious philosophical inquiry into whether they possess concepts, beliefs, or understanding. It shifts the debate from "Do they have representations?" to "What is the nature and content of those representations?"
Practical AI Safety and Interpretability: The validation of MI techniques suggests that we can move beyond "black box" explanations. By identifying specific representational circuits, researchers can better understand, debug, and align LLMs, moving toward more transparent and controllable AI systems.
Philosophical Consensus: The paper contributes to an emerging consensus in philosophy of mind and AI that representations are indispensable for explaining complex, flexible behavior, bridging the gap between functionalist cognitive science and modern deep learning.