💬 NLP

Disentangling generalization and memorization in large language models using chess

This paper utilizes chess as a controlled testbed to demonstrate that large language models rely heavily on memorizing common patterns, as their performance significantly degrades and approaches random chance when faced with novel positions lacking relevant training priors, revealing inherent limitations in their systematic generalization capabilities.

Original authors: Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsaecker

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leonard S. Pleiss, Maximilian Schiffer, Robert K. von Weizsaecker

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out if a student is actually learning a subject or just memorizing the answers to a specific test they've seen before.

This paper uses the game of Chess as a giant, controlled classroom to test Large Language Models (LLMs)—the AI brains behind tools like ChatGPT, Claude, and Gemini. The researchers wanted to know: When these AIs solve a problem, are they using genuine reasoning, or are they just pulling a memorized answer from their massive database of training data?

Here is the breakdown of their experiment and findings, using simple analogies.

1. The Setup: Three Types of Chess Puzzles

To test the difference between memorization and reasoning, the researchers created three types of chess positions, like three different levels of a video game:

The "Famous" Puzzles (Within-Distribution): These are standard, well-known chess openings (like the "Ruy López"). They appear in millions of games and are likely in the AI's training data.
- Analogy: This is like asking a student a question that is literally written in their textbook. If they get it right, they might just be reciting the book.
The "Familiar but New" Puzzles (Near-Distribution): These look like normal chess games but have never been played before. They follow the rules and look like a standard game, but the specific arrangement is unique.
- Analogy: This is like a math problem that uses the same numbers and formulas as the textbook, but the story about "apples and oranges" is slightly different. The student has to connect the dots, not just copy the answer.
The "Alien" Puzzles (Out-of-Distribution): These are weird, random board setups. The pieces are placed in ways that almost never happen in real life (e.g., pawns blocking the kings). They break all the usual patterns.
- Analogy: This is like asking the student to solve a logic puzzle using a language they've never heard before, with rules they've never seen. There is no textbook to memorize; they must think from scratch.

2. The Experiment: How the AI Performed

The researchers asked various top-tier AIs (GPT-3.5, GPT-4o, GPT-5, Claude, and Gemini) to make the best move in these three types of puzzles. They measured success by how close the AI's move was to the move a super-computer chess engine would make.

Here is what they found:

The "Famous" Puzzles: AI is a Super-Reciter

When the puzzles were standard and familiar, the AIs played very well.

The Takeaway: The models are incredibly good at memorization. If they've seen a pattern before, they can recall the solution perfectly.

The "Alien" Puzzles: The AI Crashes

When the puzzles were weird and new (Out-of-Distribution), the AIs fell apart.

The Takeaway: Their performance dropped to the level of random guessing. In fact, they started making illegal moves (like moving a knight like a bishop) at a high rate.
The Metaphor: It's like a student who knows the answers to every question on the final exam but, when handed a blank sheet of paper and asked to "write a story," starts speaking gibberish. They don't actually understand the rules of the game; they only recognize the patterns they've seen.

The "Familiar but New" Puzzles: A Steep Drop-Off

As the puzzles got slightly less familiar, the AI's performance didn't just dip; it slid down a steep cliff.

The Takeaway: The more the puzzle looked like something the AI hadn't seen before, the worse it got.

3. The "Thinking Harder" Test

The researchers also tested the newer models (like GPT-5) that have a "reasoning mode," where the AI is told to "think step-by-step" before answering.

The Result: Thinking harder helped, but only so much.
The Catch: When the puzzle was weird (Out-of-Distribution), the AI would spend a lot of time "thinking" (using many more words/tokens), but it still couldn't solve the problem.
The Metaphor: Imagine a student who, when faced with a hard math problem, starts writing a 10-page essay about the history of math instead of solving the equation. They are working hard, but they are just rearranging things they already know, not inventing a new solution. The "thinking" process amplified their memorization but didn't create true reasoning.

4. The Big Conclusion

The paper concludes that current Large Language Models are brilliant at pattern matching but struggle with genuine generalization.

Scaling isn't the magic bullet: Making the models bigger (adding more data and parameters) helps them memorize more, but it doesn't seem to help them learn how to reason about things they've never seen.
The "Crystallized" Trap: The models rely on "crystallized" knowledge (memorized chunks). When those chunks aren't available, they don't have a "fluid" intelligence to fall back on. They can't derive the rules of the game from first principles; they can only recognize the game if it looks exactly like something they've seen before.

In short: These AIs are like encyclopedias that can recite the entire history of chess, but if you put them in a room with a chessboard and a set of rules they've never seen, they might not even know how to move the pieces correctly. They are masters of the past, but not yet masters of the new.

Technical Summary: Disentangling Generalization and Memorization in Large Language Models Using Chess

Problem Statement

Despite the remarkable capabilities of Large Language Models (LLMs), a fundamental debate persists regarding the origin of their performance: do improvements stem from enhanced retrieval of vast training corpora (memorization) or from genuine, systematic reasoning abilities (generalization)? Current benchmarks often fail to disentangle these faculties because the availability of "relevant priors" (data similar to the training distribution) in modern, opaque training sets is unknown. Consequently, existing studies frequently conflate recall with reasoning, leaving the extent to which LLMs can generalize beyond their training data manifold unclear.

Methodology

To address this, the authors propose a controlled empirical framework using chess as a testbed. Chess offers a unique combination of clear rules, deep combinatorial complexity (approx. $10^{43}$ states), and computationally verifiable performance metrics via superhuman engines. The study constructs a taxonomy of chess positions based on the estimated density of relevant priors in an LLM's training data:

Within-Distribution (WD): Positions occurring frequently in human play (e.g., standard opening lines), likely present in training data. These test memorization.
Near-Distribution (ND): Positions generated by random legal moves from the start that are not in the database but retain structural homology to familiar states (e.g., standard pawn structures). These test associative recall and interpolation.
Out-of-Distribution (OOD): Positions generated by random piece placement, creating atypical board states with minimal structural similarity to human games. These test generalization and first-principles reasoning.

Evaluation Protocol:

Models: The study evaluates the GPT lineage (GPT-3.5, GPT-4o, GPT-5) and contemporary models (Claude Opus, Gemini). GPT-5 is tested with both minimal and moderate reasoning (Chain-of-Thought) configurations.
Metrics: Performance is measured using Centipawn Loss (CPL), a standard metric for decision regret relative to a superhuman engine (Stockfish 17.1). Illegal moves are penalized heavily (CPL = 1,000). A random legal-move baseline and Leela Chess Zero (Lc0) serve as lower and upper bounds, respectively.
Normalization: To isolate move quality from rule adherence, results are also analyzed after excluding illegal moves and normalizing against a random legal-move policy.

Key Contributions

A Principled Taxonomy for Priors: The paper introduces a method to approximate the density of relevant priors in opaque training corpora by leveraging the combinatorial structure of chess, allowing for a systematic distinction between memorization and generalization without explicit knowledge of training data.
Longitudinal and Cross-Lineage Analysis: It provides a rigorous evaluation of model evolution across generations (GPT-3.5 to GPT-5) and across different architectures (Anthropic, Google, OpenAI), isolating the effects of scale and reasoning augmentation.
Quantification of the Generalization Gap: The study quantifies the "generalization gap" ( $R_{gen}$ ), demonstrating that performance degrades systematically as the density of relevant priors decreases.

Results

The analysis reveals a consistent and steep performance gradient across all tested models:

Performance Degradation: Model performance (measured by CPL) consistently declines as positions move from WD to ND to OOD.
- In WD positions, models perform reasonably well, often outperforming random play.
- In ND positions, performance drops significantly.
- In OOD positions, base model performance regresses to the random-play baseline. Even GPT-5, the state-of-the-art model, fails to outperform a random legal-move policy in OOD scenarios when illegal moves are excluded.
Illegal Moves: The frequency of illegal moves increases drastically in OOD conditions (e.g., >50% for Claude Opus and >70% for Gemini in OOD), indicating a failure to maintain rule consistency when memorized patterns are unavailable.
Diminishing Returns on Scale: While newer model generations show steady improvements, the rate of improvement diminishes sharply for tasks with sparse priors. The gap between GPT-3.5 and GPT-5 is substantial in WD but negligible in OOD.
Reasoning Limitations: While reasoning-augmented inference (Chain-of-Thought) improves absolute performance and reduces illegal moves, its relative marginal benefit per token decreases as priors become sparse. In OOD conditions, reasoning widens the generalization gap relative to WD, suggesting it acts as an amplifier of memorization rather than an engine for novel generalization.

Significance and Claims

The paper claims that these findings highlight severe limitations in the systematic generalization of current state-of-the-art LLMs.

Memorization vs. Reasoning: The results suggest that LLMs possess a vast library of surface-level heuristics and patterns but lack the ability to construct valid strategies from first principles when those patterns are absent.
Scaling Laws: The study challenges the notion that scaling alone will bridge the generalization gap. It posits that increasing model capacity primarily enhances recall within data-rich regions of the training distribution, leaving structurally novel regions unresolved.
Nature of Reasoning: The authors argue that current Chain-of-Thought mechanisms function primarily as retrieval optimizers (helping locate correct memories) rather than as independent engines for conceptual synthesis or fluid reasoning.
Broader Implications: While the study is domain-specific, the authors suggest these motifs (robust memorization, brittle generalization, diminishing returns) may reflect a general phenomenon in LLMs, indicating a systemic difficulty in inferring structural rules to solve novel tasks without close priors. This has profound implications for the safety and reliability of deploying LLMs in rapidly evolving or novel domains where historical data is insufficient.

The paper concludes that achieving robust performance in the absence of relevant priors requires mechanisms beyond simple scaling, pointing to a need for architectural changes that support genuine fluid intelligence.