💬 NLP

Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs

This paper introduces a method to mitigate memorization in Large Language Models by distinguishing between gist and verbatim memory, revealing that existing benchmarks on popular fictional works often overestimate true character understanding due to data contamination.

Original authors: Yuxuan Jiang, Francis Ferraro

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuxuan Jiang, Francis Ferraro

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are taking a test on a famous TV show you've watched a hundred times. The question asks, "Who said this line?"

If you are a human who actually watched the show, you might answer by remembering the character's personality: "Oh, that's definitely Sheldon. He's the one who always complains about the temperature and uses big words." You are using reasoning.

But what if you are a super-smart computer that has read the entire script of that show thousands of times? It might not "think" about the personality at all. Instead, it might just say, "I remember this exact sentence from my memory bank. The answer is Sheldon." This is memorization.

This paper, titled "Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning," investigates a big problem: Are AI models actually "thinking," or are they just cheating by memorizing the answers?

Here is the breakdown of their study using simple analogies:

1. The Problem: The "Cheat Sheet" Effect

The researchers noticed that when AI models (LLMs) take tests about stories, movies, or TV shows, they often get perfect scores. But the authors suspect the AI isn't really understanding the story. It's just recalling the exact words it saw during its training, like a student who memorized the answer key instead of learning the math.

In real life, we want AI to be able to understand a story even if it hasn't seen that exact sentence before. We want it to use reasoning (connecting clues) rather than verbatim recall (copying from memory).

2. The Experiment: Two Ways to Test the AI

To figure out if the AI is thinking or just memorizing, the researchers created two different "test conditions" using a framework inspired by how human memory works:

The "Gist" Test (Inductive Setting):
Imagine you are taking a test, but the teacher gives you a hint: "Don't just guess the name. Look at the clues! Who is being funny? Who is being angry? Who is the detective?"
The researchers told the AI to ignore the exact names and focus on the meaning (the "gist") of the story. They asked the AI to use its knowledge of character personalities to solve the puzzle.
- Result: The AI's score dropped a little bit (about 10%). This means the hint worked! The AI was forced to stop relying on its "cheat sheet" and actually try to reason.
The "Name Swap" Test (Restrictive Setting):
This was the real knockout punch. The researchers took the story and changed all the character names.
- Instead of "Sheldon," they used "Jie Zhang."
- Instead of "Monica," they used "Bojing."
- Instead of "Ross," they used "Meilin."
  They did this to break the AI's memory. If the AI was just memorizing "Sheldon said this line," it would be completely lost when it saw "Jie Zhang said this line."
- Result: The AI's performance crashed. In some cases, accuracy dropped by nearly 45%. This proved that the AI was heavily relying on memorizing the original names and scripts, not actually understanding the story logic.

3. The Big Discovery

The study found that:

AI loves to cheat: When allowed, AI models prefer to just recall exact facts from their training data because it's easier and faster than thinking.
AI can learn to think: When the researchers forced the AI to stop using its "cheat sheet" (by changing names or giving "reasoning" hints), the AI could still solve the problems, but it got them wrong more often. This shows that the AI has the ability to reason, but it usually doesn't bother to use it if it can just memorize.
Current tests are lying: Many of the popular tests we use to say "AI is smart" are actually just measuring how well the AI has memorized famous movies and books.

4. The Solution: "Steering" the AI

The paper suggests a new way to test and train AI. Instead of just asking, "Who said this?", we should design tests that block the memorization shortcuts.

If you change the names (like swapping "Monica" for "Bojing"), the AI can't cheat.
If you give it a prompt that says, "Use logic, not memory," the AI performs better at reasoning.

Summary Analogy

Think of the AI like a student taking a history exam.

Current Benchmarks: The student gets an A because they memorized the textbook word-for-word.
The "Name Swap" Test: The teacher changes all the historical figures' names (e.g., "George Washington" becomes "John Smith"). The student fails miserably because they didn't learn the events, they just memorized the names.
The "Gist" Prompt: The teacher says, "Don't look at the names. Look at the actions. Who was the general who crossed the river?" The student has to think. They might get it right, but it takes more effort.

The Bottom Line:
This paper argues that to truly know if an AI is smart, we have to stop letting it use its "photographic memory" of famous stories. We need to test it in a way that forces it to use its brain (reasoning) rather than its hard drive (memorization). When we do that, we find that AI is much less perfect than we thought, but also capable of real understanding if we guide it correctly.

1. Problem Statement

Large Language Models (LLMs) have demonstrated remarkable performance on reasoning benchmarks, particularly in math and logic. However, recent studies suggest that much of this success stems from verbatim memorization of training data rather than genuine generalizable reasoning.

While prior work has focused on eliminating memorization in domains like math, many real-world applications (such as story understanding) require memorization-constrained reasoning. In these scenarios, models must integrate background knowledge (memorized facts) with specific narrative context to infer character motivations, relationships, and emotions. The core problem addressed by this paper is: Do current LLMs genuinely reason when solving character-centric tasks, or do they rely on surface-level recall of famous scripts?

Current benchmarks often fail to distinguish between these two modes because they allow models to "cheat" by recalling exact character names and dialogue from pretraining. The authors argue that without constraining this memorization, we cannot accurately assess a model's true reasoning capabilities.

2. Methodology

The authors propose a two-tier intervention framework inspired by cognitive science distinctions between verbatim memory (exact recall of surface details) and gist memory (semantic abstraction). They apply this framework to six character-centric story understanding benchmarks using three state-of-the-art LLMs: GPT-4o, LLaMA3.3-70B, and DeepSeek V3.

A. Datasets

CharBench: A collection of six existing NLP benchmarks covering diverse tasks:
- TVShow Guess: Speaker identification in TV dialogues.
- Screenplay: Character coreference resolution in movie scripts.
- PERSONET: Personality inference from literary passages.
- CSI: Role detection (identifying criminals) in crime shows.
- FriendsQA: Multi-turn dialogue question answering.
- SummScreen: TV episode summarization.
CharScript: A newly constructed dataset of 200 text segments from 40 fictional sources (TV, movies, novels) used specifically to test source identification and verbatim memorization.

B. The Two-Tier Framework

The study evaluates model performance under two distinct settings:

Inductive Setting (Prompt-Guided):
- Goal: To see if prompting can steer models away from rote memorization toward semantic reasoning without altering the input text.
- Method: The authors compare two prompting strategies:
  - Verbatim Prompt: Explicitly instructs the model to use direct memory retrieval without reasoning.
  - Gist Prompt: Instructs the model to analyze relationships, events, and personality traits to infer answers, explicitly forbidding direct name recall.
- Hypothesis: If models rely on memorization, Gist prompts should cause a performance drop compared to Verbatim prompts.
Restrictive Setting (Input-Modified):
- Goal: To test intrinsic reasoning ability by physically blocking access to memorized surface cues.
- Method: All character names in the input text are replaced with culturally distinct alternatives. Three levels of replacement are tested:
  - Name Masking: Replacing names with placeholders (e.g., P0, P1).
  - Same-Cultural Replacement: Swapping names with culturally consistent alternatives (e.g., Sheldon $\to$ Andrew).
  - Cross-Cultural Replacement: Swapping names with culturally distinct alternatives (e.g., English names $\to$ Chinese names).
- Hypothesis: If performance collapses under these constraints, the original benchmark scores were driven by memorization, not reasoning.

3. Key Contributions

Domain Expansion: Extends the study of memorization vs. reasoning from math/logic to character-centric story understanding, a domain where controlled use of memory is essential.
Inductive-Restrictive Framework: Introduces a novel two-tier design that separates behavioral steerability (can prompts guide reasoning?) from intrinsic robustness (does reasoning survive when memory is blocked?).
Diagnosis of Benchmark Flaws: Provides empirical evidence that current benchmarks overestimate reasoning capabilities by allowing verbatim memorization to dominate.
Demonstration of Steering: Shows that while models default to memorization, specific prompting strategies (Gist) can partially align them toward constrained reasoning.

4. Key Results

A. Memorization Reliance (Inductive Setting)

Verbatim vs. Baseline: Performance under "Verbatim" prompts was nearly identical to the baseline, confirming models naturally default to memorization.
Gist Impact: Switching to "Gist" prompts caused a non-trivial performance drop (approx. 10%). This indicates that models struggle to suppress memorization when explicitly asked to reason, but the drop is not total, suggesting some reliance on memory persists even with guidance.

B. Reasoning Under Constraints (Restrictive Setting)

Performance Collapse: When character names were replaced (especially with Cross-Cultural names), accuracy dropped significantly.
- GPT-4o: Dropped by 45.8% on the TVShow Guess task.
- LLaMA3.3-70B: Dropped by 29.5%.
- DeepSeek V3: Dropped by 31.1%.
Source Identification: In the CharScript task, Cross-Cultural Name Replacement caused the most severe "forgetting" effect, reducing source identification accuracy from ~98% (Baseline) to ~22-56%, proving that models rely heavily on specific name-to-source associations.

C. Interaction of Prompts and Constraints

Combination Effect: The combination of Name Replacement + Gist Prompting yielded the lowest accuracy, effectively isolating the model's true reasoning ability (which was found to be significantly lower than benchmark scores suggest).
Task Sensitivity: Action-oriented tasks (e.g., Speaker Guessing, Role Detection) suffered the largest drops, as they rely on direct cue-label associations. Motivation-oriented tasks (e.g., Personality Understanding) were more robust, as they naturally require semantic inference.

5. Significance and Implications

Redefining "Reasoning": The paper argues that current evaluations of LLM reasoning are flawed because they conflate "retrieval" with "reasoning." True reasoning in narrative contexts requires integrating bounded, context-relevant knowledge, not just recalling scripts.
Evaluation Standards: The authors propose that future benchmarks must include memorization-constrained settings (like name replacement) to provide a realistic assessment of a model's ability to generalize and reason.
Steering Mechanisms: The findings suggest that while LLMs are prone to memorization, they can be steered toward reasoning through specific prompting (Inductive Setting), though this does not fully eliminate the reliance on surface cues without input modification (Restrictive Setting).
Cognitive Alignment: By adopting the "verbatim vs. gist" framework, the study bridges NLP evaluation with cognitive science, offering a more nuanced view of how AI processes information compared to human cognition.

In conclusion, the paper demonstrates that the impressive performance of LLMs on story understanding tasks is largely an artifact of memorization. When surface-level cues are removed, performance degrades drastically, revealing a significant gap between current benchmark scores and genuine reasoning capabilities.