Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?

This paper experimentally evaluates whether Large Language Models summarize well-known books better using their internal training knowledge or by processing the full text, revealing that while full-text access generally yields more detailed summaries, internal knowledge can sometimes outperform it, thereby challenging the reliability of long-context summarization.

The Gist

Imagine you are trying to write a book report for a famous novel, like Harry Potter or The Great Gatsby. You have two ways to do it:

1. The "Memory" Method: You close your eyes, rely entirely on what you remember from reading the book years ago, and write the summary from scratch.
2. The "Reading" Method: You sit down with the actual book in front of you, read it cover-to-cover, and write the summary based on what you see on the page right now.

For a long time, we assumed that Method 2 (Reading) would always be better. After all, you have the facts right in front of you, so you can't forget anything or make things up, right?

This paper asks a very interesting question: Is that actually true for Artificial Intelligence (AI)?

The Experiment: AI vs. The Bookshelf

The researchers took 25 famous books (like Dracula, Pride and Prejudice, and The Divine Comedy) and asked two super-smart AI models (GPT-4.1 and Gemini-2.5) to summarize them.

They tested the AI in two scenarios:

• Scenario A (Remembering): They told the AI, "Summarize this book," but didn't give them the text. The AI had to rely on its internal "brain" (the data it learned while being trained).
• Scenario B (Reading): They gave the AI the entire text of the book (which is huge, sometimes half a million words) and said, "Summarize this text, don't use your memory."

Then, they had the AIs judge each other's work to see which summary was better.

The Big Surprise: Sometimes, Memory Wins!

Here is the twist: For most books, "Reading" was indeed better. When the AI had the full text, it could give a more detailed and accurate summary.

However, for some specific books, the AI did a better job when it relied on its memory than when it tried to read the whole book.

Think of it like this:

• The "Reading" Problem (The Lost in the Middle): Imagine trying to read a 1,000-page novel in one sitting. By the time you get to page 900, you might have forgotten the details from page 50. The AI has a similar issue. Even with a "giant brain" (a huge context window), when you feed it a massive book, it sometimes gets confused, forgets the beginning, or gets lost in the middle. It's like trying to drink from a firehose; too much information at once can cause a spill.
• The "Memory" Advantage: For some complex or poetic books (like Ulysses or The Divine Comedy), the AI's internal training data had already "digested" the story perfectly. It knew the main points, the characters, and the ending without getting overwhelmed by the sheer volume of text. In these cases, the AI's "gut feeling" (memory) was actually more accurate than its attempt to read the whole thing.

The Verdict: Reading vs. Remembering

The paper concludes that while having the full text is usually the safest bet, it's not always the best.

• For simple, straightforward stories: Reading the book is better.
• For very long, complex, or poetic stories: The AI's internal memory can sometimes outperform its ability to process the whole book at once.

Why Does This Matter?

This is a wake-up call for how we use AI. We often assume that if we just give the AI "more data" (the whole book), it will do a better job. This study shows that more data isn't always better. Sometimes, the AI gets overwhelmed by the volume of text and makes mistakes it wouldn't have made if it just relied on what it already knew.

In short: For AI, sometimes it's better to trust its memory than to force it to read the whole book, especially if the book is a massive, complicated masterpiece. It turns out that for some tasks, remembering is actually better than reading.

Technical Summary

Here is a detailed technical summary of the preprint "Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?"

1. Problem Statement

The paper addresses a critical question in the era of Large Language Models (LLMs) with massive context windows (up to millions of tokens): Is it more effective for an LLM to summarize a book by processing the full text (external memory) or by relying solely on its internal parametric knowledge acquired during training?

While modern LLMs can theoretically ingest entire books in a single prompt, they also possess extensive internal knowledge of well-known classics. The authors investigate whether providing the full text actually improves summary quality or if the model's internal "memory" (parametric knowledge) is sufficient, or even superior, in certain cases. This touches on the interaction between parametric memory (weights) and dynamic context (input), and the phenomenon where models might "hallucinate" or ignore context in favor of pre-trained knowledge.

2. Methodology

The study employs a controlled experimental framework to compare two modes of summarization across 25 diverse, well-known literary works.

• Dataset: 25 public domain books covering various genres (novels, poetry, drama), authors, and lengths (ranging from ~30k words in A Doll's House to ~500k words in The Count of Monte Cristo).
• Models: Two state-of-the-art LLMs with 1-million-token context windows were selected: GPT-4.1 and Gemini-2.5-Flash.
• Experimental Conditions:
  1. Internal Knowledge (Remembering): The model is prompted to summarize a specific book title without providing the text, relying entirely on training data.
  2. External Knowledge (Reading): The model is provided the full text of the book and instructed to summarize only based on the provided text, explicitly forbidden from using internal knowledge.
• Evaluation Protocol (LLM-as-a-Judge):
  • Summaries were generated at two temperature settings (0.4 for consistency, 0.9 for creativity), with 5 generations per condition per book.
  • A pairwise comparison framework was used where the LLMs acted as judges. They compared an "Internal" summary against an "External" summary for the same book.
  • Scoring: Judges assigned a score of 1 (Internal better), 0 (Equal), or -1 (External better).
  • Bias Mitigation: To detect self-bias, the study utilized a cross-evaluation matrix: GPT-4.1 judged GPT-4.1's outputs, GPT-4.1 judged Gemini's outputs, and vice versa.
  • Aggregation: For each book, 25 pairwise comparisons were made (5 internal × 5 external), resulting in a total score range of -25 to +25.

3. Key Contributions

1. First Comparative Evaluation: The paper presents the first systematic evaluation comparing LLM book summarization using internal parametric memory versus full-text external context using state-of-the-art long-context models.
2. Open Repository: The authors released the code, the specific book datasets, and all generated summaries to ensure reproducibility.
3. Counter-Intuitive Findings: The study demonstrates that while full-text access generally yields better results, internal knowledge summaries can outperform full-text summaries for specific types of literature (e.g., complex narratives, poetry, or very long texts).
4. Analysis of Failure Modes: The paper identifies specific failure modes in long-context processing, such as the "Lost in the Middle" phenomenon, where models struggle to integrate information from distant parts of a long text, leading to better performance when relying on compressed internal knowledge.

4. Key Results

• General Trend: Across the majority of the 25 books, external summaries (Reading) received negative scores, indicating they were consistently preferred over internal summaries. Providing the full text generally improves detail and accuracy.
• The Exception (Remembering > Reading): For specific outliers, internal summaries (Remembering) achieved positive scores, outperforming the full-text approach. Notable examples include:
  • The Divine Comedy and Hamlet: Gemini-2.5-Flash generated significantly better summaries from internal knowledge.
  • Ulysses and The Count of Monte Cristo: Showed high variance, with internal summaries often preferred, particularly at higher temperatures.
• Model-Specific Behaviors:
  • Gemini-2.5-Flash: Showed a tendency to produce more consistent internal summaries but struggled with the structural complexity of The Divine Comedy when processing the full text (likely due to a rigid chapter-by-chapter processing failure).
  • GPT-4.1: As a judge, it tended to favor internally generated summaries more than Gemini did.
• Impact of Complexity: The "Reading" approach struggled most with:
  • Structural Complexity: Non-linear narratives (e.g., Ulysses).
  • Length: Extremely long texts where context fragmentation occurs.
  • Genre: Poetry and drama, where internal knowledge of the work's themes and structure seemed to aid summarization more than raw text processing.

5. Significance and Implications

• Rethinking Long-Context Processing: The results challenge the assumption that "more context is always better." For well-known, complex, or structurally unique texts, the cognitive load of processing millions of tokens may introduce noise or fragmentation that degrades performance compared to the model's distilled internal representation.
• Limitations of Current LLMs: The study highlights that even with million-token windows, LLMs still suffer from "Lost in the Middle" and context fragmentation issues, particularly with massive or non-linear texts.
• Evaluation Caution: The findings suggest that for specific domains (classics, complex literature), relying on a model's parametric memory might be a viable, and sometimes superior, strategy for summarization, reducing the computational cost of ingesting full texts.
• Future Research: The paper calls for further investigation into why internal knowledge outperforms external context in specific cases and whether this holds true for less familiar texts or different model architectures.

Conclusion: While "Reading" (using full text) is generally superior for summarization, "Remembering" (using internal knowledge) is not obsolete. In cases of high structural complexity or extreme length, a model's pre-trained knowledge can sometimes bypass the limitations of context window processing, yielding higher-quality summaries.