💬 NLP

Chain of Summaries: Summarization Through Iterative Questioning

The paper introduces Chain of Summaries (CoS), a dialectical method that iteratively refines initial summaries through questioning to create information-dense, plain-text repositories that significantly outperform existing baselines in downstream QA tasks while reducing token usage and enhancing LLM accessibility to web content.

Original authors: William Brach, Kristián Košťál, Lukas Galke Poech

Published 2026-02-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: William Brach, Kristián Košťál, Lukas Galke Poech

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Wall of Text"

Imagine you have a massive, 100-page instruction manual for a complex machine. You want to ask a very smart robot (an AI) a specific question about it, like, "What color is the safety valve?"

If you feed the robot the entire 100-page manual, two things go wrong:

The Robot Gets Overwhelmed: The robot has a limited "working memory." Reading 100 pages at once is like trying to drink from a firehose; it might miss the important details in the middle.
It's Too Slow and Expensive: Reading every single word takes a long time and costs a lot of money (in computing power).

Usually, people try to solve this by asking the robot to "summarize" the manual. But if you just ask for a summary once, the robot might give you a vague overview that misses the specific details you need later.

The Solution: The "Chain of Summaries" (CoS)

The authors of this paper propose a clever new way to summarize. They call it Chain of Summaries (CoS).

Instead of asking the robot to summarize once and hoping for the best, they use a method inspired by an old philosopher named Hegel. Think of it as a three-step conversation between a Writer, a Skeptic, and a Refiner.

Step 1: The Thesis (The First Draft)

The robot reads the long document and writes a first draft summary.

Analogy: This is like a student writing a first draft of an essay. It's okay, but it probably has holes in it.

Step 2: The Antithesis (The Skeptic)

Here is the magic trick. The robot doesn't just stop. It acts as a Skeptic. It looks at the first draft and asks itself: "What questions can I NOT answer using this summary?"

Analogy: Imagine a strict teacher looking at the student's draft and saying, "You forgot to mention the safety valve color! You didn't explain how the engine starts! You missed the warranty date!"
The robot generates a list of these "missing" questions.

Step 3: The Synthesis (The Refinement)

The robot takes the original document and the list of "missing" questions. It rewrites the summary to specifically fill those holes, making it denser and more useful.

Analogy: The student goes back to the textbook, finds the answers to the teacher's questions, and updates the essay.

The Loop: The robot repeats this cycle (Draft → Skeptic → Refine) many times. With every loop, the summary gets smarter, catching more details and anticipating more questions.

Why Is This Better?

The paper tested this method on real-world data (like trivia questions and reading comprehension tests). Here is what they found:

It's a "Super-Shortcut": The final summary is tiny (like a 1-page cheat sheet) compared to the original 100-page manual.
It's Smarter Than the Original: Surprisingly, the robot could answer questions better using the tiny summary than it could using the full 100-page manual. Why? Because the summary removed all the "noise" and distractions, leaving only the pure facts.
It Saves Money: Because the summary is so small, the robot doesn't have to read as much. The authors calculated that after about 14 questions, the cost of creating the summary pays for itself. If you ask the robot about this document 100 times, you save a massive amount of money and time.
It Works Without Human Help: The robot can generate its own "Skeptic" questions. You don't need a human to write a list of questions for it to work.

A Real-World Example: Sherlock Holmes

The paper used a story about Sherlock Holmes to show how this works.

A normal summary might say: "Sherlock Holmes is a famous detective created by Arthur Conan Doyle who lives in London." (Good, but vague).
The Chain of Summaries would ask itself: "What year did he appear? Who inspired him? What is his address?"
The final CoS summary would say: "Sherlock Holmes, created by Arthur Conan Doyle in 1887, lives at 221B Baker Street. He was inspired by Joseph Bell, distrusts women, and uses abductive reasoning. He appeared in 'The Strand' magazine."

This tiny paragraph contains all the specific facts a robot might need to answer any trivia question about Holmes, without needing to read the whole book.

The Bottom Line

The Chain of Summaries is like a personal librarian who doesn't just give you a book, but reads the book, anticipates every question you might ever ask, and writes you a perfect, one-page cheat sheet that contains only the answers you need.

It makes AI faster, cheaper, and smarter, while still keeping the information accurate enough for humans to check if they want to. It turns the "firehose" of the internet into a manageable "water bottle" that fits perfectly in the robot's pocket.

1. Problem Statement

Large Language Models (LLMs) are increasingly used to navigate and process web content. However, they face two primary challenges when interacting with raw web data:

Format Incompatibility: Much web content is not in an LLM-friendly format (e.g., complex HTML, multimedia), making it difficult to parse.
Context Limitations: The sheer volume of text often exceeds the context window limits of LLMs, forcing models to truncate information or suffer from the "lost-in-the-middle" phenomenon, where critical details in the middle of long contexts are ignored.

Existing solutions, such as converting websites to Markdown, do not solve the issue of information density. The authors propose a need for information-dense, plain-text summaries that act as universal, model-agnostic caches. These summaries must capture the essence of the original content across various, potentially unpredictable, dimensions to answer future questions without requiring access to the full source document.

2. Methodology: Chain of Summaries (CoS)

The core innovation is the Chain of Summaries (CoS) framework, which is inspired by Hegel's dialectical method (Thesis $\rightarrow$ Antithesis $\rightarrow$ Synthesis). The process iteratively refines a summary to maximize its utility for downstream Question Answering (QA) tasks.

The Dialectical Process

Thesis (Initial Summary): An LLM generates a zero-shot summary ( $s_0$ ) from the source document ( $D$ ). This initial draft inevitably contains gaps and limitations.
Antithesis (Question Generation): The system generates a set of synthetic questions ( $Q$ ) based on the source document. These questions act as a challenge to the summary's comprehensiveness.
Evaluation: The current summary ( $s_{k-1}$ ) is tested against the generated questions. Questions that the summary fails to answer correctly are identified as "unanswered" ( $U_k$ ).
Synthesis (Iterative Refinement): A synthesis engine takes the source document, the current summary, and the set of unanswered questions as input. It produces an improved summary ( $s_k$ ) that incorporates the missing information while maintaining conciseness.
Iteration: This cycle repeats for $i$ iterations. The final summary ( $s^*$ ) is selected based on its performance on a validation set of questions (never seen during refinement) to prevent overfitting.

Key Design Choices

Synthetic vs. Human Questions: The method utilizes LLMs to generate synthetic questions, eliminating the need for expensive human-annotated QA pairs during the training/refinement phase.
Hyperparameters: The process is controlled by the number of iterations ( $i$ ) and the number of questions per iteration ( $i_q$ ). Experiments showed that distributing questions over many iterations (e.g., 10 iterations with 1 question each) outperforms presenting all questions at once.
Model Agnosticism: The resulting summary is plain text, making it compatible with any downstream LLM, unlike dense vector embeddings.

3. Key Contributions

The CoS Framework: A novel, dialectical approach to summarization that optimizes for downstream QA performance rather than traditional metrics like fluency or ROUGE.
Superiority over Baselines: Empirical evidence showing CoS outperforms zero-shot summarization, specialized abstractive models (BRIO, PEGASUS), and other iterative methods (Chain of Density).
Efficiency Analysis: A demonstration that CoS summaries reduce token consumption by over 98% compared to full documents, with a "break-even" point (where the cost of generating the summary is amortized) occurring after only 13–15 queries.
Synthetic Question Viability: Proof that LLM-generated synthetic questions are as effective as human-crafted questions for guiding the refinement process.

4. Experimental Results

The authors evaluated CoS on TriviaQA, TruthfulQA, and SQuAD v1.1 using three LLM backbones: GPT-4o-mini, Llama-3.2:3B, and Qwen-2.5:7B.

Performance Gains:
- CoS outperformed zero-shot baselines by up to 66% (e.g., Llama-3.2:3B on TruthfulQA).
- It surpassed specialized summarization models (BRIO, PEGASUS) by up to 27%.
- It outperformed the "Chain of Density" method, particularly with smaller models (e.g., +35% with Llama-3.2:3B).
The "Summary Advantage": In several cases (e.g., TriviaQA with GPT-4o-mini), the CoS summary achieved a higher QA score (0.80) than the full source document (0.76). This suggests that removing noise and focusing on high-density facts improves LLM reasoning.
Generalization: The method successfully generalized to unseen questions, proving that the summaries captured robust knowledge rather than just memorizing the training questions.
Cost Efficiency:
- For a 10,000-token document, the CoS summary required only ~170 tokens.
- This resulted in a 98.3% reduction in inference tokens per query.
- The break-even point for the one-time indexing cost was reached after approximately 14 queries.

5. Significance and Implications

Web Accessibility for AI: CoS offers a practical solution for website maintainers to make their content "LLM-ready." By hosting a CoS summary, they ensure their content is accessible to agents without consuming massive context windows.
Human-in-the-Loop: Because the output is plain text, human overseers can easily read, validate, and correct the summaries, ensuring accuracy and preventing the spread of hallucinations.
Green AI: By drastically reducing the number of tokens processed during inference, CoS lowers the energy consumption and carbon footprint of LLM operations, especially for frequently accessed documents.
Scalability: The method is particularly valuable for retrieval-augmented generation (RAG) systems, where it acts as a high-efficiency cache that reduces latency and computational costs.

In conclusion, the paper demonstrates that by iteratively refining summaries through a dialectical process of questioning and synthesis, it is possible to create information-dense artifacts that are not only shorter but more useful to LLMs than the original source text.