💬 NLP

Summarization is Not Dead Yet

Despite claims that large language models have surpassed human capabilities in summarization, a comprehensive multi-track evaluation reveals that while LLMs excel in fluency and coherence, human-written references remain superior in informativeness, faithfulness, and factual reliability.

Original authors: Dongqi Liu, Chenxi Whitehouse, Zheng Zhao, Zhuchen Cao, Jian Li, Yabiao Wang

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dongqi Liu, Chenxi Whitehouse, Zheng Zhao, Zhuchen Cao, Jian Li, Yabiao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Did AI Win the Game?

Recently, many people started saying that Large Language Models (AI) have "solved" the problem of writing summaries. The argument was: "AI writes summaries that are smoother, faster, and sometimes even better than humans. Why do we need researchers to study this anymore?"

This paper says: Not so fast.

The authors argue that while AI has gotten very good at the surface of writing, it still hasn't reached the depth of human understanding. They ran a massive, multi-layered test to prove that human summaries are still superior in the ways that actually matter.

The Experiment: A Multi-Track Race

Imagine a race where five different AI models (like GPT, Claude, and Gemini) race against human writers to summarize five different types of content: long Chinese novels, science news, multiple news articles, video lectures, and EU legal documents.

The researchers didn't just ask, "Who won?" They looked at the race through four different lenses:

1. The "First Impression" Test (Human & AI Judges)

The Analogy: Imagine you are hiring a speaker.

The AI's Strength: The AI speakers are incredibly smooth. They don't stutter, they use perfect grammar, and their sentences flow like a river. If you just listen to the style, the AI wins.
The Human's Strength: The human speakers might be a little less polished, but they pack more actual information into their speech. They tell you the "why" and the "how," not just the "what."

The Result: When judges looked at just the style (fluency), AI won. But when they looked at information density and faithfulness (did they tell the truth and include the important details?), humans won by a landslide. The paper found that previous studies were fooled because they only looked at the "smoothness" and missed the "substance."

2. The "Fact-Checker" Test

The Analogy: Imagine a student writing a book report.

The AI's Mistake: Sometimes, the AI tries to sound smart by making up facts that aren't in the book but could be true in the real world. It's like a student adding a random fact about the author's life that they read on Wikipedia, even though the teacher said, "Only use the book."
The Human's Move: Humans also add outside knowledge, but they do it intentionally to help you understand the context.

The Result: The paper found that when you check the facts against the real world (not just the source text), humans are more reliable. The AI tends to "hallucinate" (make things up) more often when it tries to reason or synthesize complex ideas. The old way of judging AI was unfair because it treated any outside knowledge as a "lie," which penalized humans who were trying to be helpful. When judged fairly, humans are still the more trustworthy fact-checkers.

3. The "Linguistic DNA" Test

The Analogy: Think of language like a garden.

The AI Garden: The AI garden is very neat. It uses the same types of flowers (words) over and over. The paths (sentence structures) are all straight and simple. It looks tidy, but it's a bit boring and repetitive.
The Human Garden: The human garden is wilder. It has a huge variety of flowers (vocabulary), winding paths (complex sentence structures), and layers of depth.

The Result: The AI summaries are linguistically "shallow." They use simpler sentence structures and repeat the same words. Human summaries are more diverse and complex. This "neatness" is actually why the AI sounds so fluent, but it comes at the cost of richness and information density.

4. The "Why It Matters" Test (Downstream Effects)

The Analogy: Imagine a summary is a map given to a tourist.

If the map is smooth and pretty (AI) but misses a few key streets or gets a landmark wrong, the tourist gets lost.
If the map is a bit rougher (Human) but includes all the shortcuts and accurate details, the tourist gets where they need to go.

The Result: The paper warns that if we use these "smooth but shallow" AI summaries in real-world tools (like search engines, legal analysis, or medical records), the errors will spread. If the summary misses a key detail, the AI system building on top of it will also fail.

The Final Verdict

The paper concludes with a clear metaphor: AI has raised the "floor" of summarization, but it hasn't reached the "ceiling."

The Floor: The baseline quality of summaries is now much higher because AI is so good at writing grammatically correct, fluent text.
The Ceiling: The absolute best possible summary (the human level) is still higher than what AI can currently do.

In short: Summarization is not dead. AI is a great tool for making things look good, but humans are still the masters of making things mean something. We still need researchers to figure out how to get AI to understand the deep stuff, not just the pretty stuff.

Technical Summary: Summarization is Not Dead Yet

Problem Statement

The rapid advancement of Large Language Models (LLMs) has fueled a narrative suggesting that model-generated summaries now rival or surpass human-written references, potentially rendering summarization a "solved" problem. Previous studies supporting this view often rely on holistic preference judgments, LLM-as-Judge protocols prone to bias, and source-only factuality checks that conflate valid external knowledge with hallucinations. This paper challenges the conclusion that summarization research is obsolete, arguing that current evaluations fail to disentangle surface fluency from deep informational fidelity. The authors posit that while LLMs have raised the baseline quality of summarization, they have not yet reached the "ceiling" of human capability, particularly regarding informativeness, faithfulness, and linguistic diversity.

Methodology

To re-examine the state of summarization, the authors conducted a multi-track evaluation involving five state-of-the-art LLMs (GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, Qwen3.5, and Kimi-K2.5) across five diverse datasets (CNNSum, SciNews, DiverseSumm, VISTA, and EurLexSum). The evaluation framework consists of four distinct tracks:

Controlled Human Assessment: Annotators rated summaries on four specific dimensions (informativeness, faithfulness, coherence, conciseness) using a 1-to-5 Likert scale and performed blind pairwise rankings. Inter-annotator agreement was monitored via Krippendorff's $\alpha$ .
Bias-Mitigated LLM-as-Judge: The same five models acted as judges for all six sources (including human references). To mitigate self-preference and position bias, each model's own output was excluded when it served as a judge, and rankings were derived from the consensus of the remaining four models.
Factuality Verification: The authors employed four complementary metrics (FaStFact, SAFE, FActScore, VeriScore) to decompose summaries into atomic claims. Crucially, verification was performed against external knowledge sources rather than relying solely on the source document, distinguishing between genuine hallucinations and valid extrinsic contextualization.
Corpus-Level Linguistic Analysis: The study analyzed lexical, syntactic, and discourse-level properties using seven metrics, including Type-Token Ratio (TTR), dependency tree depth, noun phrase modifiers, topic progression, and compression ratios.

Key Contributions

Dimension-Specific Evaluation: The authors demonstrate that holistic preference judgments often conflate surface fluency with information density. When evaluated independently, human summaries retain significant advantages in informativeness and faithfulness, whereas LLM outputs are preferred primarily for coherence and conciseness.
Refined Factuality Verification: By introducing external knowledge verification, the study reveals that human references are more reliable for claims requiring reasoning or synthesis. The authors show that source-only protocols systematically disadvantage human references, which often add valid extrinsic context, while penalizing LLMs less for generation-time fabrication.
Linguistic Divergence Analysis: The paper provides evidence that LLM summaries exhibit lower lexical diversity, shallower syntactic structures, and weaker discourse-level organization compared to human summaries. This suggests a pattern of stylistic homogeneity across different model families.
Robustness Checks: The findings hold across different generation configurations (temperature, prompting strategies) and are validated using both human annotators and bias-mitigated LLM judges.

Results

Human vs. Model Preference: In pairwise human evaluations, human references won against all five models with win rates ranging from 0.66 to 0.90. This advantage was most pronounced in informativeness (+0.17 to +0.51) and faithfulness, while models often outperformed humans in coherence.
LLM-as-Judge Consistency: Bias-mitigated LLM judges corroborated human findings, awarding human summaries win rates of 0.70 to 0.95. Judges consistently favored human outputs on content-oriented dimensions but showed a shift toward model outputs on form-oriented dimensions.
Factuality: Human summaries achieved higher factuality scores across all four verification metrics (average margins of 0.04 to 0.13). When evaluated against external knowledge, the human advantage became more pronounced, particularly in datasets like SciNews where valid external context is expected.
Linguistic Properties: Human summaries demonstrated higher lexical diversity (TTR and MATTR), deeper dependency trees, and more complex noun phrase modifiers. Conversely, models adhered more strictly to source ordering and applied less aggressive compression, resulting in shallower syntactic structures.

Significance and Claims

The paper asserts that summarization is not dead yet. The authors argue that current LLMs have successfully raised the "floor" of summarization quality, making them competent for surface-level tasks, but the "ceiling" of their performance remains below human capabilities in terms of deep reasoning, information compression, and faithful synthesis.

The significance of this work lies in:

Correcting the Narrative: It challenges the claim that general-purpose LLMs have solved summarization, highlighting persistent gaps in informativeness and faithfulness that single-track evaluations miss.
Methodological Rigor: It advocates for dimension-specific evaluation and external knowledge verification to avoid conflating fluency with quality and to properly distinguish between hallucination and valid contextualization.
Future Directions: The findings suggest that future research should prioritize faithful reasoning, information compression, and linguistically diverse generation, particularly for downstream applications like Retrieval-Augmented Generation (RAG), professional workflows, and knowledge management where factual precision is critical.

The authors conclude that summarization remains an open research problem, serving as a vital testbed for advancing the capabilities of AI in faithful reasoning and information handling.