💬 NLP

Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

This paper argues that current summarization evaluation metrics fail to capture individual user needs by ignoring reader personas, demonstrating through perturbation tests and expert human evaluation that both traditional and LLM-based metrics poorly correlate with human judgments of information satisfaction.

Original authors: Isabel Cachola, William Walden, Reno Kriz, Mark Dredze

Published 2026-08-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Isabel Cachola, William Walden, Reno Kriz, Mark Dredze

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a library, but instead of books, the shelves are filled with millions of digital documents. Somewhere in this vast library, a robot is trying to write a short "summary sheet" for you. This field of science is called Natural Language Processing, and the specific task is Summarization: taking a long, complicated text and condensing it into a few sentences. For years, scientists have tried to teach computers how to write these summary sheets and then asked, "How good is it?" To answer this, they built metrics—mathematical rulers that measure things like how many words the robot used, how similar the words are to a "perfect" example, or how factually accurate the text sounds.

But here is the catch: a summary sheet isn't useful unless it fits the person holding it. If you are a medical student, you need a summary of a new vaccine that explains the chemistry and the test results. If you are a parent, you need a summary that explains if the vaccine is safe for your child and what the side effects are. The source document is the same, but your needs are totally different. For a long time, the "rulers" used to grade these summaries ignored the reader entirely. They just checked if the summary looked like a standard textbook answer, regardless of who was reading it. This paper asks a simple, vital question: Do our current rulers actually measure whether a summary satisfies your specific needs, or are they just measuring how well the robot mimics a generic style?

The authors of this paper decided to test the rulers by introducing a new concept called Information Satisfaction. Think of this as asking, "Did this summary actually give the reader the specific information they were looking for?" To do this, they didn't just look at the text; they looked at the Persona—the specific role and background of the reader, like "a biomedical researcher" versus "a family doctor."

First, the team put the popular "rulers" (the metrics) through a series of stress tests, like a video game level designed to break bad software. They took a perfect summary and started messing with it in specific ways. They added random, nonsense sentences to see if the ruler would notice the summary got worse. They chopped sentences out to see if the ruler noticed the summary got incomplete. They even rewrote the summary for a completely different audience (like changing a summary for a scientist into one for a journalist) to see if the ruler could tell the difference.

The results were a bit of a shock. Many of the most popular metrics, including the fancy new ones powered by massive AI models (called LLMs), failed the tests spectacularly. When the researchers added nonsense sentences, some of the AI judges actually gave the summary a higher score. When they shortened the summary, the scores went up and down in ways that made no sense. Most importantly, when they changed the summary to fit a different audience, the metrics often couldn't tell that the summary was now useless for the original reader. It was as if a teacher grading a math test couldn't tell the difference between a correct answer and a completely wrong one, or even between a test written for a 5th grader and one written for a PhD student.

Next, the researchers brought in real humans to see what they actually preferred. They recruited 20 experts from fields like medicine and computer science. Each expert was given a specific job (their persona) and a real question they had. The computer then generated four different summaries: some tailored to that expert's background, and some generic ones. The experts then played a "tournament," picking the best summary in head-to-head matchups.

Here is the twist: The experts consistently preferred the summaries that were tailored to their specific background. However, when the researchers compared the experts' choices to the scores given by the computer metrics, the computers were almost completely clueless. The metrics agreed with the human experts no better than if they had just been guessing. Even the new "Persona" metrics, which were specifically designed to check if the right information was included, failed to match human judgment. In fact, some of these new metrics agreed with humans less than random chance would suggest.

The paper concludes that we are currently using the wrong tools to measure the value of summaries. The old rulers and the new AI judges are great at checking if a summary sounds fluent or matches a reference text, but they are terrible at answering the most important question: "Does this help this specific person?" The authors suggest that until we build evaluation systems that truly understand who the reader is and what they need to learn, we won't be able to tell if a summary is actually good or just looks good on paper.

Technical Summary: Information Satisfaction: A Reader-Centered Axis for Summarization Evaluation

Problem Statement

Current automatic summarization evaluation largely relies on metrics that treat summary quality as an intrinsic property, independent of the reader. Standard approaches, such as lexical overlap (ROUGE) and embedding-based similarity (BERTScore), compare candidate summaries against reference texts or source documents. While recent advancements utilize Large Language Models (LLMs) as judges to assess dimensions like coherence and factuality, these methods still generally assume a "one-size-fits-all" notion of quality.

The authors argue that this framing obscures a fundamental reality: a summary's utility is realized only when it serves the specific informational needs of a particular reader. For instance, a biomedical researcher and a family physician reading the same paper on mRNA vaccines have divergent needs (methodological detail vs. clinical implications). Existing metrics fail to capture this mismatch because they do not model the reader's background, expertise, or "persona." Furthermore, while Query-Focused Summarization (QFS) attempts to address user needs via queries, users rarely provide sufficient context in a single short query to distinguish complex personas. Consequently, there is a lack of evaluation frameworks that measure information satisfaction—the extent to which a summary resolves the specific informational need of a reader given their persona.

Methodology

The study investigates whether existing summarization metrics can measure information satisfaction through two primary research questions (RQs):

RQ1: Are evaluation metrics sensitive to variations in informational content and reader persona?
RQ2: Do evaluation metrics agree with human judgments of information satisfaction?

Experimental Setup for RQ1 (Perturbation Tests)

The authors evaluated a broad set of metrics across four scientific corpora (arXiv, PubMed, SciTLDR, eLife) using 50 documents per dataset. They applied five controlled perturbations to reference summaries to test metric robustness:

Distractor Sentences: Appending unrelated sentences (Expected: Score decrease).
Incremental Addition: Reconstructing the summary sentence-by-sentence (Expected: Score increase).
Lengthen/Shorten: Rewriting the summary to change length without altering content (Expected: Score stability).
Different Audience: Rewriting the summary for a different target audience (e.g., student vs. expert) (Expected: Score decrease relative to the original audience).

Metrics tested included:

Lexical/Reference-based: ROUGE, BLEU, chrF, METEOR.
Embedding/Model-based: BERTScore, BLANC, SummaQA, SUPERT.
LLM-as-Judge: Prompts using Llama-3.3-70B and Prometheus-7B evaluating relevance, consistency, fluency, coherence, and informativeness.
Persona-Specific Variants: Custom metrics (Persona Precision and Persona Recall) that decompose summaries into "nuggets" and judge their relevance to a specific persona.

Experimental Setup for RQ2 (Human Evaluation)

The authors conducted an expert human evaluation involving 20 volunteer annotators from diverse domains (CS, medicine, materials science).

Protocol: Annotators provided a self-reported persona profile and a genuine information need (query). The system retrieved relevant papers and generated four candidate summaries using a 2x2 design: two models (Llama-3.3-70B, DeepSeek-V3.1) crossed with two conditions (with persona profile, without).
Ranking: Annotators ranked the four summaries via a pairwise single-elimination tournament.
Analysis: The authors measured the correlation between automatic metric scores and human pairwise preferences using Krippendorff's $\alpha$ .

Key Results

Perturbation Testing (RQ1)

Most metrics failed basic robustness tests regarding informational content and audience shifts:

Distractor Injection: Lexical metrics performed as expected, but many "content-aware" metrics (BERTScore, SummaQA) and LLM judges were inconsistent or failed to detect off-topic content.
Incremental Addition: Lexical metrics showed negative correlations (scoring partial summaries higher than complete ones due to length normalization artifacts). Only the Llama-3.3-70B judge showed positive correlation, though with low monotonicity.
Length Sensitivity: Almost all metrics, including BERTScore and most LLM judge dimensions, were highly sensitive to length changes, failing the stability test ( $\rho \approx -1.0$ ).
Different Audience: This was the most critical failure. Many LLM-based metrics (especially under Prometheus-7B) moved in the wrong direction, rating audience-shifted summaries as higher quality. While the majority of metrics failed, a few specific variants showed the expected directional signal: under the Llama-3.3-70B judge, the Annotator-style judge passed with $\rho = -0.77$ , and under Prometheus-7B, Consistency ( $\rho = -0.40$ ) and Persona Recall ( $\rho = -0.60$ ) passed. However, compared to the strong responses these metrics showed on distractor injection, their responses to audience shifts were generally weak or non-existent for the majority of the metric suite. Lexical metrics (ROUGE-1/2, chrF, METEOR, coverage, density, compression) and BERTScore sat exactly at the pass threshold ( $\rho = -0.40$ ), giving only a weak directional signal, while ROUGE-L, BLEU, BLANC, SummaQA, and SUPERT failed outright. Crucially, most evaluated metrics were insensitive or anticorrelated with audience-shift rewrites, the dimension where information satisfaction varies most between users.

Human Evaluation Correlation (RQ2)

Human Preferences: Annotators generally preferred summaries generated by DeepSeek-V3.1 over Llama-3.3-70B. Personalized summaries (conditioned on the persona) were preferred over generic ones, but this effect was heavily dependent on the underlying model's capability; DeepSeek's personalized outputs were significantly preferred, while Llama's personalized outputs did not show a strong advantage.
Metric Correlation: No metric family achieved agreement with human preferences significantly above chance ( $\alpha = 0$ $α = 0$ ).
- Traditional metrics (ROUGE, BERTScore) ranged from $\alpha = -0.029$ to $0.173$.
- LLM-as-judge metrics ranged from $\alpha = -0.179$ to $0.158$.
- Persona-aware variants (Persona Precision/Recall) performed particularly poorly, with $\alpha$ values of $-0.150$ under Prometheus-7B, indicating systematic disagreement with human judgment rather than mere noise.

Key Contributions

Definition of Information Satisfaction: The paper operationalizes "information satisfaction" as a reader-centered evaluation axis, defined by the intersection of a specific query and a reader's persona (role, expertise, and context).
Robustness Analysis: The study demonstrates that popular summarization metrics, including state-of-the-art LLM-as-judge approaches, fail to detect meaningful changes in informational content relative to a reader's needs. They are often more sensitive to surface-level corruptions (like length) than to audience-relevant shifts.
Human Evaluation Benchmark: The authors provide a dataset and protocol where human judgments are anchored to specific personas and queries, revealing that current metrics have near-zero correlation with these user-centered preferences.
Persona Metric Limitations: The paper shows that simply adding persona information to a metric's prompt (as done in their custom Persona Precision/Recall metrics) is insufficient to capture information satisfaction if the underlying evaluation logic does not properly weight information importance or account for the reader's prior knowledge.

Significance and Claims

The paper claims that existing summarization metrics are insufficient measures of information satisfaction. The authors argue that progress in user-centered summarization cannot be achieved by simply applying current metrics to personalized outputs. Instead, the field requires evaluation frameworks that explicitly model:

Who the summary is for.
What they seek to learn.
How to weight information based on its importance to that specific reader.

The authors conclude that the bottleneck is not the absence of persona data in prompts, but the fundamental inability of current metrics to simulate the comparative judgment a human makes when determining if a summary satisfies their specific informational needs. The work provides a foundation and dataset for future research into metrics that truly measure information satisfaction.