💬 NLP

Abstractiveness Metrics for Evaluating Text Summarization: A Refined Formulation with Empirical Validation

This paper introduces Reference Abstraction, Summary Abstraction, and Abstraction Ratio as principled heuristic metrics to quantify the degree of abstraction in text summaries, demonstrating through empirical validation on the XSUM dataset that these measures effectively distinguish between extractive and abstractive models while identifying potential hallucinations.

Original authors: Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC0 1.0

Original authors: Praveenkumar Katwe, Rakesh Chandra Balabantaray, Kali Prasad Vittala

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're a teacher grading a student's book report. The student has to summarize a 50-page novel into a single page.

For years, teachers have used a tool called ROUGE to grade these reports. ROUGE is like a strict copy-paste detector. It gives you a high score if the student's report uses the exact same words and sentences as the "perfect" answer key written by another teacher. But here's the catch: ROUGE doesn't care if the student actually read the book. If the student just copied the answer key word-for-word, they get an A. If they wrote something completely new but missed a few keywords, they get a C.

The authors of this paper, Praveenkumar, Rakesh, and Kali, say, "Wait a minute! We need a way to measure how much the student is actually thinking and rewriting (abstracting) versus just copying (extracting)." They also want to catch students who are making things up (hallucinating) that aren't in the book at all.

To do this, they invented three new "super-metrics": Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR).

The Three New Tools

Think of the original book as a Source, the perfect answer key as the Reference, and the student's work as the Summary.

Reference Abstraction (RA): This measures how much the answer key itself is rewriting the book. Is the key just copying sentences, or is it summarizing them creatively?
Summary Abstraction (SA): This measures how much the student's summary is rewriting the book.
Abstraction Ratio (AR): This is the star of the show. It compares the student's creativity to the answer key's creativity. It asks: "Is the student rewriting more, less, or about the same amount as the teacher did?"

How They Measure It (The Secret Sauce)

The authors didn't just count words. They used a special formula involving a harmonic mean (a type of average that punishes extreme differences in size) and a cubic non-overlap factor.

Here is the magic trick: They take the part of the text that doesn't overlap with the original book and raise it to the third power (cubed).

Why cube it? Imagine you are trying to spot a tiny difference between two almost-identical twins. If you look at them normally, they look the same. But if you put them under a microscope that magnifies differences by 1000x, suddenly you see a tiny freckle on one that the other doesn't have.

If a student copies 95% of the text, the "new" part is tiny.
If they copy 98%, the "new" part is even tinier.
By cubing the difference, the math blows up that tiny gap. A 3% difference in copying becomes a 16-times bigger difference in the score. This makes the metric super-sensitive to students who are barely trying to rewrite anything.

What They Found (The Experiment)

The team tested this on 100 articles from the XSUM dataset (which uses very creative, short summaries) and the CNN/DailyMail dataset (which uses more copy-paste style summaries). They ran four different AI models through the test: BART-large-cnn, Pegasus-xsum, DistilBart, and MT5-small.

Here is what the numbers told them:

The Copycats: The models named BART and DistilBart were the ultimate copycats. On the XSUM test, they had an SA (Summary Abstraction) score between 0.12 and 0.26. This means they were barely rewriting anything; they were just stitching together sentences from the original text.
The Creatives: The models named Pegasus and MT5 were the writers. On XSUM, their SA scores were between 1.15 and 1.99. They were actually rewriting the content.
The Hallucination Trap: This is the most important finding, but it depends on the type of "answer key" you are using.
- When the "answer key" (Reference) is already very creative (like in XSUM), the models generally rewrote less than the teacher (AR < 1).
- However, when the "answer key" is mostly just copying the book (like in CNN/DailyMail), something weird happened with the creative models (Pegasus and MT5). Their Abstraction Ratio (AR) went way above 1.0.
- An AR of 1.0 means the student is rewriting exactly as much as the teacher.
- An AR of 3.74 (which Pegasus hit on CNN/DailyMail for unigrams) means the student is rewriting way more than the teacher.
- Crucially, the authors found that when the Reference is extractive (copy-heavy) and the AR > 1, it flags a high risk of hallucination. The model is making up facts that aren't in the book just to sound creative, going beyond what the human reference deemed necessary.

Why This Matters (And Why ROUGE Isn't Enough)

The paper argues that ROUGE is blind to this.

BART got a high ROUGE score (0.393) on CNN/DailyMail because it copied the text faithfully. It looked perfect to ROUGE.
Pegasus got a lower ROUGE score (0.197) because it rewrote things. ROUGE thought it was worse.
BUT, the new AR metric showed that on the extractive dataset, Pegasus was actually hallucinating (making things up) because its AR was 3.74 (for unigrams).

The paper suggests that without these new metrics, we might think the copycat (BART) is the best model, while the creative one (Pegasus) is actually dangerous because it's inventing facts.

What They Don't Claim

The authors are careful not to say they have "solved" the problem of fake news or perfect summarization.

They admit their metric only looks at surface-level words (n-grams), not deep meaning. If a student uses different words but means the exact same thing, the metric counts it as "rewriting," which is what they want for this specific test.
They say the choice of using the cubic power (3) was a design choice they tested and found worked well, not a law of physics.
They suggest these metrics are a screening tool. They flag the summaries that need a human to check them for lies. They don't automatically fix the lies; they just point the finger.

The Bottom Line

The paper introduces a new way to grade AI summaries that doesn't just check for copy-pasting. It uses a special math trick (cubing the non-overlapping parts) to spot when an AI is being too creative and making things up.

They proved that on 100 documents, these metrics could clearly tell the difference between a model that copies (SA ≈ 0.12–0.26) and one that writes (SA ≈ 1.15–1.99). They also showed that when a model's Abstraction Ratio shoots past 1.0 on datasets with copy-heavy references, it indicates the model is abstracting more than the human reference, signaling a potential hallucination risk that other scores miss.

It's like giving the teacher a new magnifying glass that reveals the tiny freckles of "made-up facts" that the old grading system completely missed.

Technical Summary: Abstractiveness Metrics for Evaluating Text Summarization

Problem Statement
Current evaluation paradigms for text summarization rely heavily on surface-level metrics like ROUGE and BERTScore, which measure the overlap between a generated summary and a human-written reference. While effective for assessing quality relative to a reference, these metrics fail to measure faithfulness (grounding in the source) or abstractiveness (the degree to which a summary diverges from extractive copying). A summary can achieve high ROUGE scores by faithfully reproducing a reference that itself contains hallucinated content, or by being overly extractive. Existing proxies for abstractiveness, such as the novel n-gram ratio, are binary at the token level and fail to account for document length asymmetry or provide graduated discrimination across different overlap levels. There is a lack of principled, quantitative measures that assess the relationship between generated text and the source document independent of the reference.

Methodology
The authors propose a set of three complementary, source-grounding metrics: Reference Abstraction (RA), Summary Abstraction (SA), and Abstraction Ratio (AR). These metrics are designed to quantify abstractiveness through a formulation that combines harmonic-mean length weighting with a cubic non-overlap amplification factor.

Formulation:
- Reference Abstraction (RA): Measures the inherent abstractiveness of the gold-standard reference. It is calculated as the harmonic mean of the reference length ( $R$ ) and source length ( $O$ ), modulated by the cube of the non-overlap fraction: $RA = \frac{2RO}{R+O} \times (1 - c_R)^3$ , where $c_R$ is the normalized overlap between the reference and source.
- Summary Abstraction (SA): Analogous to RA, this measures the abstractiveness of the generated summary ( $S$ ) relative to the source: $SA = \frac{2SO}{S+O} \times (1 - c_S)^3$ .
- Abstraction Ratio (AR): A dimensionless ratio comparing the abstractiveness of the generated summary to the reference. It isolates the non-overlap factors to ensure independence from absolute document lengths: $AR = \frac{(1 - c_S)^3}{(1 - c_R)^3 + \epsilon}$ .
Design Rationale:
- Harmonic Mean: Used to penalize extreme length asymmetry, ensuring that a very short summary against a long source yields a lower base score, reflecting higher abstractive demand.
- Cubic Amplification: The term $(1-c)^3$ provides non-linear sensitivity near the extractive boundary (where overlap $c \approx 1$ ). This amplifies small differences in overlap (e.g., between 0.95 and 0.98) to allow for sharp discrimination between extractive and abstractive behaviors.
- Dimensionless Ratio: AR is defined without the harmonic mean factor to ensure it is a pure comparison metric interpretable around a threshold of 1.0 (where $AR \approx 1$ indicates the generated summary matches the reference's level of abstraction).

Key Contributions

Principled Metrics: The introduction of RA, SA, and AR as heuristic metrics that measure divergence from extractive copying using source-grounding rather than reference-matching.
Continuous Grading: Unlike binary faithfulness detectors (e.g., FactCC), these metrics provide continuous scores with interpretable thresholds, allowing for the differentiation between mild paraphrasing and severe fabrication.
Length Normalization: The use of harmonic means and normalized overlap ratios addresses the limitations of prior work regarding document length asymmetry.
Screening Tool: The metrics serve as a screening mechanism to identify summaries that may require manual evaluation for potential hallucination, specifically flagging cases where the model abstracts significantly more than the human reference ($AR > 1$).

Experimental Results
The authors evaluated the metrics on 100 documents (50 from XSUM and 50 from CNN/DailyMail) across four summarization models: BART-large-cnn, Pegasus-xsum, DistilBart, and MT5-small.

Discrimination of Model Types: The metrics successfully distinguished between extractive-biased models (BART, DistilBart) and abstractive models (Pegasus, MT5).
- Extractive Models: Showed low Summary Abstraction ( $SA \approx 0.12–0.26$ ) and very low Abstraction Ratios ( $AR \approx 0$ ), indicating high overlap with the source and minimal abstraction.
- Abstractive Models: Showed higher SA values ( $SA \approx 0.96–1.77$ ).
Dataset Characterization: RA confirmed that XSUM references are substantially more abstractive ( $RA_1 = 3.55$ ) than CNN/DailyMail references ( $RA_1 = 0.61$ ).
Hallucination Detection:
- On the CNN/DailyMail dataset (which has extractive references), abstractive models like Pegasus and MT5 produced $AR > 1$, indicating they introduced novel content beyond what the reference warranted. For instance, Pegasus triggered a "High Hallucination Risk" classification in 44% of CNN/DM articles.
- On XSUM (abstractive references), all models had $AR < 1$, with Pegasus coming closest to the reference level ( $AR_2 = 0.92$ ).
Complementarity with ROUGE: The study demonstrated that ROUGE and AR capture independent dimensions. For example, on CNN/DM, BART achieved the highest ROUGE-1 score (0.393) with low hallucination risk ( $AR \approx 0$ ), while Pegasus had a lower ROUGE-1 (0.197) but high hallucination risk ($AR = 3.74$). This suggests that ROUGE alone cannot detect when a model is "hallucinating behind good quality scores."

Significance and Claims
The paper claims that RA, SA, and AR provide a necessary complement to existing evaluation frameworks by addressing the disconnect between summary quality (match to reference) and faithfulness (grounding in source). The authors position these metrics not as replacements for ROUGE or BERTScore, but as essential screening tools that:

Quantify the degree of abstraction in a principled, bounded manner.
Identify summaries where the model's abstraction level deviates significantly from the human reference, flagging potential hallucinations for manual review.
Offer a continuous scale that allows for the prioritization of manual evaluation (e.g., summaries with $AR > 2.0$ warrant immediate attention).

The authors note that these metrics serve as a precursor to more fine-grained hallucination decomposition, such as the Entity Hallucination Index (EHI), by first identifying that hallucination exists before analyzing its specific nature. The work emphasizes that high abstractiveness does not inherently imply poor quality, but rather that a mismatch between the model's abstraction level and the reference's abstraction level (indicated by extreme AR values) signals a need for further verification.