💬 NLP

Stress Testing Factual Consistency Metrics for Long-Document Summarization

This paper systematically evaluates six reference-free factual consistency metrics on long-document summarization, revealing their significant limitations in handling long-range dependencies and information-dense claims through a series of factuality-preserving perturbations, while proposing specific directions for future improvements.

Original authors: Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very long, complicated book—maybe a legal contract, a scientific paper, or a fantasy novel. You ask a smart computer (an AI) to write a short summary of it. The summary sounds smooth and professional, but did the computer actually tell the truth, or did it make things up?

This paper is like a "stress test" for the tools we use to check if those summaries are true. The authors, Zain Muhammad Mujahid, Dustin Wright, and Isabelle Augenstein, wanted to see if the current "truth detectors" work well when the source material is huge and complex.

Here is the breakdown of their findings using simple analogies:

The Problem: The "Short-Story" Detectors

Currently, we have several automatic tools (metrics) designed to check if a summary is factual. Think of these tools as security guards at a museum.

The Issue: These guards were trained on small, simple paintings (short documents).
The Challenge: Now, we are asking them to guard a massive, sprawling cathedral with thousands of stained-glass windows (long documents). The authors suspected these guards might get confused, miss details, or panic when the building is too big.

The Experiment: The "Shape-Shifting" Test

To test these guards, the researchers didn't just look at the summaries; they played tricks on them. They took a summary that was 100% true and made small, harmless changes to it, like a magician changing a card's color without changing its value.

They used seven different tricks:

Paraphrasing: Rewriting sentences in different words.
Simplifying: Making complex sentences easier to read.
Synonym Swap: Replacing words with their synonyms (e.g., "big" to "huge").
Negation: Flipping the logic (e.g., "It is not the case that he didn't go" instead of "He went").
Compression: Making the summary even shorter.
Vocabulary Reduction: Using fewer, simpler words.
Adding Noise: Inserting a random true sentence from the original book that doesn't fit the summary's main point.

The Goal: If a truth detector is good, it should give the same score to the original summary and the tricked version, because the facts haven't changed. If the score jumps up and down wildly, the detector is unreliable.

The Results: The Guards Are Stumbling

The researchers tested six popular truth detectors across three types of long documents: Science Fiction (stories), Legal (court rulings), and Scientific (research papers).

Here is what they found:

1. The "Surface-Level" Trap
Most of the detectors are easily fooled by surface changes.

Analogy: Imagine a guard who only checks if a person is wearing a red hat. If you swap the red hat for a blue one (even if it's the same person), the guard says, "Not allowed!"
Finding: When the researchers changed the wording or simplified the sentences, the detectors gave wildly different scores. Some detectors hated legal language; others struggled with scientific jargon. They were reacting to how the words looked, not what they meant.

2. The "Needle in a Haystack" Problem
Long documents have information scattered everywhere.

Analogy: If you are looking for a specific fact in a 500-page book, a short summary might pull that fact from page 10, page 200, and page 400.
Finding: The detectors struggled when a summary sentence relied on information from many different parts of the book. They got confused when the evidence was "entangled" or spread out. They worked better when the evidence was right next to each other.

3. The "Context Window" Issue
To check a summary sentence, the detectors need to look at the original text.

Analogy: Imagine trying to understand a joke by reading only the punchline. You need the setup (the context) to get it.
Finding: When the researchers gave the detectors a wider "view" of the original text (more context), some detectors got better at their job. However, even with more context, none of them were perfect. Some detectors (like SummaC) seemed to ignore the extra context entirely, sticking to their narrow view.

4. The "Legal" vs. "Story" Difference

Legal Text: The detectors were most unstable here. Legal language is precise and logical. Changing a single word or structure (like a negation) threw the detectors into a panic.
Science Fiction: The detectors were slightly more stable but still inconsistent.
Scientific Papers: Interestingly, when the summary came from multiple documents, the detectors were more stable. It seems having redundant information (the same fact repeated in different papers) helped the detectors feel more confident.

The Conclusion: We Need Better Tools

The paper concludes that the current "truth detectors" are brittle. They are like a scale that gives you a different weight every time you step on it, even if you haven't moved.

They fail when the summary is rephrased.
They fail when the facts are spread out across a long document.
They fail when the logic gets complex (like double negatives).

The Takeaway: We cannot trust these tools to automatically verify long summaries yet. To fix this, we need new tools that can:

Reason across the whole book (not just look at one sentence at a time).
Understand the meaning regardless of how the words are arranged.
Handle the "messiness" of long, complex documents without getting confused.

The authors have released their code and data so others can try to build these better, more robust detectors.

1. Problem Statement

Evaluating the factual consistency (faithfulness) of abstractive text summaries is a critical challenge, particularly for long documents. While Large Language Models (LLMs) have improved fluency, they frequently hallucinate or omit facts.

Limitations of Current Metrics: Traditional metrics (e.g., ROUGE, BLEU) rely on n-gram overlap and fail to capture factual correctness. Reference-free metrics (e.g., FactCC, SummaC, AlignScore) have shown promise on short documents but struggle with long-form inputs due to:
- Input Length Constraints: Many models cannot process thousands of tokens simultaneously.
- Dispersed Evidence: Facts in long documents are often scattered across distant sections or multiple documents, creating long-range dependencies.
- Brittleness: Existing metrics are often sensitive to superficial lexical changes (paraphrasing, synonym replacement) rather than semantic meaning, leading to inconsistent scores for semantically equivalent summaries.
Gap: There is a lack of systematic benchmarking of these metrics under long-context conditions, specifically regarding their robustness to meaning-preserving perturbations and their sensitivity to information density.

2. Methodology

The authors propose a stress-testing framework to evaluate six widely used, reference-free factuality metrics across three diverse long-document datasets.

A. Datasets

The study utilizes three benchmarks spanning different domains:

SQuALITY: Science fiction stories (narrative domain).
LexAbSumm: European Court of Human Rights judgments (legal domain).
ScholarQABench: Multi-document computer science papers (scientific domain).

B. Metrics Evaluated

Six reference-free metrics representing different architectures were tested:

BARTScore: Generation-based (likelihood of summary given source).
SummaC-Conv & SummaC-ZS: Entailment-based (NLI models).
AlignScore: Contrastive alignment learning.
UniEval: Multi-dimensional QA-based evaluation.
MiniCheck: Lightweight sentence-level classifier.

C. Stress-Testing Strategy (Perturbations)

To test robustness, the authors applied seven meaning-preserving perturbations to the original summaries using GPT-4o. A robust metric should assign similar scores to the original and perturbed summaries. The perturbations include:

Paraphrased: Rewriting with alternate phrasing.
Simplified: Reducing complex sentence structures.
Synonym Replaced: Swapping content words with synonyms.
Less Diverse: Reducing vocabulary variety.
Negated: Introducing logically equivalent negations (e.g., "not unimportant" $\to$ "important").
Summarized: Further compressing the text.
Added Source Text: Inserting a factual sentence from the source unrelated to the main summary content.

D. Retrieval-Based Scoring Framework

Since most metrics cannot ingest full long documents, the authors adopted a retrieval-augmented strategy:

For each summary sentence, retrieve the top- $K$ most similar source sentences using SBERT embeddings.
Expand these retrieved sentences into a context window of size $w$ (surrounding sentences).
Compute the factuality score for the summary sentence against the retrieved context snippet.
Aggregate sentence-level scores to produce a global summary score.
The authors varied the window size $w$ to test sensitivity to retrieval granularity.

E. Information Density Analysis

The authors analyzed the relationship between claim information density (measured by the mean cosine similarity between a summary sentence and all source sentences) and metric reliability. High similarity implies the claim is "entangled" with the source (hard to verify), while low similarity implies specific, localized claims.

3. Key Results

A. Inconsistency Under Perturbation

General Instability: All six metrics exhibited significant score fluctuations when faced with meaning-preserving edits.
Domain Sensitivity:
- Legal (LexAbSumm): Metrics were highly unstable. BARTScore showed consistent negative shifts across almost all perturbations. AlignScore struggled with logical consistency in structured legal texts.
- Negation Sensitivity: Most metrics (especially UniEval and MiniCheck) failed to handle logically equivalent negations, often treating them as factual contradictions.
Robustness Hierarchy: MiniCheck and UniEval were generally the most robust to surface changes, though they still struggled with negations. AlignScore and SummaC-ZS were the least reliable across domains.

B. Impact of Retrieval Context

Context Window Size ( $w$ ): Increasing the retrieval context window generally improved scores for most metrics, particularly in the legal domain where arguments span multiple sentences.
NLI Limitations: NLI-based metrics (SummaC) showed little sensitivity to increased context windows, suggesting they rely on localized comparisons rather than broader discourse understanding.

C. Sensitivity to Information Density

The "Hubness" Problem: For LexAbSumm and SQuALITY, metrics assigned lower scores to claims with high semantic overlap (high information density) with the source. This suggests metrics struggle when evidence is dispersed or when a claim is a compressed generalization of the text.
Multi-Document Exception: In ScholarQABench (multi-document), high-similarity claims sometimes received higher scores, likely due to redundancy across documents reinforcing the claim.
Metric Behavior: AlignScore and BARTScore showed sharp declines for high-similarity claims, indicating a reliance on local lexical alignment rather than multi-span reasoning.

4. Key Contributions

Systematic Benchmarking: The first comprehensive evaluation of six popular factuality metrics specifically tailored for long-document and multi-document summarization settings.
Stress-Testing Framework: Introduction of a rigorous perturbation methodology (7 types) adapted for long-form contexts to expose metric brittleness.
Retrieval-Aware Analysis: Demonstration that standard metrics behave inconsistently when integrated into retrieval-based scoring frameworks, particularly regarding context window size.
Information Density Insight: Identification of a failure mode where metrics degrade on "information-dense" claims that are semantically entangled with large portions of the source text.

5. Significance and Future Directions

Reliability Warning: The study reveals that current state-of-the-art factuality metrics are not reliable for long-form summarization. They often react to surface-level stylistic changes rather than factual truth, leading to misleading evaluations in high-stakes domains like law and medicine.
Design Guidelines: The authors propose that future metrics must move beyond local pairwise alignment. They should incorporate:
- Multi-span Reasoning: The ability to link evidence across distant parts of a document.
- Context-Aware Calibration: Adjusting scores based on retrieval context size and information density.
- Robust Training: Training on meaning-preserving variations to improve stability against paraphrasing and negation.
Limitations: The study relies on automatically generated perturbations (GPT-4o) rather than human-verified edits, and does not include human factuality judgments for the long-document setting due to the scarcity of such annotated data.

In conclusion, the paper argues that without addressing these robustness and context-awareness issues, automated evaluation of long-document summarization remains fundamentally flawed, potentially masking hallucinations in critical applications.