💬 NLP

LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

The paper introduces LongSumEval, a unified framework that bridges evaluation and generation for long document summarization by using structured question-answering feedback to provide interpretable scores and actionable guidance, thereby significantly improving summary quality and alignment with human judgments without requiring model retraining.

Original authors: Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Haihua Chen, Junhua Ding

Published 2026-04-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huyen Nguyen, Haoxuan Zhang, Yang Zhang, Haihua Chen, Junhua Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, 50-page legal contract or a dense scientific report. You ask a smart AI to summarize it into a few paragraphs. The AI does its best, but how do you know if the summary is actually good?

This is the problem the paper LongSumEval tries to solve.

The Problem: The "Black Box" Grader

Currently, most computer programs that grade summaries are like a strict math teacher who only checks if the student used the exact same words as the textbook. If the student rephrases a sentence perfectly but uses different words, the computer might give them a bad grade.

Worse, these programs just give you a single number (like "75/100"). They don't tell you why you failed. Did you miss a key point? Did you invent a fact that wasn't there? It's like getting a "F" on a test with no comments, leaving you with no idea how to study for the next one.

The Solution: The "Quiz Master" Approach

The authors created a new system called LongSumEval. Instead of just counting words, they treat the summary like a student taking a quiz.

Here is how it works, using a simple analogy:

1. The Quiz (Evaluation)
Imagine the original long document is a textbook. The system first acts as a teacher and writes a list of important questions based on that textbook (e.g., "What was the main cause of the event?" or "Who was involved?").

Then, it asks the AI's summary to answer these questions.

Coverage Check: Can the summary answer the questions? If the summary misses the answer to "Who was involved," the system knows a key piece of information is missing.
Fact Check: If the summary does answer the question, does it match the truth in the original textbook? If the textbook says "The meeting was on Tuesday" and the summary says "Wednesday," the system catches this lie.

2. The Report Card (Structured Feedback)
Instead of just giving a score, this system generates a specific "to-do list" for the AI.

Bad Feedback: "Your summary is 60% good." (Useless)
LongSumEval Feedback: "You missed the answer to 'Who was involved,' and you got the date wrong. Here is the correct date from the original text."

3. The Study Session (Self-Refinement)
This is the magic part. The system takes that specific "to-do list" and gives it back to the AI as an instruction: "Hey, you missed this person and got the date wrong. Please rewrite your summary to fix these specific things."

The AI then rewrites the summary, fixing exactly what was wrong. It can do this over and over again, getting better each time, without needing to be retrained or taught new skills.

What They Found

The researchers tested this on seven different types of documents, ranging from short news articles to massive 27,000-word government reports and patent documents.

Better Grading: Their "Quiz Master" method agreed much more with human experts than the old word-counting methods. It was especially good at spotting when summaries of long documents were missing important details or making up facts.
Better Summaries: When they used the system's feedback to help the AI fix its own work, the summaries got significantly better. In some cases, the "missing information" score jumped by over 80%, and the "factual accuracy" improved by nearly 50%.
New Benchmark: They also created a new test set specifically for patent documents (legal documents about inventions) because existing tests didn't cover these well.

The Bottom Line

LongSumEval changes the game by turning evaluation into a conversation. Instead of just saying "You failed," it says "Here is exactly what you missed, here is the truth, and here is how to fix it." This allows AI to learn from its mistakes in real-time, creating summaries that are not just shorter, but actually accurate and complete.

1. Problem Statement

The paper addresses two critical bottlenecks in long document summarization:

Evaluation Limitations: Existing metrics (e.g., ROUGE, BLEU, BERTScore) rely on lexical overlap or semantic similarity, which correlate poorly with human judgments regarding factual consistency and coverage. They produce opaque aggregate scores without diagnostic feedback, failing to explain why a summary is poor or how to fix it. This is particularly problematic for long documents (>10,000 words) where information is distributed and hallucinations are common.
Generation-Evaluation Disconnect: Current self-refinement methods often lack structured, actionable feedback. They operate independently of evaluation metrics, meaning the generation process does not receive specific instructions on which facts are missing or incorrect, leading to ineffective iterative improvements.

2. Methodology: The LongSumEval Framework

The authors propose LongSumEval, a unified framework that bridges evaluation and generation using a Question-Answering (QA) paradigm. The framework consists of two interconnected modules:

A. QA-Based Evaluation Module

Instead of comparing text directly, this module operationalizes quality as the answerability and factual alignment of question-answer pairs.

Question Generation: An LLM generates diverse questions (What, When, Where, Who, How, Why) from the source document ( $Q_d$ ) and the generated summary ( $Q_s$ ).
Coverage Assessment: The system attempts to answer $Q_d$ $Q_{d}$ using only the summary.
- Metric: Ratio of answerable questions.
- Feedback: Unanswered questions are flagged as coverage gaps.
Factual Consistency Assessment: The system extracts facts from the summary as QA pairs ( $Q_s, A_s$ $Q_{s}, A_{s}$ ) and verifies them against the source document ( $A_d$ $A_{d}$ ).
- Metric: Similarity between $A_s$ and $A_d$ (using Exact/Partial Match, ROUGE-1 F1, or Cosine Similarity) above a threshold $\tau$ .
- Feedback: Inconsistent fact triplets $(q, A_s, A_d)$ are flagged as factual errors with ground-truth corrections.
Output: Quantitative scores ( $score_{cov}, score_{cons}$ ) and structured feedback ( $F_{cov}, F_{cons}$ ) identifying specific missing information and erroneous claims.

B. Self-Refinement Module

This module converts the structured feedback into natural language instructions to guide iterative improvement without retraining the model.

Process:
1. Generate an initial summary $S^{(0)}$ .
2. Evaluate to get feedback $F$ .
3. Convert $F$ into prompts (e.g., "Revise the summary to address these unanswered questions..." or "Correct these specific factual inconsistencies...").
4. Generate a refined summary $S^{(i+1)}$ based on the prompt.
5. Repeat until quality thresholds are met or max iterations are reached.

3. Key Contributions

Interpretable QA Evaluation Framework: A novel method producing fine-grained, structured feedback (unanswered questions and fact triplets) alongside quantitative scores, enabling transparent, reference-free assessment of long documents.
Feedback-Driven Self-Refinement: A mechanism that bridges the gap between assessment and generation by converting evaluation results into executable instructions for targeted revision.
Comprehensive Meta-Evaluation: Validation across seven human-annotated datasets spanning five domains (News, Scientific, Government, Social Media, Patents) and varying lengths (400–27,000 words).
PatentSumEval Benchmark: Introduction of a new human-annotated benchmark for patent summarization (30 patents, 180 summaries) with multi-dimensional quality annotations, addressing a gap in legal/technical summarization evaluation.

4. Experimental Results

The study utilized Kendall's $\tau_b$ to measure correlation with human judgments and tested refinement on five datasets.

Evaluation Performance (RQ1 & RQ2):
- LongSumEval demonstrated substantially stronger agreement with human judgments than existing QA-based baselines (QuestEval, QAEval, SummaQA).
- Consistency: Achieved a peak correlation of $\tau_b = 0.683$ on SummEval and $\tau_b = 0.800$ on PatentSumEval.
- Coverage: Achieved $\tau_b = 0.738$ on PatentSumEval.
- Robustness: Performed consistently well across short news articles and extremely long scientific/government documents, whereas baselines often failed on long inputs.
- Design Choices: ROUGE-1 F1 similarity and 5–7 questions for consistency evaluation were found to be optimal.
Self-Refinement Performance (RQ3):
- Targeted Improvement: The framework significantly improved low-quality summaries.
  - Coverage: Up to +83.72% improvement on low-coverage Patent summaries.
  - Consistency: Up to +47.42% improvement on low-consistency Patent summaries.
- General Improvement: Even on all summaries (not just low-quality ones), coverage improved by up to +39.71% (Patents) and +35.51% (PubMed).
- Trade-offs: Universal refinement sometimes slightly degraded consistency on long documents (e.g., Arxiv), suggesting a need for threshold-based triggering (refining only when scores drop below specific limits).
Human Validation:
- Generated questions were 91.67% focused on key information, and 98% were answerable from the source.
- Extracted answers showed 99% factual consistency with the source.

5. Significance and Implications

Paradigm Shift: The paper establishes a generalizable paradigm where evaluation feedback serves as executable instructions for generation. This moves beyond "black-box" scoring to actionable, transparent quality control.
High-Stakes Applications: By ensuring verifiable accuracy and providing specific corrections, the framework is directly applicable to critical domains like biomedical literature, legal analysis, and clinical decision support, where hallucinations are unacceptable.
Efficiency: The approach enables significant quality improvements through self-refinement without model fine-tuning, making it cost-effective and adaptable to new domains.
Reproducibility: The release of the PatentSumEval dataset and all code/prompts facilitates future research in controllable text generation and long-document understanding.

In conclusion, LongSumEval solves the "evaluation-generation disconnect" by using structured QA feedback to not only diagnose errors but also to explicitly guide the LLM in fixing them, resulting in summaries that are both more comprehensive and factually accurate.