PlainQAFact: Retrieval-augmented Factual Consistency Evaluation Metric for Biomedical Plain Language Summarization

1. Problem Statement

The paper addresses the critical challenge of factual consistency evaluation in Biomedical Plain Language Summarization (PLS). While Large Language Models (LLMs) have advanced the ability to simplify complex medical texts for lay audiences, they frequently produce hallucinations.

A specific and under-addressed issue in PLS is the "elaborative explanation phenomenon." To make scientific abstracts understandable to non-experts, summaries often include external content (definitions, background context, examples) that is not explicitly present in the source text.

• Limitation of Existing Metrics: Current automatic evaluation methods (e.g., entailment-based, similarity-based, or standard QA-based metrics) rely heavily on the source document for verification. They fail to evaluate the factual accuracy of these "added" elaborative explanations because the source text does not contain the necessary information to verify them.
• Data Gap: Existing benchmarks often rely on LLM-generated summaries or rule-based perturbations, lacking high-quality, human-annotated ground truth that distinguishes between source simplifications and external elaborations.

2. Methodology

The authors propose a two-pronged solution: a new benchmark dataset and a novel evaluation metric.

A. The PlainFact Benchmark

To enable rigorous evaluation, the authors constructed PlainFact, a fine-grained, expert-annotated dataset:

• Source: Derived from the CELLS dataset (human-authored plain language summaries by original authors) and the Cochrane Database.
• Annotation: Medical experts annotated 400 summary-abstract pairs (200 factual, 200 non-factual via GPT-4o perturbation).
• Granularity: Each sentence is labeled with:
  1. Factuality Type: Source Simplification (derived from abstract) vs. Elaborative Explanation (introduces new external content).
  2. Functional Role: Background, Definition, Example, Method/Result, etc.
  3. Alignment: Mapping to specific sentences in the source abstract.
• Statistics: Approximately 44% of sentences in the dataset are elaborative explanations, highlighting the prevalence of this phenomenon.

B. The PlainQAFact Framework

PlainQAFact is a retrieval-augmented, sentence-aware, QA-based evaluation metric designed to handle both simplifications and elaborations. It operates in a dual-stage pipeline:

1. Sentence Classification:

   • A fine-tuned PubMedBERT classifier first identifies whether a sentence is a Source Simplification or an Elaborative Explanation.
   • This step is crucial to determine the verification strategy.

2. Retrieval-Augmented QA Scoring:

   • For Source Simplifications: The metric generates questions based on the summary and extracts answers strictly from the source abstract.
   • For Elaborative Explanations: The metric retrieves external domain knowledge (from StatPearls and Medical Textbooks) using MedCPT (a retriever pre-trained on PubMed logs) to verify the added content.
   • QA Pipeline:
     • Answer Extraction (AE): Extracts key entities from the summary (using Llama 3.1 or GPT-4o).
     • Question Generation (QG): Generates questions based on extracted answers (using fine-tuned BART-large).
     • Question Filtering (QF): Removes unanswerable questions.
     • Question Answering (QA): An extractive QA model (Electra-large) answers questions using the Source Abstract (for simplifications) or Source + Retrieved Knowledge (for explanations).
     • Answer Overlap Evaluation (AOE): Compares the model's answer with the gold answer using BERTScore (preferred over LERC for semantic flexibility).

3. Final Scoring: The final score is a weighted average of the scores for simplification sentences and explanation sentences.

3. Key Contributions

1. PlainFact Dataset: The first expert-annotated benchmark specifically designed to distinguish between source-grounded simplifications and external elaborative explanations in biomedical PLS.
2. PlainQAFact Metric: A novel, retrieval-augmented evaluation metric that selectively applies external knowledge retrieval only when necessary (i.e., for elaborative explanations), balancing accuracy with computational efficiency.
3. Comprehensive Analysis: Extensive ablation studies and error analyses demonstrating that existing metrics fail on elaborative content, while PlainQAFact significantly outperforms them.

4. Experimental Results

The authors evaluated PlainQAFact against five state-of-the-art metrics (DAE, AlignScore, SummaC, QAFactEval, QuestEval) and two LLM-based judges (Llama 3.1, GPT-4o) across three datasets: PlainFact, CELLS, and FareBio.

• Overall Performance: PlainQAFact achieved the highest correlation with human ratings on PlainFact (Pearson = 81.3) and CELLS (Pearson = 75.2), outperforming all other automatic metrics.
• Elaborative Explanations: In an "explanation-only" evaluation setting, PlainQAFact significantly outperformed baselines (AUC-ROC = 83.3 vs. lower scores for others), proving its ability to verify content not found in the source.
• Robustness: The metric maintained high performance even when tested on datasets with heuristic entity swaps (non-LLM perturbations), demonstrating robustness against specific factual errors.
• Comparison with GPT-4o: While GPT-4o showed competitive performance, it exhibited high variance (instability) and higher computational costs. PlainQAFact, built on open-source models (Llama 3.1), offers a more transparent and reproducible alternative.
• Ablation Insights:
  • Removing the classifier and retrieving for all sentences degraded performance, confirming the value of selective retrieval.
  • Combining the source abstract with external knowledge bases (Textbooks + StatPearls) yielded the best results for elaborative content.
  • Sentence-level evaluation significantly outperformed summary-level evaluation, reducing noise from unrelated context.

5. Significance and Future Impact

• Safety in Healthcare: By accurately detecting hallucinations in "added" medical explanations, PlainQAFact helps ensure that plain language health information is safe for lay audiences to use for decision-making.
• Paradigm Shift: The work moves beyond simple source-text verification, acknowledging that high-quality summarization often requires external knowledge. It establishes a new standard for evaluating "explainability" and "factual consistency" simultaneously.
• Reproducibility: The authors provide the dataset, annotation protocols, and the metric code, enabling the community to develop more reliable biomedical summarization systems.
• Limitations: The authors acknowledge challenges with numerical value extraction and the subjective nature of distinguishing simplifications from explanations, suggesting future work in domain-specific classifiers and improved numerical handling.

In conclusion, PlainQAFact represents a significant advancement in the evaluation of biomedical AI, providing a specialized tool to mitigate the risks of hallucinations in plain language medical communication.