Technical Summary: Less is More: Quality-Aware Training Data Selection for Scientific Summarization

Problem Statement

Scientific long-document summarization faces two primary challenges. First, existing datasets often treat author-written abstracts as "gold" reference summaries without verifying their alignment with the full source text. Prior research indicates that abstracts can diverge from full articles through omissions, inconsistencies, selective reporting, or overly strong interpretations, leading to unreliable supervision signals and evaluation targets that may not reflect source-grounded factuality. Second, publicly available scientific summarization datasets, such as the widely used PubMed benchmark, are limited in scale and lack the hierarchical document structure required for modern long-context Large Language Models (LLMs). These limitations compromise the reliability of supervision signals and the adequacy of current benchmarks for contemporary summarization systems.

Methodology

Dataset Construction: PMC-Large

The authors constructed and released PMC-Large, a large-scale dataset derived from PubMed Central (PMC) Open Access articles.

• Scale: The final corpus contains 1.88 million article-abstract pairs.
• Structure: Unlike previous datasets that use plain text, PMC-Large serializes article bodies in a Markdown-style format. This preserves section hierarchy (using heading markers) and structural cues directly in the model input, which is critical for long-context LLMs.
• Normalization: Non-narrative content (e.g., references, acknowledgements, funding statements) is filtered out using a three-step section filter based on XML tags, JATS attributes, and regex patterns. Non-textual elements (figures, tables, formulas, code) are replaced with explicit placeholders (e.g., [FIG_REF], [MATH], [CODE]) to maintain structural context without tokenizing irrelevant content.
• Filtering: The corpus is restricted to English articles with permissive licenses (CC BY, CC0). Examples with target leakage (where the abstract appears in the body) are removed. Length filtering retains articles between the 10th and 80th percentiles of token distribution to avoid outliers.

Quality Analysis

To assess the reliability of author-written abstracts, the authors analyzed a 10,000-article subset using four source-grounded, model-based metrics rather than traditional reference-based metrics (like ROUGE):

1. AlignScore: Measures alignment between summary and source.
2. FineSurE: Detects sentence-level factual errors.
3. SummaC (ZS): An NLI-based metric using a biomedical backbone (PubMedBERT-MNLI-MedNLI) to check if summary claims are entailed by the source.
4. G-Eval: An LLM-based judge evaluating factual consistency, coherence, and relevance.

The analysis revealed that while many abstracts score highly, there is significant variance in quality. The metrics showed only partial agreement (e.g., SummaC had weak correlation with others), indicating that no single metric fully captures reference quality.

Quality-Aware Data Selection Experiments

The authors conducted Supervised Fine-Tuning (SFT) experiments using Qwen2.5-3B as the base model to test if selecting high-quality training examples improves performance.

• Setup: Models were trained on subsets of varying sizes (1K, 2K, 5K, 10K) selected based on metric scores (AlignScore, FineSurE, SummaC, G-Eval, or an aggregate of all).
• Baselines: Performance was compared against random sampling of the same size and training on the full candidate pool (5K or 100K).
• Evaluation: Models were evaluated on a held-out 500-example test set using the same four source-grounded metrics.

Key Contributions

1. PMC-Large Dataset: Release of a 1.88M article-abstract dataset specifically formatted for long-context LLMs with preserved hierarchical structure and normalized non-textual elements.
2. Reference Quality Analysis: A large-scale analysis demonstrating that author-written abstracts vary significantly in their alignment with source articles, challenging the assumption of uniform reliability.
3. Quality-Aware Selection Strategy: Empirical evidence that training on subsets selected by source-grounded quality metrics outperforms random sampling at matched sizes. Furthermore, smaller, high-quality selected subsets can match or exceed the performance of larger, unfiltered random subsets on factuality-oriented metrics.

Results

• Metric Variance: In the 10,000-article analysis, abstracts showed a wide distribution of quality scores. For instance, AlignScore had a median of 0.626 but a bottom decile below 0.473.
• Selection Efficiency:
  • Small Scale (5K Pool): A model trained on a 1K subset selected by an aggregate metric (averaging percentile ranks of all four metrics) outperformed a model trained on a random 1K subset across all metrics. Notably, this selected 1K model matched or exceeded the performance of a model trained on the full 5K pool on most metrics.
  • Large Scale (100K Pool): When scaling the candidate pool to 100K and using AlignScore for selection, the selected subsets consistently outperformed random subsets of the same size. A 5K AlignScore-selected model achieved a higher overall score than the model trained on the full 100K pool.
  • Two-Stage Selection: A two-stage approach (filtering 100K down to 10K via AlignScore, then re-ranking with multiple metrics) yielded the best overall results. The Aggregate-All 5K model achieved the highest scores across the full table for Overall, SummaC, and FineSurE.
• Metric Specificity: Different selection metrics favored different outcomes (e.g., AlignScore selection maximized AlignScore and FineSurE, while G-Eval selection maximized G-Eval), supporting the use of aggregate signals for balanced performance.

Significance and Claims

The paper claims that reference quality is a critical factor in scientific summarization. The authors argue that author-written abstracts are useful supervision targets but are not uniformly reliable. Consequently, quality-aware data selection serves as a simple yet effective mechanism to improve training efficiency and factuality-oriented performance.

The findings suggest that "less is more" in the context of training data: selecting a smaller subset of high-quality examples can yield better or comparable results to training on significantly larger, unfiltered datasets. This challenges the prevailing assumption that more data is always better, proposing instead that the quality of the supervision signal, measured by source-grounded alignment, is a more decisive factor for model performance in scientific summarization.

Limitations Acknowledged by the Authors:

• Reliance on automatic metrics which may not capture all aspects of quality and were not specifically designed for biomedical domains.
• Experiments conducted on subsets of the full 1.88M corpus due to computational constraints.
• Results based on a single base model (Qwen2.5-3B) and training setup, limiting generalizability to other architectures.
• Potential bias propagation from the original author-written abstracts to the trained models.
• The dataset is limited to English PMC Open Access articles.