Technical Summary: A Human-in-the-Loop Corpus for LLM-Based Simplification of Scientific Summaries

Problem Statement

Interdisciplinary research is accelerating, yet scientific papers remain inaccessible to researchers outside their specific fields due to domain-specific terminology, compressed arguments, and field-specific conventions. While Large Language Models (LLMs) offer a promising mechanism for simplifying specialized texts, existing approaches face significant challenges:

• Quality Risks: LLMs may introduce hallucinations, lose nuance, or alter the strength of scientific claims.
• Benchmark Limitations: Current evaluation benchmarks primarily rely on news articles or Wikipedia passages, lacking coverage of scientific discourse and cross-disciplinary audiences.
• The Readability-Precision Trade-off: Fully automated systems struggle to balance improving surface readability with preserving scientific precision and domain fidelity.

There is a critical need for suitable datasets and evaluation protocols that address scientific text simplification for cross-disciplinary communication.

Methodology

The authors propose a Human-in-the-Loop (HITL) framework designed to operationalize accessibility for STEM readers without expertise in the source domain while maintaining technical accuracy. The methodology utilizes SciSummNet (a corpus of 1,000 highly cited ACL papers with expert-written summaries) as the source material and proceeds through three main stages:

1. Baseline Generation: Initial simplifications are generated using GPT-4o-mini in a zero-shot setting. The model is prompted to simplify text for non-experts, identify complex phrases, and retain the original paragraph structure.
2. Phase 1: Cross-Disciplinary Reader Evaluation:
   • Participants: STEM readers fluent in English but outside the computer science domain.
   • Task: Participants compare original summaries against GPT-simplified versions. They perform three tasks:
     • Select sentences/phrases they find difficult to understand.
     • Provide comparative judgments on Understanding, Naturalness, and Simplicity (preferring Original, GPT, or No Difference).
     • Highlight specific complex words or phrases.
   • Outcome: 92 comparative judgments per dimension and detailed difficulty annotations.
3. Phase 2: Expert-Edited Reference Simplifications:
   • Participants: Computer science graduate students or researchers.
   • Task: Experts create "reference simplifications" for 47 items using the original summary, the GPT output, and the Phase 1 feedback (highlights of difficult terms).
   • Guidelines: Editors follow four scenarios (e.g., refining merged content, verifying GPT edits, correcting over-simplification) with a focus on retaining necessary technical terminology, preserving claim strength, and improving sentence structure.

Key Contributions

The paper presents three primary contributions:

1. A Human-in-the-Loop Methodology: A framework that centers comprehension by cross-disciplinary STEM readers while maintaining technical accuracy through expert post-editing.
2. A Curated Corpus: A new resource derived from SciSummNet containing:
   • Baseline LLM outputs.
   • Sentence- and phrase-level difficulty annotations from non-specialist readers.
   • Expert-edited reference simplifications.
   • Human judgments and automatic evaluation results.
3. An Evaluation Analysis: An assessment demonstrating where LLMs improve surface readability and where expert intervention is crucial for claim calibration and terminology fidelity.

Results

Phase 1: Reader Perceptions

• Preference: Cross-disciplinary readers showed a clear preference for GPT-generated summaries regarding Understanding (73/92 judgments) and Simplicity (70/92 judgments).
• Naturalness: Judgments were more balanced; 42/92 indicated no difference, while 14/92 preferred the original. This suggests LLM outputs occasionally suffer from stylistic inconsistencies or overly informal phrasing.
• Difficulty Annotations: Readers most frequently flagged multi-noun compounds, unexplained abbreviations, and domain-specific terminology.
• Qualitative Tensions: Two recurring issues were identified:
  1. Precision vs. Approachability: Replacing terms like "latent variables" or "NP-hard" with generic paraphrases improved readability but reduced technical precision.
  2. Style: LLMs improved local clarity but sometimes introduced awkward formulations, whereas originals retained conventional academic style.

Phase 2: Expert-Edited vs. LLM Outputs

Automatic metrics were applied to 47 items comparing Original vs. GPT and Original vs. Expert-Edited:

• Source Similarity: GPT outputs retained higher lexical overlap with the source (BLEU: 0.949 vs. 0.349; BERTScore: 0.9907 vs. 0.9031) compared to expert edits.
• Simplification Operations: Reference-free SARI scores were higher for GPT (46.76) than expert edits (33.72), indicating more aggressive edit operations by the LLM.
• Readability: GPT summaries achieved better Flesch-Kincaid scores (Grade Level: 12.10 vs. 13.00; Reading Ease: 48.54 vs. 40.89).
• Qualitative Findings: Expert edits followed a hybrid strategy, preserving domain-critical terminology (e.g., "computational lexicon") while adopting clearer LLM-generated phrasing for general explanations.

Crucial Observation: While automatic metrics favored GPT outputs due to their proximity to the source and simpler surface features, qualitative analysis revealed that expert post-editing was necessary to preserve domain terminology and scientific claim calibration. The authors note that higher similarity to the source does not necessarily imply better accessibility or meaning preservation.

Significance and Claims

The paper claims to address the gap in scientific text simplification by providing a resource that links original summaries, LLM simplifications, reader annotations, and expert references. Its significance lies in:

• Operationalizing Accessibility: Demonstrating a workflow that makes scientific summaries accessible to non-specialists without sacrificing scientific meaning.
• Revealing Trade-offs: Highlighting that while LLMs effectively improve surface readability, they often require expert oversight to maintain terminological accuracy and appropriate claim strength.
• Benchmarking Utility: Providing a reproducible resource for training and evaluating models that aim to simplify scientific texts without compromising factual fidelity or terminology.

The authors position LLM-generated simplifications as useful initial drafts that benefit significantly from cross-disciplinary reader feedback and domain-informed expert review, rather than as directly publishable scientific texts.

Limitations

The authors explicitly note several limitations:

• Scale: Only 47 of 1,000 summaries received expert editing.
• Sampling: Phase 1 participants were limited to STEM non-CS fields; Phase 2 editors were CS researchers. Other audiences (humanities, policy) were not included.
• Metrics: Standard metrics (BLEU, SARI, etc.) do not fully capture semantic nuance or rhetorical clarity.
• Scope: The study is limited to English-language computer science texts.
• Model Dependence: Results are specific to GPT-4o-mini and a single zero-shot prompt.
• Ethical Risks: Simplification carries a risk of distorting scientific claims, which expert validation mitigates but does not eliminate.