Technical Summary: Automatic Generation of Highlights for Academic Papers via Prompt-based Learning

Problem Statement

Academic literature is expanding rapidly in volume and diversity, creating significant challenges for researchers attempting to locate, read, and analyze relevant works efficiently. While "Highlights"—condensed summaries of critical findings—are recognized as an effective tool for quick comprehension and search engine optimization, many journals do not provide them. Existing solutions for automatic highlight generation rely on supervised learning methods that require large-scale, domain-specific training datasets. These traditional approaches face two primary limitations: they necessitate substantial computational resources to train multiple models for different domains, and they demand significant time and effort to construct large training corpora, often diverting focus from model optimization.

Methodology

This study proposes a Prompt-based Learning framework to generate academic paper highlights directly from abstracts, aiming to reduce dependency on large training datasets while maintaining high performance.

Data Collection

The researchers utilized three existing datasets covering distinct domains:

• CSPubSum: Computer Science.
• AIPubSum: Artificial Intelligence.
• BioPubSum: Biomedical.
  These datasets provided paper abstracts and corresponding author-written highlights. The authors crawled the original Elsevier links to retrieve the full text of abstracts and highlights, cleaning the data to remove empty records and formatting issues.

Prompt Design

The core of the methodology involves transforming the input (abstract) into a format suitable for pre-trained language models via prompt templates.

• Template Construction: Nine distinct prompt templates were designed, varying in phrasing (e.g., "Summarize," "Extract," "Refine") and constraints (e.g., specifying the number of highlights or requesting bullet points).
• In-Context Learning: For the ChatGPT model, the authors implemented an "In-context Learning" approach. This involved adding a "Demonstration" section to the prompt, containing $k$ examples of abstract-highlight pairs from the training set. The number of demonstrations was capped at six to accommodate the model's token limit (4096 tokens) while leaving space for the inference input.
• Sample Selection: To optimize the demonstration content, the study compared random sample selection against two relevance-based strategies: semantic similarity using text-embedding-ada-002 vectors and surface-level N-gram overlap.

Language Models

Five models were evaluated:

1. GPT-2: A pre-trained decoder-only model.
2. T5: A text-to-text transformer model.
3. Flan-T5: An instruction-tuned version of T5.
4. BART: A denoising autoencoder model.
5. ChatGPT (gpt-3.5-turbo-0613): A conversational model trained with Reinforcement Learning from Human Feedback (RLHF).

The local models (GPT-2, T5, Flan-T5, BART) were run using the Hugging Face Transformers library with random sampling for token generation. ChatGPT was accessed via API.

Key Contributions

1. Novel Application: This is the first study to apply prompt-based learning specifically to the task of academic highlight generation, utilizing both standard pre-trained models and large language models (LLMs) like ChatGPT.
2. Data Efficiency: The proposed method achieves performance comparable to state-of-the-art (SOTA) supervised models without requiring extensive training on domain-specific corpora. It effectively operates in zero-shot and few-shot scenarios.
3. Prompt Sensitivity Analysis: The study thoroughly investigates how prompt content (wording, constraints) and demonstration content (quantity and selection strategy) impact generation quality.
4. Demonstration Selection Strategy: The research identifies that selecting demonstration samples based on N-gram vocabulary overlap yields more stable and higher performance than random selection or deep semantic embedding similarity, particularly in low-shot settings.

Experimental Results

The models were evaluated using ROUGE-N and ROUGE-L metrics (specifically F1 scores) across the three datasets.

• Zero-Shot Performance: Without any training samples, the ChatGPT model coupled with optimized prompt templates achieved performance comparable to previous supervised learning methods. Notably, it surpassed the SOTA on the AIPubSum and BioPubSum datasets.
• Few-Shot Performance: When a small number of training samples (demonstrations) were added to the prompts, the ChatGPT model's performance significantly improved, surpassing SOTA approaches on two of the three datasets.
• Model Comparison:
  • ChatGPT and Flan-T5 showed the most dramatic performance gains when prompts were added, likely due to their instruction-tuning.
  • GPT-2 and BART showed little to no improvement with prompting; BART performed best in the "no prompt" condition among local models.
  • Prompt Impact: Prompt 8 ("Refine 4~5 innovative points... in bullet point format without additional explanation") consistently yielded the best results.
• Case Study: A qualitative analysis of specific papers revealed that the model-generated highlights had high recall, effectively covering the key points emphasized by human authors. However, precision was lower, with generated outputs occasionally containing irrelevant or redundant information.

Significance and Claims

The authors claim that this study offers a practical, widely applicable solution for automatically generating highlights for papers that currently lack them. By eliminating the need for training on domain-specific corpora, the method lowers the barrier for applying highlight generation across diverse academic fields.

The study emphasizes that while large language models possess powerful inherent capabilities, their performance on specific tasks is heavily dependent on the information provided in the prompt templates. The proposed method provides data support for subsequent text mining and bibliometric research by enabling the analysis of highlight information for a broader range of academic literature.

The authors maintain a modest tone regarding limitations, acknowledging that:

• Manual prompt engineering is time-consuming and relies on expert intuition.
• The generated content may still require post-processing (filtering) to remove redundancy.
• Deep analysis of prompt sensitivity at the model level is constrained by API access to closed-source models like ChatGPT.