Enhancing Unsupervised Keyword Extraction in Academic Papers through Integrating Highlights with Abstract

1. Problem Statement

Automatic keyword extraction is a critical task in Natural Language Processing (NLP) and information retrieval, aiding in the construction of knowledge graphs and the identification of research trends. While unsupervised methods are widely used due to their generalizability, their performance is often constrained by the quality and richness of the input text.

• Current Limitation: Most existing unsupervised approaches rely solely on the Abstract or the full text. While abstracts are concise, they often contain background noise or lack specific keywords that authors explicitly highlight in other sections.
• The Gap: Academic papers increasingly include a "Highlights" section (typically 3–5 bullet points provided by authors) to summarize key innovations and contributions. However, the potential of using these highlights to enhance unsupervised keyword extraction has not been systematically explored.

2. Methodology

The study proposes a framework to integrate Highlights with Abstracts to improve the performance of unsupervised keyword extraction models.

A. Dataset Construction

• Source: Papers were collected from the Elsevier platform (Scopus database) across two domains: Library and Information Science (LIS) and Computer Science (CS).
• Scale: The final dataset comprises 2,590 LIS papers and 2,996 CS papers (published 2012–2023).
• Preprocessing: Papers were filtered to ensure they contained author-assigned keywords (3–6 keywords) and author-provided highlights. A web crawler was used to extract highlights from online versions.

B. Input Integration Strategies

The study evaluated seven distinct input configurations to test how combining text sources affects performance:

1. A (Abstract Only): Baseline.
2. H (Highlights Only): Baseline.
3. A+H: Direct concatenation of Abstract + Highlights.
4. H+A: Direct concatenation of Highlights + Abstract (testing positional bias).
5. FA (Filtered Abstract): Abstract sentences filtered based on semantic similarity to Highlights (retaining top-k sentences).
6. FA+H: Filtered Abstract + Highlights.
7. H+FA: Highlights + Filtered Abstract.

Filtering Mechanism: Sentences in the abstract were ranked by cosine similarity to the highlights vector (using all-MiniLM-L6-v2). The top-k sentences (k=4) were retained to reduce noise.

C. Extraction Models

Four unsupervised models were employed to evaluate the inputs:

• Graph-based:
  • TextRank: Uses co-occurrence graphs and PageRank.
  • PositionRank: Incorporates word position and frequency into the graph ranking.
• Embedding-based (Pre-trained Language Models):
  • MDERank: Uses masked document embeddings to calculate semantic similarity.
  • PromptRank: Uses prompt learning with encoder-decoder architectures to generate keyword sequences.
• Large Language Models (LLMs): Experiments were also conducted with GPT-4o and LLaMA-3 to test generalizability.

D. Evaluation

• Metrics: F1 scores at top-k (F1@5, F1@10, F1@15) against gold-standard author keywords.
• Statistical Analysis: Paired t-tests were used to determine the significance of performance improvements.

3. Key Contributions

1. Novel Integration: First study to systematically explore integrating Highlights with Abstracts for unsupervised keyword extraction.
2. Empirical Validation of Concatenation: Demonstrated that simple text concatenation (A+H or H+A) significantly outperforms single-source inputs, with the order of concatenation having minimal impact on most models.
3. Semantic Filtering Insight: Validated that filtering abstract sentences based on semantic similarity to highlights can retain core information, though simple concatenation proved more robust in practice.
4. Cross-Domain Analysis: Provided a comparative analysis of LIS vs. CS domains, revealing that LIS papers show higher extraction performance, likely due to differences in how keywords are derived (direct extraction vs. abstract summarization).
5. Mechanism Analysis: Quantified the keyword coverage differences, showing that Highlights often contain unique keywords absent from the Abstract, thereby raising the theoretical upper limit of extraction performance.

4. Key Results

• Performance Improvement: Integrating Highlights with Abstracts consistently improved extraction performance across all models and datasets.
  • Example (LIS, MDERank): F1@5 increased from 19.48% (Abstract only) to 22.32% (A+H).
  • Example (CS, PromptRank): F1@5 increased from 16.44% to 16.91%.
• Model Superiority: Embedding-based models (MDERank, PromptRank) and LLMs significantly outperformed traditional graph-based models (TextRank, PositionRank).
• LLM Performance: LLMs (LLaMA-3, GPT-4o) achieved the highest absolute scores. Notably, LLaMA-3 showed significant gains with fused inputs, while GPT-4o showed mixed results on the CS dataset, suggesting sensitivity to prompt design and model architecture.
• Sentence Length Analysis: Retaining only 4–5 sentences from the abstract (filtered by similarity to highlights) was often sufficient to achieve performance comparable to using the full abstract, indicating that much of the abstract is noise for keyword extraction.
• Multidisciplinary Robustness: The method was validated on a multidisciplinary corpus (540 papers across 27 fields) and a non-Elsevier journal (Environmental Processes), confirming the generalizability of the approach.

5. Significance and Implications

• Scientometric Impact: The study highlights the value of Highlights as a distinct structural element in academic papers. It suggests that bibliometric analyses and knowledge graph construction can be significantly improved by incorporating this data source.
• Theoretical Insight: The results support Polyrepresentation Theory, demonstrating that combining two different representations of the same document (Abstract and Highlights) provides complementary semantic cues that enhance information retrieval tasks.
• Practical Application:
  • For Publishers: Reinforces the importance of the "Highlights" section for discoverability and indexing.
  • For Researchers: Provides a low-cost, unsupervised method to improve keyword extraction without requiring new training data, simply by leveraging existing metadata.
• Future Directions: The paper identifies the need for better prompt engineering for LLMs in this context and suggests future work on automatically generating highlights for papers that lack them, thereby extending the applicability of this method to the broader academic corpus.

In conclusion, the paper establishes that Highlights are a high-value, underutilized resource for unsupervised keyword extraction. By integrating them with abstracts, researchers can achieve more accurate and comprehensive keyword sets, bridging the gap between author intent and automated extraction.