How Do Algorithms Interact in Scientific Research? Large-Scale Semantic Relation Identification and Analysis Based on Full-Text Papers

Technical Summary: How Do Algorithms Interact in Scientific Research?

Problem Statement
In the era of artificial intelligence, algorithms are central drivers of scientific innovation. While existing research has successfully identified algorithm entities and analyzed their mention frequencies and co-occurrences, these approaches offer limited insight into the nature of interactions between algorithms. Co-occurrence analysis can indicate that two algorithms appear together but fails to explain why they appear together, how they compete, collaborate, or evolve. There is a lack of large-scale, fine-grained semantic relation identification that moves beyond simple entity lists to structured relation graphs, hindering a systematic understanding of algorithm innovation trajectories and community evolution.

Methodology
This study focuses on the Natural Language Processing (NLP) domain to identify and analyze semantic relations between algorithm entities in full-text academic papers. The methodology proceeds through four main stages:

1. Data Collection and Preprocessing:

   • Corpus: Full-text papers from three major NLP conferences (ACL, EMNLP, NAACL) published between 1979 and 2025 were collected.
   • Entity Extraction: An updated training corpus was constructed by augmenting existing data (1979–2020) with manually annotated sentences from 2021–2025. The model SciBERT + BiLSTM + CRF was fine-tuned to extract algorithm entities and filter relevant sentences.
   • Standardization: A rigorous normalization pipeline was applied, including tokenization, lemmatization, and vector-based clustering (using Gemma embeddings and ChromaDB) to merge variant names of the same algorithm, resulting in a final set of 30,274 unique algorithm entities.

2. Relation Framework and Annotation:

   • A semantic relation framework was defined with six categories: Compare with, Use, Cooperate with, Same as, Superclass of, and Other.
   • A gold-standard corpus of 16,531 sentences containing multiple algorithm entities was manually annotated by four annotators, achieving a pairwise inter-annotator agreement (Cohen's Kappa) of 0.7630.

3. Model Evaluation for Relation Extraction:

   • Large Language Models (LLMs): Several models (GPT-5, DeepSeek-Reasoner, Gemini-3.1-pro-preview, Claude-Sonnet-4-6) were evaluated using zero-shot, few-shot, and chain-of-thought (CoT) prompting strategies.
   • Pre-trained Language Models (PLMs): Models including BERT, SciBERT, and DeBERTa (with and without BiLSTM layers) were supervised fine-tuned. Additionally, Qwen-series models were fine-tuned using Low-Rank Adaptation (LoRA).
   • Selection: SciBERT achieved the best performance and was selected as the final model for large-scale relation extraction.

4. Network Analysis:

   • Interaction intensity was calculated based on the frequency of algorithm pairs appearing in papers over time, normalized by total publication volume.
   • Yearly relation networks were constructed for each relation type to analyze evolution in terms of network size (nodes/edges), internal connectivity (average degree, clustering coefficient), and renewal dynamics (edge similarity, new/disappearing edges).

Key Results

• Model Performance: SciBERT achieved the highest macro-F1 score (77.79%) for relation identification, outperforming both fine-tuned LLMs (e.g., Qwen-8B at 75.60%) and direct LLM prompting. While LLMs with few-shot learning and CoT reasoning showed competitive results (e.g., GPT reaching 74.63% F1), they did not surpass the fine-tuned domain-specific PLM.
• Relation Distribution: The Compare relation is dominant, accounting for 54.37% of all identified semantic relations. Synonym (12.54%) and Use (10.67%) follow, while Collaborate is the least frequent (5.68%).
• Interaction Patterns:
  • High-intensity pairs often reflect technological clustering (e.g., BERT–RoBERTa, GPT-3.5–GPT-4).
  • Compare relations involve both intra-category competition (e.g., BERT vs. RoBERTa) and cross-paradigm comparisons (e.g., BERT vs. ELMo).
  • Superclass relations reveal hierarchical structures, particularly in the lineage of pre-trained models and their variants.
• Network Evolution:
  • All relation networks have transitioned from sparsity to expansion since 2000.
  • The Compare network shows the most rapid growth in nodes and edges, the highest average degree, and the highest clustering coefficient, indicating a deepening and densifying competitive ecosystem.
  • Networks exhibit high renewal dynamics (high entry and disappearance of edges), suggesting that algorithm relations are not static accumulations but dynamic systems. The Compare network, however, shows greater stability in edge retention compared to Use or Collaborate networks.

Significance and Claims
The authors claim that this study provides a critical shift from isolated entity identification to structured relation identification, offering a new basis for exploring knowledge organization and evolution patterns in scientific research. By constructing relation-specific networks, the study reveals that the NLP field is characterized by an increasingly intense ecosystem of algorithm competition, driven largely by performance benchmarking and method substitution.

The paper posits that the dominance of the Compare relation reflects not only technological competition but also the evolving research practices, writing conventions, and evaluation logics of the academic community (e.g., the necessity of baseline comparisons). The study provides data resources and methodological references for constructing knowledge graphs among algorithm entities, though the authors modestly note limitations regarding the scope of data (NLP conferences only) and the need for further refinement in entity disambiguation and relation classification methods.