Exploring Academic Influence of Algorithms by Co-occurrence Network Based on Full-text of Academic Papers

Technical Summary: Exploring Academic Influence of Algorithms by Co-occurrence Network Based on Full-text of Academic Papers

Problem Statement
In the era of artificial intelligence, algorithms are pivotal research objects. While existing methods evaluate the academic influence of algorithms, they predominantly rely on individual metrics such as mention frequency or citation counts. These approaches treat algorithms as isolated entities, neglecting the collective impact arising from their interconnectedness within a research community. Consequently, current evaluations fail to capture the structural roles algorithms play within clusters, such as their ability to bridge different eras or mediate between distinct algorithmic modules. There is a lack of research analyzing whether algorithms in a specific field are mentioned together, or exploring the interconnection, separation, and status gaps between algorithm individuals within a community.

Methodology
The study constructs a co-occurrence network of algorithm entities within the Natural Language Processing (NLP) domain using the full-text content of academic papers from the Annual Meeting of the Association for Computational Linguistics (ACL) spanning 1979 to 2020.

1. Data Collection and Entity Extraction:

   • Corpus: The study utilizes a training corpus of 4,604 ACL papers (1979–2015) with manually annotated algorithm sentences and a raw corpus of 2,793 papers (2016–2020) in PDF format.
   • Extraction Model: The task is framed as Named Entity Recognition (NER). Three deep learning models (BiLSTM+CRF, BERT+CRF, and BERT+BiLSTM+CRF) were trained and compared. The BERT+BiLSTM+CRF model achieved the highest F1 score (0.9642) and was selected for extracting candidate algorithm entities.
   • Normalization: A rigorous normalization process was applied to handle variations in naming (e.g., "SVM" vs. "Support Vector Machine") and ambiguity (e.g., "BP" referring to back-propagation vs. belief propagation). This involved edit distance algorithms for full names and rule-based/manual review for acronyms, resulting in a final dictionary of 3,841 unique algorithm entities.

2. Network Construction:

   • Nodes and Edges: Algorithms serve as nodes. An edge is established between two nodes if they appear in the same paper. Edge weights are calculated based on the number of co-occurring papers, normalized by the total number of ACL papers published that year.
   • Network Types: Three network configurations were built: (1) an Overall Network (all papers), (2) Cumulative Networks (aggregated from 1979 to each specific year), and (3) Annual Networks (papers published in a single year).

3. Influence Evaluation:

   • The study employs four network centrality metrics to assess academic influence from different perspectives:
     • Popularity: Measured by Degree Centrality (DC), reflecting the number of direct connections.
     • Control: Measured by Betweenness Centrality (BC), indicating the algorithm's ability to act as a bridge between unconnected nodes.
     • Central Position: Measured by Closeness Centrality (CC), reflecting the efficiency of information dissemination to other nodes.
     • Balance: Measured by Adjacency Information Entropy (AC), assessing the uniformity of an algorithm's influence across its neighbors.

Key Results

• Network Topology: The algorithm co-occurrence network exhibits characteristics of a complex network, specifically a "small world" structure with a high clustering coefficient (82.5%), a large giant component (99.2%), and a power-law degree distribution (scale-free). This indicates that a small number of algorithms hold the majority of connections.
• Evolution of Connectivity: Over four decades, the network evolved from a loose structure to a tightly connected small world. The average degree increased significantly, with 2001 marking a turning point in growth rates. The average separation between nodes peaked in 1994 and subsequently decreased, indicating improved communication efficiency.
• High-Influence Algorithms:
  • Overall: Algorithms that score highly across multiple centrality metrics share common traits: they are classic, high-performing, and often emerge at the junction of different eras. Long Short-Term Memory (LSTM) (Neural Networks), Support Vector Machine (SVM) (Classification), Adaptive Moment (Adam) (Optimization), and Conditional Random Field (CRF) (Probabilistic Graphical Models) emerged as top influencers in their respective categories.
  • Temporal Shifts: The dominant algorithms shifted over time:
    • 1979–1993: Grammar-based algorithms (e.g., ATN, CFG, TAG) were central.
    • 1994–2015: Statistical machine learning algorithms (e.g., HMM, MaxEnt, SVM) dominated.
    • 2016–2019: Deep learning models, particularly LSTM, became the core.
    • 2020: The pre-training model BERT rapidly ascended to the top position, signaling a new era.
• Influence Evolution Patterns: The study identifies a lifecycle for algorithm influence. As an algorithm's influence wanes, it typically loses its core position (Betweenness Centrality) first, followed by a decline in its balanced influence (Adjacency Information Entropy), and finally a reduction in the number of connected neighbors (Degree Centrality).

Significance and Contributions
The paper claims to be the first large-scale analysis of algorithm networks based on full-text content, offering a methodology that moves beyond simple frequency counting to evaluate influence through network topology.

• Methodological Innovation: By utilizing full-text data rather than metadata, the study captures a more complete landscape of algorithm usage and relationships.
• New Perspective on Influence: The results suggest that traditional frequency-based leaderboards may overlook the structural importance of "bridge" algorithms. Network-based metrics reveal that algorithms acting as transitional links between different eras or sub-fields (e.g., SVM bridging statistical and deep learning) possess unique, high-value roles that frequency alone cannot quantify.
• Foundation for Future Research: The constructed network serves as a cornerstone for building multifaceted networks interlinking algorithms, scholars, and tasks. This facilitates future exploration of scientific roles, semantic relations, and the development of algorithm recommendation and retrieval services.

Limitations
The authors acknowledge that the study is limited to the NLP domain (ACL papers), and results may not generalize to other fields with different citation or mention practices. Additionally, the analysis is based on article-level co-occurrence, lacking sentence-level granularity, and the network reflects co-occurrence relationships without explicitly modeling semantic nuances like cooperation or competition.