Usage frequency and application variety of research methods in library and information science: Continuous investigation from 1991 to 2021

Technical Summary: Usage Frequency and Application Variety of Research Methods in Library and Information Science (1991–2021)

Problem Statement
Previous investigations into research methods within Library and Information Science (LIS) have relied heavily on manual content analysis. While foundational studies by scholars such as Järvelin and Vakkari (1990) and Chu (2015) established classification schemes, the manual nature of these analyses restricts the volume of data that can be processed. Consequently, most prior studies are limited to single years or small subsets of literature, preventing a comprehensive, longitudinal understanding of methodological evolution. Furthermore, existing research often treats research methods and research topics as separate entities, lacking a dynamic analysis of their interrelationships over extended periods. This study addresses the need for a large-scale, automated approach to classify research methods and analyze their usage frequency and application variety across a 31-year span (1991–2021).

Methodology
The study employs a machine learning (ML) pipeline to analyze 26,965 research articles from 21 core LIS journals. The methodology is structured into three primary components:

1. Dataset Construction and Expansion:

   • The dataset spans 1991–2021. A preliminary training set of 1,977 manually annotated articles (based on Chu and Ke, 2017) was used.
   • To address class imbalance (where certain methods like "Delphi study" had fewer than 300 instances), the authors expanded the training data. Using TextRank to extract keywords, they searched Web of Science for relevant abstracts, which were then manually filtered and annotated. This process created an expanded dataset for training.
   • Confident Learning: To mitigate noise introduced during data expansion, the study utilized "confident learning" via the Cleanlab tool to identify and prune mislabeled data before final model training.

2. Automatic Classification of Research Methods:

   • Coding Scheme: The study adopted Chu and Ke's (2017) taxonomy of 16 research methods (e.g., Bibliometrics, Content analysis, Interview, Theoretical approach).
   • Model Architecture: Several models were evaluated, including traditional multi-label classifiers (BR-XGB, CC-XGB, etc.) and deep learning architectures. The final selected model combined SciBERT (a BERT model pre-trained on scientific text) with TextCNN for feature extraction and classification.
   • Training Strategy: The models were trained using Adam and SGD optimizers with Binary Cross Entropy (BCE) loss. Five-fold cross-validation was used for evaluation.
   • Validation: The model's performance was validated against a manually labeled test set (N=80), yielding a Kappa value of 0.74, indicating acceptable consistency between machine and human annotation.

3. Topic Modeling and Metric Calculation:

   • Topic Identification: Top2Vec was used for topic modeling. The optimal number of topics was determined to be 18 based on CV (context-vector-based coherence) scores.
   • Metrics: Two key metrics were calculated:
     • Usage Frequency: The proportion of articles reporting a specific method in a given year.
     • Application Variety: An entropy-based measure quantifying the diversity of research topics a specific method is applied to in a given year. Higher entropy indicates broader applicability across different topics.

Key Results

• Classification Accuracy: The SciBERT-TextCNN model achieved the highest performance on the cleaned dataset, with a Micro-F1 score of 0.850, Macro-F1 of 0.848, and Samples-F1 of 0.859. This demonstrates that the automated approach achieves accuracy comparable to manual coding.
• Usage Frequency Trends:
  • Content analysis (18.63%) and Bibliometrics (17.70%) were the most frequently used methods.
  • A longitudinal shift was observed: Theoretical approach and Content analysis showed a downward trend, while Interview and Questionnaire showed an upward trend. This indicates a strategic shift from conceptual research to empirical research strategies in LIS.
• Application Variety Patterns:
  • Content analysis exhibited the highest application variety (7.084), acting as a "super-method" applicable to a wide range of topics.
  • Delphi study had the lowest variety (0.134), indicating narrow applicability.
  • Four distinct patterns emerged: methods with overall increased variety (e.g., Bibliometrics, Interview), those that increased then decreased (e.g., Questionnaire, Transaction log analysis), those with overall decreased variety (Theoretical approach), and those that decreased then increased (Historical method).
• Topic-Method Relationships:
  • Research topics shifted from system-centered issues (e.g., "Information retrieval/models and algorithms") to user-centered topics (e.g., "Information services").
  • Specific method-topic pairings were identified: Experiments were heavily used for "Text processing/retrieval" and "Information seeking behavior"; Theoretical approach was linked to "Information ethics/philosophy of information"; and Bibliometrics was central to "Science evaluation" and "Webometrics."
  • The study visualized these dynamic relationships via an interactive map, allowing for year-by-year longitudinal observation.

Significance and Contributions
The paper claims three primary contributions to the field of LIS research methodology:

1. Scalability via Machine Learning: By utilizing deep learning and confident learning, the study overcomes the data limitations of manual content analysis. It successfully processed a dataset of nearly 27,000 articles over 31 years, providing a panoramic view of methodological trends that was previously unattainable.
2. Dual-Perspective Analysis: Unlike previous studies that treated methods and topics separately, this research simultaneously analyzes usage frequency and application variety. It reveals how the breadth of a method's application changes over time, offering a more nuanced understanding of methodological utility.
3. Validation of Automated Classification: The study demonstrates that machine learning can achieve classification accuracy (Kappa = 0.74) comparable to manual human coding. This validates the use of automated techniques for large-scale bibliometric studies, suggesting that while manual coding remains essential for developing coding schemes, ML is a viable complement for executing large-scale analyses.

The authors maintain a modest stance, noting that their methodology is a complement rather than a complete replacement for manual analysis, particularly because human expertise is still required to refine coding schemes as research methods evolve. The study does not propose new theoretical frameworks but rather provides empirical evidence of the field's evolution through rigorous data-driven analysis.