Technical Summary: Data-Driven Evolution of Library and Information Science Research Methods (1990–2022)

Problem Statement

Since the 1990s, the Library and Information Science (LIS) field has undergone a paradigm shift from empirically driven qualitative research to data-driven quantitative research, fueled by advancements in big data, the World Wide Web, and artificial intelligence. While existing literature has explored the evolution of research methods using bibliometric perspectives, these analyses are often "coarse-grained," focusing on the textual characteristics of theory and method usage rather than the methods themselves. There is a lack of fine-grained, quantitative analysis regarding how specific data-driven method entities evolve in response to changing data resources and technologies. This study addresses the need to uncover the evolutionary trends of LIS research methods by extracting and analyzing specific, named method entities from academic literature.

Methodology

The study employs a multi-stage computational framework to analyze 25,351 full-text academic papers published between 1990 and 2022 from 14 high-quality LIS journals.

1. Data Collection and Preprocessing:

   • A corpus was constructed from 14 representative journals (Q1–Q3 in JCR 2021), filtering out non-academic documents and those with missing full-text content.
   • The dataset comprises 25,351 valid articles containing 153,230 chapters.

2. Chapter Structure Identification:

   • To isolate the specific methods used in studies, the "Methods" chapter must be accurately identified.
   • The study established a six-category structural classification system (Introduction, Related Work, Method, Evaluation and Results, Discussion and Conclusion, Other).
   • Eleven machine learning models were evaluated, including traditional ML (SVM, RF, LR), deep learning (SciBERT variants), and Large Language Models (T5 variants).
   • Selection: The T5_base model achieved the highest performance with a weighted average F1 score of 84.23% and was used to classify chapter structures across the corpus.

3. Fine-Grained Method Entity Extraction:

   • Four categories of Research Method Entities (RMEs) were defined: Algorithms and Models, Data Resources, Software and Tools, and Metrics.
   • A "silver standard" annotated corpus was constructed using ChatGPT for term generation, followed by manual verification.
   • Model Architecture: A SciBERT+BiLSTM+CRF model was trained for sequence labeling.
   • Performance: The model achieved a comprehensive F1 score of 73.38%. Data augmentation (adding 10% of augmented data) further optimized performance.
   • The model was applied to the entire corpus to extract 6,145,694 sentences containing method entities.

4. Evolutionary Analysis Metrics:

   • Distribution Density: Calculated as the ratio of method entity tokens to total tokens within a chapter to confirm that the "Methods" chapter contains the highest density of RMEs.
   • Jaccard Distance: Used to measure the dissimilarity (distribution discrepancy) of RME groups between adjacent time windows (3-year intervals).
   • Growth Rate of Jaccard Distance: Calculated to determine whether the differences in method usage are widening (emergence) or narrowing (stabilization).
   • Topic and Method Stratification: The Top2Vec model identified 19 research topics, and an automated classifier (Zhang & Tian, 2023) categorized papers by research method (e.g., experiment, bibliometrics) to analyze evolutionary patterns across these dimensions.

Key Contributions

• Fine-Grained Entity Extraction: The study moves beyond coarse bibliometric analysis by automatically extracting four specific types of data-driven method entities (Algorithms, Data, Tools, Metrics) from full-text academic papers.
• Methodological Framework: It proposes a robust pipeline combining structural identification (T5) and entity extraction (SciBERT+BiLSTM+CRF) to analyze research method evolution at a microscopic level.
• Quantitative Evolutionary Metrics: The application of Jaccard distance and its growth rate provides a quantitative measure of the "stability" versus "emergence" of research methods over time.
• Multi-Dimensional Analysis: The study analyzes evolution across three distinct dimensions: temporal trends, different research topics, and different research methodologies.

Results

• Chapter Distribution: The "Methods" chapter exhibits the highest average distribution density of RMEs (~1.94%), validating the approach of focusing on this section to identify actual research methods.
• Temporal Evolution (1990–2022):
  • Data Resources: Showed the largest distribution differences over time, indicating they are a primary driver of methodological change.
  • Cyclical Pattern: The study identified a cyclical pattern of "emergence" (increasing Jaccard distance, indicating new tools/data) followed by "stabilization/practical application" (decreasing Jaccard distance, indicating normalization).
  • Specific Trends: Algorithm and model entities showed a gradual increase in proportion, particularly post-2015, likely due to large language models. Metric entities showed decreasing proportions, suggesting a stabilization in evaluation standards.
• Topic-Specific Evolution:
  • Topics like Information Seeking and Informetrics/Scientometrics showed a trend of decreasing then increasing distribution variance in data resources, linked to the open access movement and resource abundance.
  • Scientific Evaluation topics showed stabilization in metric entities but decreasing variance in other categories.
• Method-Specific Evolution:
  • Experimental Methods: Showed an early decrease followed by a recent increase in distribution variance, particularly for algorithm and model entities, reflecting the evolving nature of data-driven experiments.
  • Content Analysis and Bibliometrics: Exhibited sustained decreases in distribution variance, indicating stable, established usage patterns.

Significance and Claims

The paper claims that data resources are the pivotal driver of methodological evolution in LIS. The abundance of data and the rise of data intelligence have fundamentally altered how research problems are solved, shifting the discipline toward data-driven models.

The study posits that research methods in LIS do not evolve linearly but follow a cyclical pattern of "emergence" and "stabilization/practical application." By focusing on fine-grained method entities, the authors argue that this approach offers a more objective and interpretable view of disciplinary development than traditional coarse-grained bibliometrics. The methodology is presented as a universal framework that can be applied to other fields to understand the development processes of research methods through the lens of specific knowledge entities.

The authors acknowledge limitations, noting that the current entity classification may not comprehensively cover all method usage characteristics and that the corpus, while large, may still be insufficient for a complete thematic evolution analysis. Future work is suggested to involve larger-scale data and more universal classification systems to enhance the objectivity and adaptability of the findings.