Generative AI and the future of scientometrics: current topics and future questions

Problem Statement
The field of scientometrics, which quantitatively studies science and technology, is increasingly adopting Generative Artificial Intelligence (GenAI) for tasks ranging from topic identification and classification to research evaluation and citation prediction. However, the current state of the field is characterized by a "trial-and-error" approach. Existing literature largely relies on empirical comparisons between GenAI outputs and human judgment or traditional techniques, often lacking a principled understanding of why GenAI succeeds or fails in specific contexts. There is no clear consensus on the reliability of GenAI in scientometrics, with debates ranging from its potential to democratize data access to concerns regarding its stability and quality. Furthermore, the field lacks a theoretical framework to anticipate performance based on task characteristics, and there is a growing concern that the automated generation of scientific language by GenAI may fundamentally alter the textual markers (vocabulary, authorship, references) upon which scientometric indicators rely.

Methodology
This paper is an opinion piece that proposes a conceptual framework to interpret GenAI performance and a theoretical discussion on its future impact. The methodology involves:

1. Linguistic Framework Construction: The authors adapt Charles Morris's tripartite distinction of language—syntax, semantics, and pragmatics—to categorize scientometric tasks.
   • Syntax: The well-formedness of text (grammar, structure).
   • Semantics: The meaning of words inferred from a corpus (classification, labeling).
   • Pragmatics: The relationship of text to specific communicative situations and contexts (evaluation, judgment).
2. Literature Synthesis: The authors review recent empirical studies (2024–2025) applying GenAI to scientometric tasks, mapping these applications against the proposed linguistic framework to identify patterns in performance.
3. Exploratory Theoretical Analysis: The paper speculates on the consequences of widespread GenAI adoption in academic writing, analyzing how it might distort core scientometric dimensions: vocabulary, authorship practices, productivity measures, and reference usage.

Key Contributions

1. A Conceptual Framework for Task Classification: The paper introduces a distinction between tasks requiring semantic inference (understanding meanings within a text) and those requiring pragmatic judgment (evaluating texts within specific institutional or social contexts).
2. Performance Prediction Criteria: It argues that GenAI performance is not uniform but depends on three key parameters:
   • The nature of the task (semantic vs. pragmatic).
   • The level of granularity (aggregate/broad vs. individual/fine-grained).
   • The goal of the analysis (descriptive/inferential vs. evaluative).
3. Strategic Guidance: The authors propose that GenAI is highly effective for semantic tasks (e.g., topic labeling, document classification) at an aggregate level but performs inconsistently in pragmatic tasks (e.g., peer review, identifying prominent scholars) where individual context and institutional nuance are critical.
4. Warning on Indicator Validity: The paper highlights a critical, often overlooked risk: that GenAI's generation of scientific text may alter the statistical distributions of words, authorship patterns, and citations, thereby invalidating established scientometric indicators that assume stable relationships between text and scientific practice.

Results and Findings

• Semantic Tasks: GenAI demonstrates strong performance in inferring meanings, such as labeling topics or classifying documents, particularly when applied to broad categories (e.g., scientific fields) rather than fine-grained distinctions. However, "hallucinations" and stochastic variation require calibration and validation against reference data.
• Pragmatic Tasks: Performance is mixed and unstable for tasks requiring context-dependent judgment, such as evaluating research quality, predicting citation impact, or identifying prominent scholars. While GenAI may show statistical regularities at an aggregate level, it fails to replicate the reliability of human judgment in individual cases.
• Human-Machine Integration: The paper concludes that robust adoption strategies require human-in-the-loop approaches. For semantic tasks, humans should focus on calibration and validation. For pragmatic/evaluative tasks, GenAI should serve as an auxiliary tool or assistant rather than a replacement for human judgment.
• Impact on Scientometric Measures: The authors identify four areas where GenAI threatens the validity of current measures:
  • Vocabulary: GenAI may increase stylistic homogeneity and alter word frequencies, potentially reinforcing existing topic associations and obscuring cognitive dynamics.
  • Authorship: Systematic use of GenAI for drafting may decouple listed authorship from actual intellectual contribution, complicating collaboration metrics.
  • Productivity: GenAI may lower the marginal cost of publication, potentially inflating output counts while reducing topical diversity, rendering traditional productivity indicators less comparable over time.
  • References: GenAI tools may fabricate references or select them based on word association rather than intellectual lineage, potentially distorting citation networks and the interpretation of scientific influence.

Significance and Claims
The paper claims that GenAI is not merely a new productivity tool but a force capable of altering the "textual representation of science." Its significance lies in shifting the discourse from simple performance benchmarking to a deeper theoretical and empirical re-evaluation of the field's foundations.

• For Practice: It urges scholars to move away from trial-and-error toward methodological consolidation, emphasizing that the choice to use GenAI must be dictated by the linguistic nature of the task and the required level of granularity.
• For Theory: It calls for a new research program to understand how GenAI affects established indicators. The authors argue that without careful empirical work and theoretical reflection, the field risks losing its ability to interpret evolving patterns of knowledge production.
• Modesty of Claims: The authors explicitly acknowledge that their discussion on the future impact of GenAI on scientometric measures is largely speculative due to the scarcity of existing empirical evidence. They frame their arguments as hypotheses requiring rigorous validation through longitudinal studies and controlled experiments (e.g., comparing pre- and post-GenAI corpora).