Measuring coverage of cited fields by author publication histories

Technical Summary: Measuring Coverage of Cited Fields by Author Publication Histories

Problem Statement
Bibliometric studies frequently utilize reference lists to characterize disciplinary diversity, knowledge combinations, and the age of literature. However, a critical distinction remains unaddressed: the relationship between the cited literature and the specific publication histories of the author producing the work. A diverse reference list may be well-represented in one research team's prior records while being largely absent in another's. Existing measures of expertise diversity often focus on distances between author profiles or the composition of the cited portfolio in isolation, without quantifying the intersection between the two. The paper posits that understanding team research requires a specific measurement of how much of a paper's cited reference distribution is represented in individual author histories versus the combined team history, and how much remains uncovered.

Methodology
The study introduces a reference-weighted decomposition framework to partition the cited fields of a focal paper into three components:

1. Maximum Individual Coverage ($M$): The share of cited categories covered by the single author with the most extensive relevant history.
2. Additional Coauthor Coverage ($D$): The incremental share of cited categories covered by combining the histories of all coauthors, beyond what $M$ covers.
3. Uncovered Remainder ($E$): The share of cited categories absent from every author's observed publication history.

The framework distinguishes between two observation windows: a preproject window (five years prior to the project start) and a prepublication window (five years prior to the focal paper). This allows for the calculation of net coverage change ($A = E_{pre} - E_{pub}$), separating baseline uncovered shares from subsequent gains or losses.

Data and Inferential Sequence
The study utilizes a linked dataset combining Korean National Science and Technology Information Service (NTIS) grant records, National Research Foundation (NRF) acknowledgments, and OpenAlex publications. The analysis follows a three-stage inferential sequence:

• Discovery Stage: 115 grant-linked papers.
• First Confirmation Stage: 240 papers (pooled $N=355$ for exploratory diagnostics).
• Second Confirmation Stage: A prospectively fixed protocol analyzing 194 grants (112 AI-use, 82 non-AI) selected from a larger pool of 832.

The primary application investigates whether projects classified as using Artificial Intelligence (AI) exhibit higher baseline uncovered shares ($E_{pre}$) compared to non-AI projects, testing hypotheses regarding AI's role in introducing unfamiliar knowledge.

Key Contributions and Metrics

• Decomposition of Coverage: The paper provides a reproducible method to separate individual, collective, and uncovered reference shares on a common denominator, weighted by citation frequency.
• Citation Emphasis Analysis: Through within-paper reallocation experiments, the study holds author histories, diversity, and field totals constant while permuting citation counts. This isolates the effect of where citations are placed (covered vs. uncovered fields) independent of the portfolio's composition.
• Distinction from Familiarity: The author explicitly states that $E$ measures the absence of indexed evidence, not the absence of actual expertise or learning. It is a descriptive accounting of the intersection between citations and records, not a proxy for cognitive familiarity.

Results

• Exploratory Diagnostics:

  • Reference diversity (Shannon entropy) remains positively associated with the uncovered share ($E_{pre}$) even after adjusting for author count, reference count, and history volume. In the discovery stage, adding diversity to the model increased $R^2$ from 0.328 to 0.502.
  • Citation Emphasis: Actual portfolios place significantly less weight on uncovered subfields than structural benchmarks. In the discovery stage, the observed uncovered share was 8.66 percentage points lower than the field-conditioned reallocation benchmark (95% CI: 6.22–11.36). This contrast persisted even after excluding references in the paper's own primary subfield.
  • Aggregation Effects: In AI-use papers, "additional coauthor coverage" ($D$) averages 10.89% per paper but only 3.45% per reference occurrence in the first-confirmation stage, highlighting how aggregation metrics can obscure the marginal contribution of coauthors to specific citations.

• AI Application (Hypothesis Testing):

  • Discovery: Suggested a positive association between AI use and uncovered share.
  • First Confirmation: Failed the primary test of positive net coverage change ($A$).
  • Second Confirmation: The primary endpoint was shifted to $E_{pre}$ (preproject uncovered share) based on earlier results. The AI-use coefficient was near zero (+0.455 points, 95% CI: −6.831 to 7.742) under parent-field adjustment and (+0.517 points, 95% CI: −7.395 to 8.428) under detailed-field adjustment.
  • Conclusion on AI: The study does not confirm that AI-use projects have a higher baseline uncovered share. While AI projects cite more AI literature, this literature is often already represented in the author's prior records.

Significance and Claims
The paper claims to offer a reproducible account of citation emphasis relative to documented publication histories. Its primary contribution is methodological: it establishes a framework to quantify how much of a paper's cited fields are "covered" by the team's past work versus "uncovered."

The author is modest in their claims, emphasizing that:

1. Coverage $\neq$ Familiarity: An uncovered reference indicates a lack of indexed history, not necessarily a lack of expertise or learning.
2. No Causal Mechanism: The study describes associations and structural contrasts but does not establish causal mechanisms for why teams cite uncovered fields or how knowledge is acquired.
3. Context Dependence: The results are specific to the observed, conservatively linked population of Korean grants and do not necessarily generalize to all research contexts without further validation.
4. Interpretation Limits: The framework identifies the relationship between citation emphasis and records but does not establish unfamiliarity. The failure to confirm the AI association suggests that AI research in this sample does not necessarily rely on fields outside the author's prior records.

Ultimately, the paper argues that measuring coverage provides a necessary layer of granularity to bibliometric analysis, distinguishing between the diversity of a paper's references and the alignment of those references with the author's documented expertise.