Human--LLM Collaboration Is Transforming Complexity Metrics in Scientific Texts

Technical Summary: Human–LLM Collaboration Is Transforming Complexity Metrics in Scientific Texts

Problem Statement
Large Language Models (LLMs), particularly since their widespread adoption in early 2023, are becoming integral contributors to the human–AI linguistic ecosystem. While LLMs are trained on human language, their integration into scientific communication raises questions about how they alter the emergent properties of language. There are competing theoretical perspectives: one suggests LLMs may reduce lexical complexity through simplification and the recycling of training data (potentially leading to "model collapse" if future models train on AI-generated outputs), while another suggests that mixed human–AI populations may generate novel emergent dynamics through collective biases and feedback loops. The core problem addressed is whether the influx of LLM-generated or LLM-revised text into scientific literature (specifically arXiv abstracts) produces detectable shifts in the statistical laws governing lexical complexity, such as Zipf's law, Heaps' law, and word turnover rates.

Methodology
The authors treat the rapid adoption of generative AI tools in 2023 as a "natural experiment." They analyze a corpus of over 2.7 million arXiv abstracts from January 2010 through September 2025, focusing on first-version (v1) submissions.

1. LLM Style Index Construction: To quantify the presence of LLM influence, the authors constructed a composite "LLM-associated style index." This index aggregates the standardized monthly frequencies of specific stylistic markers known to increase with LLM use: dash usage (en-dashes, em-dashes, hyphens) and specific lexical markers ("significant," "crucial," "showcase").
2. Complexity Metrics: The study tracks three primary measures of lexical structure:
   • Vocabulary Size: The number of unique word types.
   • Heaps' Law Exponent ($b$): Describing the relationship between corpus size and vocabulary growth ($V(N) = aN^b$).
   • Zipf's Law Exponent ($\alpha$): Describing the rank-frequency distribution of words ($P(r) \propto r^{-\alpha}$).
   • Top-$y$ Turnover: The fraction of new words entering the top-$y$ ranked list of content words year-over-year, serving as a diagnostic for transmission bias (conformity vs. novelty).
3. Statistical Analysis:
   • Benchmarking: Human text was compared against LLM-generated text (GPT-3.5) using the HC3 corpus to establish baseline differences.
   • Temporal Trends: Monthly statistics were calculated for the arXiv corpus from 2014 to 2025.
   • Regression Modeling: Multivariate models were employed to predict lexical complexity measures using publication year, the LLM style index, a post-2023 indicator, and their interaction terms. This approach isolates the effect of LLM style from general secular trends.
   • Early Warning Signals (EWS): Rolling lag-1 autocorrelation, skewness, and variance were analyzed to detect critical slowing down or instability in the time series.

Key Results

• Baseline Differences: In the HC3 corpus, LLM-generated text exhibited a slightly lower Heaps' exponent ($b=0.522$) and a slightly higher Zipf exponent ($\alpha=1.741$) compared to human text ($b=0.539$, $\alpha=1.681$), indicating marginally lower lexical diversity and higher concentration in LLM output, though the distributions remain similar.
• Temporal Shifts in arXiv:
  • Style Index: A sharp increase in the LLM-associated style index was observed starting in 2023, coinciding with the widespread adoption of tools like ChatGPT.
  • Lexical Metrics: Changes in the exponents of Zipf's and Heaps' laws were subtle. However, vocabulary size and the Heaps' exponent showed modestly accelerated growth post-2023.
  • Interaction Effects: Crucially, the relationship between the LLM style index and lexical complexity metrics changed after 2022. The positive correlation between the style index and vocabulary size, Heaps' exponent, and Zipf exponent became flatter in the post-2023 period. Regression models confirmed that vocabulary growth after 2023 exceeded what would be expected from prior temporal trends alone, suggesting a distinct shift in dynamics.
  • Turnover: The most significant structural change was a sharp increase in the turnover of top-ranked words starting in 2023. The scaling of turnover against list size shifted from slightly concave-up ($b=1.15$) pre-2023 to slightly concave-down ($b=0.969$) post-2023, indicating a higher rate of new words entering the top ranks.
• EWS Diagnostics: Standard early warning signal diagnostics (autocorrelation, variance) showed only modest changes, suggesting the system has not reached a critical tipping point of instability, despite the structural shifts in turnover.

Significance and Claims
The paper claims that the integration of LLMs into scientific writing is altering the emergent properties of the linguistic ecosystem in ways that are subtle but measurable. Contrary to hypotheses that LLMs might uniformly reduce lexical diversity, the authors find that current LLM contributions are associated with increased vocabulary richness and accelerated turnover in high-frequency terms.

The flattening of the relationship between LLM style and complexity metrics suggests that the dynamics of a mixed human–AI population are distinct from pure human or pure AI generation. The increased turnover implies a shift in the "transmission bias" of the system, potentially injecting more novelty or altering the conformity dynamics of scientific discourse.

The authors conclude that these shifts point to a potential tension between short-term efficiency and long-term informational diversity. They caution that if diversity in human-generated and hybrid language declines over time, it could impact the robustness of future AI systems trained on such data. The study frames LLMs not merely as tools but as a new feedback mechanism in cultural evolution, comparable in consequence to the emergence of writing or markets, necessitating ongoing monitoring of how human–AI collaboration reshapes the complexity of scientific knowledge.