💬 NLP

What Are LLMs Doing to Scientific Communication? Measuring Changes in Writing Practices and Reading Experience

This paper investigates how large language models are reshaping scientific communication in NLP by analyzing a corpus of ACL papers and synthetic datasets to reveal significant stylistic shifts toward more complex, less diverse language that experts find more understandable yet subjectively problematic.

Original authors: Filip Miletić, Neele Falk

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Filip Miletić, Neele Falk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of scientific writing as a giant, bustling library where researchers write books to explain how computers understand language. For years, these authors wrote in their own unique voices, like different chefs cooking with their own secret spices. But recently, a new, very powerful kitchen assistant has arrived: the Large Language Model (LLM), or "AI writer."

This paper asks a simple question: Has this new assistant changed the flavor of the books in the library?

The authors, who are like food critics for language, decided to investigate this by looking at two things: a massive collection of real scientific papers written before and after the AI assistant became popular, and a controlled experiment where they took old paragraphs and asked the AI to "polish" them.

Here is what they found, broken down into simple analogies:

1. The Vocabulary Shift: From "Home Cooking" to "Gourmet Plating"

The researchers noticed that the words scientists are using have changed.

Before: Writers used simple, direct words like "use," "good," or "show."
After: Writers are using fancier, more formal words like "utilize," "comprehensive," and "demonstrate."

The Analogy: Imagine a chef who used to say, "I put the salt in the soup." Now, they say, "I have incorporated sodium chloride to enhance the flavor profile." The AI assistant seems to be pushing writers toward a more "gourmet" style, swapping simple ingredients for more complex, expensive-sounding ones.

2. The Semantic Shift: Narrowing and Widening

The paper found that some words are being used in more specific ways, while others are becoming more general.

Specialization: The word "ensure" is now used more strictly to mean "guarantee a specific quality," whereas before it was used more loosely.
Generalization: The word "utilize" has become a catch-all for "using something," losing some of its specific nuance.

The Analogy: Think of a toolbox. Before, a hammer was just a hammer. Now, some tools are being used only for very specific, delicate jobs (specialization), while others are being used for almost anything, becoming "all-purpose" tools (generalization). The AI seems to be reorganizing the toolbox.

3. The Style Shift: The "AI Polish"

When the researchers compared human-written text to text that the AI had "improved," they found distinct patterns in the AI's style:

Longer, Denser Words: The AI prefers longer words and packs more meaning into fewer sentences.
Less Variety: The AI tends to reuse the same set of "fancy" words, making the text feel a bit less diverse, like a playlist that keeps looping the same few hits.
More "Polite" Structure: The AI adds more connecting words and complex sentence structures, making the text feel more elaborate and qualified.

The Analogy: If human writing is like a casual conversation at a coffee shop, the AI-polished writing is like a formal speech at a gala. It's smoother, more structured, and uses bigger words, but it might feel a bit less like a real person talking.

4. The Reader's Experience: The "Smooth Ride"

Finally, the researchers asked 20 experts (who are like seasoned drivers) to read both the original human text and the AI-polished text. They asked: "Which one is clearer? Which one feels more trustworthy? Which one is more exciting?"

The Results:

Clarity & Excitement: The experts generally preferred the AI-polished text. They found it smoother to read and more "exciting."
Trust & Authenticity: However, the experts were split. Some felt the AI text was less "authentic" or trustworthy, sensing that a machine might have touched it.

The Analogy: It's like driving two different cars. The AI-polished car (the new text) drives smoother and feels faster (more exciting), but some drivers feel the steering wheel is a bit too perfect, making them wonder if a human is actually in the driver's seat.

The Bottom Line

The paper concludes that the rise of AI writing assistants has indeed changed the "flavor" of scientific writing. The text has become more complex, formal, and polished. While this makes the writing easier and more exciting to read for many, it also introduces a subtle shift where the human voice feels slightly less distinct. The authors didn't test if this makes the science better or worse, only that the style has changed and that readers notice it.

Technical Summary: What Are LLMs Doing to Scientific Communication?

Problem Statement
The rapid integration of Large Language Models (LLMs) into scientific writing raises critical questions regarding the evolution of writing practices and their subsequent impact on the reading experience. While prior research has focused on distinguishing entirely human-written text from fully AI-generated text, this binary distinction oversimplifies the reality of scientific communication, where LLMs are predominantly used to refine, polish, or improve human-authored drafts. Furthermore, existing studies often lack systematic measurement of how these stylistic shifts affect human readers' subjective perception of clarity, trustworthiness, and engagement. This paper addresses the gap in understanding the nuanced, collaborative nature of human-LLM writing and its measurable effects on linguistic style and reading experience within the Natural Language Processing (NLP) domain.

Methodology
The study employs a dual-data approach to capture both naturalistic trends and controlled stylistic variations:

Naturalistic Corpus (ACL Anthology): The authors updated the ACL-OCL corpus to include 99.2k papers, defining two distinct time periods: $t_1$ (Jan 2020–Dec 2022, pre-ChatGPT) and $t_2$ (Jul 2023–Dec 2024, post-ChatGPT). This subcorpus contains over 37,000 papers. The authors ensured topical comparability through BERTopic analysis, confirming that while specific research trends shifted (e.g., toward LLMs), the overall distribution of topics remained broadly comparable.
Synthetic Dataset (LLM-Assisted Paraphrases): To isolate the effects of LLM intervention, the authors sampled 3,000 paragraphs from $t_1$ papers and generated LLM-improved versions using GPT-3.5-turbo with 10 distinct prompts reflecting common scientific writing refinement tasks (e.g., "improve coherence," "polish grammar"). This resulted in 6,000 paired paragraphs (human vs. LLM-modified).

Analysis Framework
The study addresses three Research Questions (RQs) through a multi-layered analysis:

RQ1 (Lexical Choices): The authors conducted diachronic lexical analyses comparing $t_1$ $t_{1}$ and $t_2$ $t_{2}$ . This involved:
- Frequency Analysis: Calculating log-likelihood scores to identify words with significant usage rate changes.
- Type-Level Embeddings: Training Word2Vec models to measure "neighborhood density" (semantic specialization vs. generalization).
- Token-Level Embeddings: Using ModernBERT to cluster contextual usages of specific target words, identifying shifts in semantic sense distributions.
RQ2 (Stylistic Features): A logistic regression model with elastic net regularization was applied to 145 linguistic features (extracted via elfen and LFTK) covering surface, syntactic, psycholinguistic, and sentiment dimensions. Stability selection was used to identify robust predictors distinguishing $t_1$ from $t_2$ and human from LLM-modified text.
RQ3 (Reading Experience): A pilot annotation study was conducted with 20 NLP domain experts. Participants performed pairwise comparisons of human-written vs. LLM-improved texts, rating them on four dimensions: Clarity, Authenticity, Trustworthiness, and Excitement, using a 4-point Likert scale.

Key Results

Lexical Shifts: Significant changes were observed in word frequency and usage contexts.
- Specialization vs. Generalization: Some terms (e.g., ensure) showed increased neighborhood density, indicating semantic specialization (narrowing to specific transitive contexts like "guaranteeing quality"). Others (e.g., utilize) showed decreased density, indicating semantic generalization (broadening to mean "make use of").
- Stylistic Replacement: General-language expressions (e.g., use, good) declined in favor of more formal or specialized terms (e.g., utilize, comprehensive). This trend was consistent across both the naturalistic corpus and the synthetic LLM dataset, suggesting LLMs drive these stylistic shifts.
Stylistic Characteristics of LLM-Modified Text: Regression analysis revealed that LLM-modified texts are characterized by:
- Complexity: Longer words, higher average age of acquisition, and more complex syntactic constructions (e.g., adverbial clauses, negations).
- Density: Lower lexical diversity (Simpson's D) and reduced verb variability, yet higher semantic density (fewer stopwords).
- Structure: A preference for proper nouns and specific punctuation patterns (more commas/dashes, fewer brackets).
- Sentiment: Increased use of sensorimotor words (e.g., enhance, highlight) and anger-related words, though trust-related words were less frequent in some contexts.
Reading Experience: The annotation study revealed a subjective preference for LLM-improved texts:
- Positive Ratings: Experts rated LLM-modified texts significantly higher in Clarity and Excitement.
- Mixed/Negative Ratings: While Trustworthiness and Authenticity showed smaller effect sizes, a notable portion of raters preferred human originals for these dimensions.
- Subjectivity: Qualitative feedback indicated that experts found the task challenging and held generally negative attitudes toward LLM writing, despite the positive quantitative ratings for clarity. This highlights a disconnect between perceived readability and trust/authenticity.

Significance and Contributions
The paper claims four primary contributions:

Empirical Evidence of Style Shift: It demonstrates that post-ChatGPT scientific papers exhibit distinct lexical and stylistic patterns (e.g., lower lexical diversity, higher word complexity) attributable to LLM use, validated by comparing naturalistic data with controlled synthetic experiments.
Link to Reading Experience: It establishes a connection between these objective stylistic changes and subjective reading experiences, finding that LLM-improved texts are perceived as clearer and more exciting, though not necessarily more authentic or trustworthy.
Resource Release: The authors release an updated ACL-OCL corpus (99.3k papers) with a script for future ingestion, and a new dataset of 3,000 human-LLM text pairs with expert annotations for 200 pairs.
Methodological Framework: The study provides a comprehensive framework for analyzing human-LLM co-authored text, moving beyond binary detection to characterize the specific linguistic fingerprints of AI-assisted writing and their impact on the scientific community.

The authors conclude that while LLMs appear to enhance the perceived clarity and engagement of scientific texts, they simultaneously introduce a more homogeneous and complex writing style, the long-term implications of which warrant further investigation.

1. The Vocabulary Shift: From "Home Cooking" to "Gourmet Plating"

2. The Semantic Shift: Narrowing and Widening

3. The Style Shift: The "AI Polish"

4. The Reader's Experience: The "Smooth Ride"

The Bottom Line

More like this