Analyzing Students' Statistics Writing Before and After the Emergence of Large Language Models

Technical Summary: Analyzing Students' Statistics Writing Before and After the Emergence of Large Language Models

Problem Statement
The undergraduate statistics and data science curriculum relies heavily on project-based learning and writing assignments to foster "writing-to-learn" (deepening conceptual understanding) and "writing-in-the-disciplines" (communicating results to stakeholders). A primary concern with the widespread adoption of Large Language Models (LLMs) is that students may offload critical cognitive tasks—such as interpreting coefficients, articulating limitations, and structuring arguments—to generative AI. This offloading risks stalling students' writing development at the "knowledge-telling" stage, preventing them from reaching the "knowledge-crafting" stage required for expertise, and potentially undermining the GAISE 2016 goal of producing critical consumers of statistical results. The paper seeks to empirically determine how the emergence of LLMs has altered the writing style and verb usage in undergraduate statistics reports.

Methodology
The authors constructed a comparative corpus analysis involving three distinct datasets:

1. Student Corpus: 1,619 data analysis reports from an introductory statistics course (36-202) at Carnegie Mellon University, collected between Spring 2021 and Fall 2025. Reports were categorized into "pre-LLM" (2021–2022), a "transition year" (2023), and "since-LLM" (2024–2025).
2. LLM Corpus: Academic text generated by four LLMs (GPT-4o, GPT-5 Mini, Gemini 2.5 Flash, and Claude Haiku 4.5) based on the Human-AI Parallel English (HAP-E) academic corpus. These models were selected based on their accessibility to students during the study period.
3. Expert Corpus: 278 peer-reviewed applied statistics articles from CHANCE, The American Statistician, and the Journal of the American Statistical Association (published 2018–2022), pre-processing to remove mathematical density and ensure stylistic comparability to student reports.

Analytical Approach
The study employed two primary analytical techniques:

• Linear Discriminant Analysis (LDA): The authors utilized 67 Biber lexicogrammatical features (e.g., nominalizations, passive voice, pronoun usage, word length) to train LDA models.
  • An Overall LDA Model compared full reports across the three sources (pre-LLM student, LLM, expert).
  • A Quintile LDA Model split student reports into five length-based segments (roughly mapping to Introduction, EDA, Modeling, Prediction, and Discussion) to detect section-specific shifts in style.
• Keyness Analysis: The authors identified the top 50 verb lemmas used by each LLM (97 unique verbs total) and conducted binomial likelihood-ratio tests to determine if the usage rates of these verbs significantly increased in "since-LLM" student reports compared to "pre-LLM" reports.

Key Results

1. Stylistic Divergence: The LDA models successfully distinguished between pre-LLM student writing, LLM-generated text, and expert writing with 99.46% accuracy. Pre-LLM student writing was characterized by interactive features (first-person pronouns, contractions, existential "there") and lower informational density. LLM text was characterized by higher informational density, longer words, nominalizations, and present participles. Expert writing relied more on agentless passives and the verb "be."
2. Shift in Student Writing: Since the emergence of LLMs, student writing has shifted toward a mixture of LLM and expert styles, though the shift is most pronounced in the first quintile (Introduction) and fifth quintile (Discussion/Conclusion).
   • The posterior probability of a student report belonging to the "pre-LLM student" class decreased significantly in these sections (e.g., the first quintile dropped from 32.6% to 26.9% probability).
   • Students adopted stylistic features associated with LLMs (e.g., increased nominalizations, attributive adjectives, and present participles) rather than those uniquely associated with human experts.
3. Verb Usage Shifts: There was a statistically significant increase in the usage of "LLM-favorite" verbs in student reports post-2023. Notable increases included enhance (an 8.71-fold increase), suggest, and underscore. Concordance analysis of underscore revealed usage patterns (e.g., "underscores the complexity of") that were absent in pre-LLM reports but common in LLM outputs.
4. Expert-Like Appearance: While student writing became more similar to expert writing in terms of certain stylistic markers, the authors note this may be an incidental byproduct of mimicking LLMs rather than a genuine acquisition of expert rhetorical strategies.

Significance and Claims
The paper claims to provide strong evidence that undergraduate statistics writing has become more "LLM-like" in both style and verb usage since the widespread adoption of generative AI. The authors argue that this shift is particularly evident in the framing (introduction) and concluding sections of reports, suggesting students consult LLMs most heavily when they feel uncertain about structuring these non-technical components.

The authors modestly claim that while student reports may appear more "expert-like," this does not necessarily equate to improved statistical thinking or reasoning. They emphasize that if the "expert-like" quality stems from offloading cognitive work to AI, the educational value of writing-to-learn assignments is compromised. Consequently, the paper proposes that educators must reevaluate assessment methods. Rather than abandoning project-based learning, the authors suggest targeted assessments (e.g., specific rhetorical exercises for introductions) and oral exams to ensure students retain the ability to think critically about statistical problems and evaluate AI-generated outputs, aligning with the GAISE 2016 guidelines to "teach statistical thinking."

Limitations
The authors acknowledge that their findings are based on a single course at one institution and that their LLM corpus may not perfectly replicate the specific prompts or conversational interactions students use. Furthermore, the study analyzes writing style but does not measure the statistical quality or correctness of the underlying data analysis in the student reports.