Generative AI Availability, Grades, and Student Satisfaction at a Large University

Technical Summary: Generative AI Availability, Grades, and Student Satisfaction at a Large University

Problem Statement
The rapid adoption of Generative AI (GenAI) in higher education has raised concerns regarding the "GenAI substitution hypothesis." This hypothesis posits that students may offload the cognitive effort necessary for learning to AI tools, thereby achieving high grades on assessments without mastering the underlying material. If true, this substitution would disproportionately inflate grades in "GenAI-susceptible" courses (those relying on take-home problem sets, essays, and open exams) compared to "non-susceptible" courses (those relying on in-class, proctored, or live performance assessments). Furthermore, such substitution could erode student satisfaction by reducing engagement with assessments and weakening social learning communities. Prior literature suggests GenAI may increase productivity in the workplace, but its impact on student learning outcomes and satisfaction at a university-wide scale remains contested, with some studies suggesting grade inflation while others remain inconclusive.

Methodology
The study utilizes a large-scale administrative dataset from a major U.S. public university covering the period from Fall 2016 to Fall 2025. The dataset comprises 138,386 unique students and 72,730 course offerings across 6,357 unique courses.

• Treatment Definition (GenAI Susceptibility): The authors define a course's susceptibility as the proportion of the final grade allocated to assessments that can be completed independently of instructor observation (e.g., take-home exams, homework, papers, projects). In contrast, non-susceptible assessments include in-person closed-book exams, presentations, and live skill demonstrations.
• Measurement Pipeline: To quantify susceptibility, the authors developed a human-validated Large Language Model (LLM) pipeline. They scraped 44,876 syllabi and used an LLM (GPT-5.1) to reconstruct assessment types and their weightings. This pipeline was validated against a stratified sample of 525 human-consensus labels, achieving a Mean Absolute Error (MAE) of 0.063 on the 0–1 susceptibility scale.
• Empirical Strategy: The study employs a Difference-in-Differences (DiD) design, comparing outcomes in high-susceptibility versus low-susceptibility courses before and after the release of ChatGPT in November 2022.
  • Anchoring: Crucially, course susceptibility is anchored to the 2019 offering (pre-COVID, pre-GenAI) to prevent the treatment variable from absorbing instructor responses to the pandemic or GenAI.
  • COVID-19 Adjustment: Recognizing that the pandemic disrupted higher education and overlaps with the pre-treatment period, the authors model the COVID effect explicitly. They estimate bounds for the GenAI effect under two assumptions: (1) COVID effects were fully transient (returning to pre-pandemic baselines by 2023), and (2) COVID effects were fully persistent (creating a permanent level shift). The "GenAI net COVID" estimate is derived by differencing the post-AI effect from the COVID effect, providing a conservative bound.
  • Fixed Effects: The models include student, course, and semester fixed effects to control for time-invariant heterogeneity and common shocks.
• Outcomes:
  • Performance: Final grade value (GPA), withdrawal rates, and failure rates.
  • Satisfaction: Median scores from course evaluations regarding self-reported subject understanding, interest in the subject, and relative workload.
  • Heterogeneity: Analyses stratify students by prior academic preparation (using a course-residualized within-cohort first-term GPA rank) to test for equalizing or widening effects.

Key Results

• Average Grade Effects: The study finds no statistically significant differential effect of GenAI availability on final grades in more susceptible courses compared to less susceptible ones. The estimated effect ranges from 0.030 to 0.045 grade points (on a 4.0 scale), which is economically small and statistically insignificant at the 5% level. This null result holds under both transient and persistent COVID modeling assumptions.
• Grade Distribution: While there is a descriptive increase in the share of A grades in susceptible courses, this shift fails parallel trends tests and is not robustly identified as causal. There is no evidence of a "floor-raising" effect; failure and withdrawal rates remain unchanged.
• Heterogeneity by Prior Performance: Contrary to findings in some workforce studies where AI aids lower performers most, this study finds no significant grade effects for students at the bottom, middle, or top of the prior academic performance distribution. The null average effect is not the result of offsetting gains and losses across ability terciles.
• Student Satisfaction: After accounting for COVID effects, GenAI availability shows no significant impact on self-reported understanding or interest. There is a suggestive, modest increase in interest and decrease in relative workload only under the assumption that COVID effects were transient; however, these findings are not robust across all specifications.

Contributions

1. Methodological Rigor: The paper introduces a systematic, human-validated LLM pipeline for extracting assessment weightings from syllabi, offering a more rigorous measure of GenAI susceptibility than prior task-composition proxies. It also demonstrates that failing to anchor susceptibility to a pre-treatment period (specifically pre-COVID) can introduce selection bias and spurious positive grade effects.
2. University-Scale Evidence on Satisfaction: This is the first study to examine the impact of GenAI availability on student satisfaction (via course evaluations) at a university-wide level, finding no robust evidence of erosion in engagement or interest.
3. Heterogeneity Analysis: The study provides the first quasi-experimental evidence on how GenAI grade effects vary by students' prior academic preparation, finding no differential impact across the ability distribution.

Significance and Claims
The authors conclude that their findings temper concerns that GenAI has already degraded the signaling value of grades or reduced student satisfaction in the immediate years following ChatGPT's release. The study challenges the emerging consensus from two prior large-scale studies (Hausman et al., 2025; Chirikov, 2026a) that found evidence of grade inflation. The authors argue that the positive effects found in prior work may be artifacts of modeling choices, specifically the failure to separate lingering COVID-19 disruptions from GenAI effects and the use of time-varying susceptibility measures that allow for selection bias.

The paper modestly claims that while GenAI substitution could theoretically degrade assessments, the data from this large university setting does not yet support the hypothesis that grades are being inflated or satisfaction eroded. The authors caution that the lingering effects of the pandemic may still eclipse the effects of GenAI, and that the absence of evidence is not evidence of absence, suggesting a need for continued monitoring rather than a rushed pedagogical response.