Measuring Changes in Instructor Class Design and Student Learning After the Release of Large Language Models (LLMs)

Technical Summary: Measuring Changes in Instructor Class Design and Student Learning After the Release of Large Language Models (LLMs)

Problem Statement
The widespread public release of Large Language Models (LLMs) in late 2022 has precipitated a shift in higher education, yet empirical evidence regarding its impact on student learning behaviors, faculty course design, and the validity of traditional grading metrics remains scarce. The authors identify a critical methodological gap: traditional metrics like grades are increasingly compromised as proxies for learning. This distortion arises from three factors: (1) the "cat and mouse" dynamic where instructors redesign assignments to be LLM-resistant, breaking longitudinal comparability; (2) the potential for undetected LLM use to artificially inflate grades without corresponding learning gains; and (3) the penalization of students caught using LLMs, which may result in failing grades despite actual conceptual understanding. Consequently, there is a lack of triangulated data combining historical grade records with qualitative perceptions from both faculty and students to assess learning trends in the "post-LLM era."

Methodology
This study employs a mixed-methods, multi-course design conducted at the University of Massachusetts, Boston (UMass Boston). The research triangulates three distinct data sources:

1. Retrospective Quantitative Analysis: Historical grade data spanning up to ten years across approximately fifteen courses was obtained from the university registrar. This data allows for a comparison of Grade Point Averages (GPA), withdrawal rates, and failure rates between the pre-LLM era (prior to Spring 2023) and the post-LLM era.
2. Instructor Surveys: A survey was distributed to faculty to document changes in course structure, assignment design, and grading criteria. Eighteen instructors completed the research survey. The survey utilized Likert scales and free-response questions to capture the extent and nature of pedagogical adaptations.
3. Student Surveys: Anonymous surveys were administered to students (primarily from computing-related programs) to assess self-reported LLM usage frequency, study behaviors, conceptual understanding, and perceptions of grade validity. To address social desirability bias, the study employed a two-pronged approach: direct questioning of personal usage and indirect questioning regarding peer usage to calculate an "underreporting index."

The study utilized thematic analysis (Braun & Clarke) for qualitative data and statistical analysis (Spearman correlations, Welch t-tests) for quantitative data. Content validity for survey instruments was established through an expert review by five university professors, achieving an overall agreement rating of 94.00% after item refinement.

Key Contributions

• Triangulated Evidence: This is the first study to simultaneously examine grade data, faculty perceptions, and student self-reports to analyze the LLM phenomenon in higher education.
• Quantification of Pedagogical Shift: The study documents the specific timeline and magnitude of course redesigns, identifying a lag between LLM availability and significant faculty adaptation.
• Underreporting Index: The research provides a methodological approach to estimating the gap between self-reported and actual LLM usage by comparing personal and peer usage estimates.
• Grade-Learning Discrepancy: The study offers empirical evidence linking the lack of course redesign to artificially inflated grades, suggesting that in unmodified courses, LLM access correlates with higher grades but potentially lower authentic learning.

Results

• Instructor Adaptation: 86.7% of responding instructors modified their courses in response to GenAI. The most common changes involved shifting toward in-class work, in-class assessments, and conceptual reasoning tasks (Figure 3). The severity of redesign increased significantly after Spring 2023, with many instructors altering 20–40% or more of their assignments.
• Grade Trends: In the post-LLM era, mean student GPA increased slightly from 2.96 to 3.00 (statistically significant but negligible effect size, Cohen's d = 0.04). Withdrawal and failure rates also decreased slightly.
• The "Unchanged" Course Anomaly: A critical finding emerged regarding the two instructors who reported no changes to their course design. Their courses showed a substantially larger increase in average GPA (+0.25 points) compared to courses taught by instructors who redesigned their assignments (+0.01 points). Withdrawal rates also declined more sharply in these unmodified courses.
• Student Behavior and Perception: Students underreported their personal LLM use for graded assignments compared to their estimates of peer use. Higher self-reported LLM use correlated with a lower belief that grades accurately reflect learning. However, in this sample, self-reported LLM use did not show a statistically significant correlation with GPA, study time, or conceptual understanding (ConCal) scores.
• Student Concerns: Students expressed significant anxiety about being punished for using AI (even when they did not) and about the fairness of the system where peers might use AI to achieve higher grades. There was a strong, unprompted demand for institutional guidance and workshops on AI use.

Significance and Claims
The authors position this work as a pilot study intended to inform broader institutional research and policy. The paper claims that its primary significance lies in providing an evidence-based assessment of learning trends using triangulated data, which is necessary for universities to navigate GenAI policy.

The study concludes that while most faculty are actively redesigning courses to mitigate AI reliance, those who have not adapted are seeing substantial grade inflation in their classes. This suggests that without pedagogical adaptation, LLM access may create a competitive disadvantage for students who do not use AI, while simultaneously obscuring the true state of student learning. The authors assert that the current data supports the need for university-level policies and student-facing guidance, noting that existing institutional resources currently focus almost exclusively on faculty. The paper explicitly avoids making broad generalizations due to its single-site, convenience-sampled nature, urging that these findings serve as a foundation for larger, multi-institutional studies.