Changes in Coding Behavior and Performance Since the Introduction of LLMs

Technical Summary: Changes in Coding Behavior and Performance Since the Introduction of LLMs

Problem Statement
The rapid integration of Large Language Models (LLMs) and agentic coding tools into software development workflows has fundamentally altered how students engage with programming tasks. While these tools promise increased productivity, they introduce significant challenges for educators: distinguishing between genuine student understanding and AI-generated output, and assessing whether learning objectives are being met when the "mechanical burden" of coding is offloaded. Existing literature offers conflicting accounts regarding LLMs' impact on learning, ranging from enhanced tutoring effects to harms caused by over-reliance. There is a lack of longitudinal, empirical evidence quantifying how coding behaviors and learning outcomes have evolved specifically before and after the mass adoption of LLMs.

Methodology
This study employs a quasi-longitudinal design analyzing source code submissions from a graduate-level Cloud Computing course at Carnegie Mellon University. The dataset spans 10 semesters (Fall 2020 to Spring 2025), covering five semesters prior to the release of ChatGPT (November 2022) and five semesters post-release.

• Data Source: The analysis focuses on 2,066 submissions from 721 unique student enrollments.
• Task Selection: The primary focus is a specific, static assignment (the "PageRank task") within an individual project. This task was chosen because its specification and starter code remained unchanged from 2020 to 2025, allowing for direct comparison of solutions. It involves implementing the PageRank algorithm on a social graph with minimal starter code (60 lines).
• Metrics Defined:
  • Coding Behavior: Number of submissions (attempts), Total Edit Distance (sum of line changes between consecutive submissions using Myers algorithm), and Average Edit Distance.
  • Performance: Task Score (auto-graded score of the final PageRank submission), IP Score (sum of top 5 individual project scores), and TP Score (Team Project scores).
  • Learning Efficiency: "Average Submission Effect," defined as the average change in task score between consecutive submissions divided by the number of submissions.
• Constraints: Collaboration and AI-assisted coding were prohibited for individual projects. The study relies on behavioral proxies (edit distances and submission patterns) rather than self-reported LLM usage, acknowledging that students may under-report misuse.

Key Results
The study identifies statistically significant shifts in coding behavior and performance correlations following the introduction of LLMs:

1. Behavioral Changes (Post-2022):

   • Increased Submission Volume: The median number of submissions per student increased by 50% (from 2 to 3 attempts).
   • Expanded Code Volume: The median Total Edit Distance increased by 500%, and the median Average Edit Distance increased by 300%. This suggests students are making larger, less incremental changes between submissions.
   • Code Bloat: Final submissions for perfect scores have significantly more lines of code (as illustrated in Figure 1), consistent with LLMs' tendency to generate unrequested formatting, documentation, or boilerplate.

2. Performance Outcomes:

   • Task Score: Remained stable, with nearly all students achieving a perfect score on the PageRank task in all semesters, indicating the task itself may be too simple to differentiate performance.
   • Individual Project (IP) Score: Showed a slight decrease in the average score post-ChatGPT.
   • Team Project (TP) Score: Showed a moderate increase, potentially suggesting AI acts as an "unofficial team member" aiding competent individuals.

3. Correlations and Efficiency:

   • Declining Learning Efficiency: The "Average Submission Effect" (score improvement per edit) decreased significantly starting in Spring 2024. Students are making more edits but achieving smaller score gains.
   • Negative Correlation: There is a statistically significant negative correlation ($corr = -0.16, p < 0.001$) between Average Edit Distance and IP Score. For every additional 60 lines of average edit distance, IP scores decreased by approximately 1%.
   • Shift in Team Dynamics: Pre-ChatGPT, higher edit distances correlated with lower team scores; this correlation disappeared post-ChatGPT, suggesting LLMs may be masking individual learning gaps within team settings.

Key Contributions

• Empirical Evidence: This is the first comprehensive study providing longitudinal data on coding behavior changes across 10 consecutive semesters, specifically isolating the period before and after LLM proliferation.
• Behavioral Proxies: The paper establishes that increased edit distances and submission counts, coupled with reduced score improvement per edit, serve as strong behavioral indicators of potential LLM over-reliance, even in the absence of direct usage logs.
• Decoupling Performance from Learning: The study demonstrates that high assignment scores (Task Score) no longer guarantee learning efficiency. Students can achieve perfect scores with significantly more "bloated" code and less strategic iteration, masking a decline in genuine problem-solving skills.

Significance and Claims
The authors claim that the observed behavioral shifts are consistent with the hypothesis that students are over-reliant on LLMs, which is negatively affecting their learning outcomes despite stable or improved surface-level performance metrics.

• Educational Alarm: The findings raise concerns regarding the first generation of graduates entering the workforce in the age of LLMs. The data suggests a potential decline in the ability to write efficient, incremental code and a reliance on tools that generate excessive, unrequested changes.
• Assessment Misalignment: Current evaluation methods may be misaligned with the skills required in an AI-augmented world. The paper argues that metrics focusing solely on final output (scores) fail to capture the degradation in the process of learning.
• The "Centaur" Model: The paper concludes by proposing a shift in educational philosophy. Drawing on the "centaur" model from chess (human + machine outperforming either alone), it suggests that the role of the human developer is shifting from tactical code production to strategic oversight (defining abstractions, integrating components, and critically evaluating output). The authors call for educational curricula and assessments to evolve to measure these strategic competencies rather than just the ability to produce code in isolation.

The paper remains modest in its claims, acknowledging that it cannot definitively prove LLM misuse for every student due to the lack of direct usage data and the possibility of other factors (e.g., solution sharing). However, the statistical trends strongly suggest a systemic change in how programming tasks are approached and learned.