💻 computer science

A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming

This paper presents a large-scale classroom study introducing the ProgFeed dataset to demonstrate that natural language AI-generated feedback significantly improves student completion rates and solution convergence in introductory programming compared to test case feedback or no feedback, highlighting the critical importance of feedback form and validity over mere presence.

Original authors: Hasnain Heickal, Andrew Lan

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hasnain Heickal, Andrew Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are learning to cook a new, complicated recipe for the first time. You try it, and it tastes terrible. You need to know why it failed so you can fix it.

This paper is about a big experiment in a computer science classroom where students were learning to "cook" (write code) using a special AI chef (a Large Language Model) to help them. The researchers wanted to see which type of "chef's advice" actually helped students fix their dishes the fastest and most successfully.

Here is the breakdown of what they did and what they found, using simple analogies:

The Setup: Three Types of Chefs

The students were randomly assigned to one of three groups when they made a mistake in their code:

The Silent Group (No AI): These students just saw a red light saying "Wrong" and a green light saying "Right" (like a simple pass/fail grade). They had to figure out the rest on their own.
The "Bad Ingredient" Group (Test Case Feedback): The AI gave them a specific example of a "bad ingredient" that caused the dish to fail. For example, if the code was supposed to add numbers, the AI might say, "If you put in 5 and 5, you get 12 instead of 10." It's like showing them a specific ingredient that ruined the soup, but not telling them how to fix it.
The "Hint" Group (Natural Language Feedback): The AI gave them a short, friendly sentence explaining the problem. For example, "You forgot to tell the computer to add the numbers together." It's like a chef whispering, "Hey, you missed a step in the instructions."

The Results: What Worked Best?

1. The "Hint" Group Won
Students who got the friendly, natural language hints were much more likely to eventually get the recipe right, and they got there faster.

The Analogy: Think of the code as a maze. The "Hint" group was given a map that said, "You're going the wrong way; turn left here." This helped them find the exit quickly.

2. The "Bad Ingredient" Group Struggled
Surprisingly, giving students a specific example of what went wrong (the "Bad Ingredient" group) didn't really help them overall. In fact, they didn't get better results than the group that got no help at all.

The Analogy: Imagine the AI handed the student a burnt piece of toast and said, "This is why your breakfast failed." The student looked at the burnt toast, but because they are beginners, they didn't know how to stop the toaster from burning the next slice. They kept trying different things randomly, often making the same mistake again.

3. The Quality of the "Bad Ingredient" Matters
The researchers found that the "Bad Ingredient" examples were only useful about two-thirds of the time. Sometimes the AI gave an example that was actually wrong or didn't make sense. When the example was correct, it was still hard for beginners to use it effectively.

The Big Takeaway

The main lesson from this study is that how you give feedback matters just as much as giving feedback.

Natural language hints act like a guide, translating the confusing computer errors into plain English that a beginner can understand.
Raw data examples (like test cases) are like handing someone a broken engine part without a manual. If the student doesn't already know how to fix engines, the broken part just confuses them.

The researchers concluded that for beginners, a simple, clear explanation is far more powerful than a complex technical example. They also released all the data from this experiment (called PROGFEED) so other researchers can study how students learn and how to build better AI teachers in the future.

In short: If you are teaching a beginner, don't just show them the mistake; explain why it happened in plain words. That's the key to helping them succeed.

Technical Summary: A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming

Problem Statement

While Large Language Models (LLMs) are increasingly deployed to provide automated feedback in introductory programming courses, there is a scarcity of empirical evidence derived from authentic classroom settings that compares different feedback modalities. Existing large-scale programming datasets (e.g., CodeNet) often lack controlled feedback variation, making it difficult to disentangle the effects of feedback from confounding factors like student ability or problem difficulty. Conversely, longitudinal datasets with rich interaction traces (e.g., Blackbox) typically lack experimental manipulation. This study addresses the gap by investigating how different forms of AI-generated feedback—specifically natural language hints versus failing test cases—affect student performance, learning trajectories, and submission behavior in an introductory Python course.

Methodology

The authors conducted an IRB-approved classroom study during the Fall 2025 semester at a U.S. R1 institution involving 215 consenting students from an introductory Python course.

Experimental Design: The study utilized a randomized protocol where feedback was assigned at the submission level for incorrect submissions on laboratory assignments. Students were randomly assigned to one of three conditions:
1. Group 0 (No AI Feedback): Received only fixed test case pass/fail verdicts.
2. Group 1 (Failing Test Case Feedback): Received an AI-generated input–output pair designed to fail the student's specific submission.
3. Group 2 (Natural Language Feedback): Received a concise natural language hint identifying a likely issue, without code suggestions or test cases.
Feedback Generation: All feedback was generated by gpt-4o-mini via the OpenAI API using fixed prompts tailored to the problem description and the student's code.
Data Collection: The resulting dataset, PROGFEED, captures 6,693 submissions across 17 labs and 43 problems. It includes source code, submission order, execution outcomes, feedback conditions, and temporal data.
Analysis: The study employed associative analyses (logistic regression and Cox proportional hazards models) to examine learning dynamics, as feedback was conditionally triggered by incorrect submissions rather than fully randomized. Metrics included the change in test case pass rate ( $\Delta$ pass rate), probability of eventual success, and time to correctness (measured in submissions).

Key Contributions

Controlled Classroom Study: A large-scale deployment enabling the controlled comparison of natural language, failing test case, and no-AI feedback conditions in an authentic educational setting.
PROGFEED Dataset: A longitudinal dataset linking complete submission trajectories to specific feedback conditions and execution outcomes, supporting fine-grained analysis of learning dynamics.
Empirical Analysis of Feedback Effectiveness: Rigorous statistical analysis demonstrating how feedback modality influences completion rates, convergence speed, and the quality of the feedback itself.

Results

Learning Dynamics and Performance

Short-Term Dynamics: Immediate improvements following feedback ( $\Delta$ pass rate) were neither consistent nor guaranteed across any condition. Mean changes fluctuated near zero, with high variability in later submission attempts.
Eventual Success: Natural language feedback was significantly associated with a higher probability of students eventually reaching a correct solution. The logistic regression showed an odds ratio of approximately 2.33 ( $p=0.004$ ) compared to the no-AI baseline. Test case feedback showed no significant association with eventual success ( $p=0.40$ ).
Time to Correctness: Survival analysis (Cox proportional hazards) indicated that natural language feedback was associated with a modest but significant increase in the hazard of reaching correctness ($HR=1.09, p=0.03$), meaning students converged to correct solutions faster. Test case feedback showed a slightly lower hazard ($HR=0.92, p=0.05$), suggesting slower convergence compared to the baseline.

Feedback Quality and Validity

Test Case Validity: Only 66% of AI-generated test cases were "valid" (failing the student's code but passing the reference solution). 24% were false positives (failing both), and 10% had no effect. Valid test cases correlated with higher pass rates but did not reduce the number of subsequent submissions, suggesting introductory students lack the systematic debugging skills to efficiently exploit counterexamples even when they are correct.
Natural Language Quality: An evaluation of 442 natural language hints using GPT-4 as a judge found that 92.5% were labeled "helpful." The remaining instances were mostly "vague" (5.4%) or "incorrect" (2.0%). Crucially, no feedback instance explicitly revealed the solution or bypassed student reasoning.

Significance and Claims

The paper argues that the form of AI-generated feedback is a critical determinant of pedagogical impact, not merely a superficial design choice.

Cognitive Load: The superiority of natural language hints is attributed to a reduction in cognitive load; novice learners struggle to interpret raw test failures, and natural language hints translate program behavior into accessible explanations, lowering the barrier to identifying fixes.
Feedback Quality vs. Presence: The study emphasizes that evaluating feedback quality is as essential as evaluating its presence. The heterogeneous effects of test case feedback depend critically on validity, and even valid feedback may not improve efficiency if the learner lacks the requisite debugging skills.
Pedagogical Implications: The results suggest that effective feedback may improve correctness without necessarily reducing the number of submission attempts. Therefore, evaluations of AI-assisted feedback should focus on learning trajectories and feedback quality rather than aggregate success rates or iteration counts alone.

The authors conclude that while natural language feedback shows promise for improving completion rates and convergence speed, the deployment of test case feedback requires careful consideration of feedback validity and learner readiness. The release of PROGFEED aims to support further research into learning dynamics and the responsible deployment of AI in education.