💬 NLP

Simulating Students' Java Programming Errors with Large Language Models

This paper demonstrates that large language models, particularly Claude Sonnet 4, can effectively simulate diverse and realistic Java programming errors from students, offering a scalable alternative to collecting authentic submissions for educational research and intelligent tutoring systems.

Original authors: Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to prepare for a big programming class. You know your students will make mistakes, but you can't predict exactly what those mistakes will be until you actually teach the class and collect their homework. This is a slow, expensive process.

This paper asks a simple question: Can we use a super-smart AI (a Large Language Model, or LLM) to pretend to be a confused student and generate those mistakes for us ahead of time?

Here is the breakdown of their experiment, explained with everyday analogies.

The Goal: The "Fake Student" Test

The researchers wanted to see if an AI could write Java code that looks exactly like a beginner student wrote it—specifically, code that has logical errors.

Logical errors are like writing a recipe that says "add salt to taste" but forgetting to actually add the salt. The computer runs the code, but the result is wrong because the thinking was flawed.
The Challenge: If you just ask an AI to "write code," it usually writes perfect code. If you ask it to "break the code," it often makes random, silly mistakes (like putting a comma where a period should be) that real students don't actually make. The goal was to make the AI make realistic mistakes.

The Experiment: The "Taste Test"

The researchers took 37 different programming problems from a real classroom (CodeWorkout). They had five different AI models try to solve these problems, but they used three different "instruction styles" (prompts) to try to force the AI to make mistakes:

Direct Order (IO): "Here is the problem, give me a wrong answer."
Thinking Aloud (CoT): "Think through the steps, decide where you might slip up, and then write the wrong code."
Self-Correction Loop (Self-Refine): "Write a wrong answer, then critique it, then fix it slightly to make it a different kind of wrong answer."

They then compared the AI's "fake mistakes" against 74,000 real mistakes made by actual students.

The Two Main Rules for a Good Fake

To be useful, the AI's fake mistakes had to pass two tests:

Diversity (The Variety Test): Did the AI make many different kinds of mistakes, or did it just repeat the same error over and over? (Imagine a chef trying to make 10 different burnt cookies. If they all look exactly the same, that's not very diverse).
Alignment (The "Realness" Test): Did the AI's mistakes look like something a human would actually do? (If the AI writes code that is wrong in a way a human would never think of, it's not a good simulation).

What They Found

1. The AI Can Make Mistakes, But Not All AIs Are Equal
All the AI models could generate a variety of errors. However, some were better at it than others.

Gemini 2.5 Pro was like a chaotic artist: it made a huge variety of mistakes (high diversity), but many of them were weird and didn't look like real student errors.
GPT-4o was like a cautious student: it made very similar mistakes over and over (low diversity).
Claude Sonnet 4 was the Goldilocks of the group. It found the best balance. It made enough different kinds of mistakes, but they were also very close to what real students actually write.

2. The "Thinking Aloud" Trick Helped
When the researchers told the AI to "think step-by-step" before writing the code (Chain-of-Thought), the mistakes became more realistic. It was as if the AI was simulating the process of a student getting confused, rather than just randomly breaking the code.

3. The "Hardness" Factor
The researchers looked at how "struggling" a problem was (based on how many times students tried to solve it).

Easy Problems: The AI did a great job mimicking students.
Hard Problems: As the problems got harder, the AI's mistakes became more diverse (it tried many different wrong paths), but they became less like real students. It's like a novice actor trying to play a complex scene; they might try many different interpretations, but none of them quite capture the nuance of a real human performance.

4. The "Blind Taste Test" (Human Experts)
The researchers hired human experts to look at a mix of real student code and AI-generated code. They didn't know which was which.

The Result: The experts were fooled 83.7% of the time. They thought the AI's code was written by a human.
The Twist: The experts actually rated the AI's mistakes as more plausible (more realistic) than the real students' mistakes. Why? Because the AI made "clean" mistakes (like a classic textbook error), while real students often make messy, complicated mistakes with multiple errors at once. The AI was like an "idealized" confused student, while real students are messier.

The Bottom Line

This paper proves that AI can act as a "stand-in" for students to generate realistic programming errors.

Claude Sonnet 4 is currently the best "student simulator."
Harder problems make the AI more creative but less accurate to real human behavior.
The Trade-off: You can have an AI that makes a huge variety of mistakes, or one that makes very realistic mistakes, but it's hard to get both perfectly at the same time.

The researchers suggest this is a powerful tool for teachers and educational software developers. Instead of waiting years to collect thousands of real student mistakes to train their systems, they can use these "fake" mistakes to build better tutoring tools and help teachers anticipate where students might get stuck.

Technical Summary: Simulating Students' Java Programming Errors with Large Language Models

Problem Statement

Understanding student errors is fundamental to programming education, particularly for identifying logical errors that reveal underlying misconceptions and flawed reasoning strategies. However, obtaining a representative set of authentic student errors for newly designed tasks is slow and costly, as it requires extensive classroom deployment and data accumulation. While Large Language Models (LLMs) offer a scalable path to simulate learner behaviors, current approaches face a critical gap: general prompting tends to yield correct solutions, while random corruption strategies produce artificial bugs lacking pedagogical value. The core challenge is indistinguishability—generating errors that are wrong in the specific ways a novice student would be wrong (reflecting partial understanding and specific misconceptions) rather than surface-level noise. Furthermore, existing evaluation infrastructure often prioritizes absolute correctness over the diversity and alignment of error patterns.

Methodology

The study leverages the CodeWorkout dataset, comprising over 74,000 unique student Java submissions across 37 introductory programming problems collected from undergraduate students at Virginia Tech.

Experimental Design

Data Preparation: The authors filtered the raw logs to retain only compilable yet incorrect code (logical errors), removing syntax errors and fully correct submissions. Problems were categorized into three struggling levels (Low, Medium, High) based on the total volume of student submissions per problem, serving as a proxy for empirical learning difficulty.
LLM Simulation: Five LLMs were evaluated: Google Gemini 2.5 Pro, OpenAI GPT-5, OpenAI GPT-4o, Anthropic Claude Sonnet 4, and xAI Grok Code Fast 1.
Prompting Strategies: Three distinct strategies were tested for each model/problem combination:
- Input-Output (IO): Direct instruction without intermediate reasoning.
- Chain-of-Thought (CoT): The model articulates a rationale for a logical slip before generating code.
- Iterative Self-Refine: An iterative loop where a critic provides feedback and a refiner rewrites the code (up to four iterations).
Evaluation Metrics:
- Diversity (RQ1): Measured via the mean pairwise Zhang–Shasha (ZSS) tree edit distance between generated erroneous codes. Higher distances indicate greater structural variation.
- Alignment (RQ2): Measured by the nearest neighbor edit distance between LLM-generated code and the closest authentic human submission. Lower distances indicate higher fidelity to real student errors.
- Qualitative Validation: A blinded expert annotation study ( $N=401$ ) compared synthetic (Claude Sonnet 4 with Self-Refine) and authentic errors. Experts rated plausibility (1–5 Likert scale), performed source attribution (Turing test), and categorized errors into eight logic-error taxonomies.

Key Results

1. Diversity (RQ1)

All models demonstrated the ability to generate diverse erroneous code, but performance varied significantly by model and strategy.

Gemini 2.5 Pro and Claude Sonnet 4 exhibited the highest diversity, with mean edit distances often exceeding 60, particularly when using CoT prompting.
GPT-4o produced the most homogeneous (least diverse) error patterns (distances ~25–40).
Prompting Impact: Strategies modulated diversity differently; for instance, Self-Refine reduced diversity in Claude Sonnet 4 but increased it in other contexts.

2. Alignment (RQ2)

The ability to mimic authentic student errors varied by model, with a clear trade-off against diversity.

Claude Sonnet 4 achieved the most balanced performance, maintaining relatively low edit distances (best case: 16.35 with Self-Refine) across all strategies, indicating strong alignment with human errors.
Grok Code Fast 1 achieved the absolute minimum distance (16.22) with Self-Refine but showed high variance across other prompts.
GPT-5 and Gemini 2.5 Pro showed poor alignment, with distances often exceeding 80, suggesting their errors diverge significantly from authentic student patterns.
Human Evaluation: Annotators misclassified 83.7% of LLM-generated submissions as human-written. Surprisingly, LLM-generated errors received a significantly higher plausibility score (4.27) than authentic errors (3.78). This is attributed to the LLMs generating "clean," single-purpose logical bugs (e.g., Condition Logic, Boundary errors) that resemble textbook misconceptions, whereas authentic student code often contained diffuse, multi-fault, or strategy-level errors that were harder to categorize cleanly.

3. Struggling Level as a Moderator (RQ3)

The difficulty of the programming task (struggling level) influenced both diversity and alignment.

Diversity: Higher struggling levels elicited more diverse (heterogeneous) error outputs from LLMs, as harder problems open up more failure paths.
Alignment: Conversely, as the struggling level increased, the alignment between LLM-generated and authentic errors decreased (edit distances increased). Models found it harder to reproduce student-like logical errors on high-struggle problems, diverging further from authentic behavior.

Significance and Contributions

The paper claims that LLMs can serve as scalable proxies for students in generating logical errors, but with specific trade-offs that must be managed based on the application goal:

Balanced Simulation: Claude Sonnet 4 is identified as the most balanced model, offering a compromise between generating diverse error patterns and maintaining high fidelity to authentic student mistakes.
Pedagogical Utility: The study validates that synthetic errors can be functionally indistinguishable from authentic ones to experts. This supports the integration of synthetic data into:
- Intelligent Tutoring Systems (ITS): To train systems to diagnose diverse misconceptions without relying solely on scarce human data.
- Teachable Agents: To allow learners to practice debugging by correcting AI-generated errors, fostering vicarious learning.
- Teacher Professional Development: To expose educators to a concentrated variety of anticipated misconceptions for better instructional planning.
Design Considerations: The findings highlight a diversity–fidelity trade-off. Models strong in diversity (e.g., Gemini) may lack alignment, while those with high alignment may be less diverse. Furthermore, the "struggling level" of a task acts as a dual moderator: it increases the breadth of simulated errors but reduces their representativeness of actual student behavior.

The authors conclude that while LLMs offer a promising avenue for augmenting student data in educational technology, their use requires careful selection of models and prompting strategies tailored to whether the goal is broad misconception coverage or the precise approximation of observed classroom error distributions.