💬 NLP

KASER: Knowledge-Aligned Student Error Simulator for Open-Ended Coding Tasks

This paper introduces KASER, a novel reinforcement learning-based approach that aligns error generation with student knowledge to effectively simulate diverse and accurate student coding errors, outperforming existing baselines in both error prediction and code diversity on real-world datasets.

Original authors: Zhangqi Duan, Nigel Fernandez, Andrew Lan

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhangqi Duan, Nigel Fernandez, Andrew Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to grade a stack of homework assignments. You know that every student makes mistakes, but those mistakes aren't random. A student who doesn't understand how to use "if" statements will make different errors than a student who just forgot to put a semicolon at the end of a line.

For a long time, computers (specifically Large Language Models or LLMs) have been great at writing perfect code, like a genius tutor. But they are terrible at pretending to be a struggling student. When asked to "act like a confused student," these computers usually just write perfect code again, or they make the same boring mistake over and over. They suffer from what the authors call "mode collapse"—imagine a comedian who only tells one joke because they think it's the only way to be funny.

The paper introduces KASER (Knowledge-Aligned Student Error Simulator), a new method designed to fix this. Think of KASER as a method actor for computer code. Instead of just guessing what a student might write, KASER studies the student's "report card" first.

Here is how it works, broken down into simple steps:

1. The "Report Card" (Knowledge Estimator)

Before the computer writes any code, it looks at the student's history. It calculates a "mastery score" for different concepts, like a report card.

Analogy: Imagine a coach looking at a player's stats. If the player is bad at "passing" but good at "running," the coach knows exactly what kind of mistakes to expect during a game. KASER does this for coding concepts (like logic or math).

2. The "Method Acting" (The Simulator)

Once KASER knows the student's weak spots, it uses a special training technique called Reinforcement Learning (think of it as training a dog with treats). It tries to generate code that looks like it came from that specific student.

To make sure the computer is actually doing a good job, the authors gave it a three-part reward system (a "hybrid reward"):

Reward 1: The Look-Alike (Similarity): The code should look somewhat like the real student's code (same style, same structure).
Reward 2: The Mistake Match (Error Matching): This is the most important part. If the real student forgot to close a bracket, the simulator must also forget to close a bracket. If the real student got the logic wrong, the simulator must get the logic wrong. It's not enough to just write bad code; it has to write the right kind of bad code for that specific student.
Reward 3: The Variety Pack (Diversity): To stop the computer from getting bored and writing the same mistake 100 times, the system rewards it for trying different ways to make the same mistake. This ensures the simulation feels like a real classroom with many different students, not a robot repeating a loop.

3. The Results

The authors tested KASER on real data from thousands of actual student coding submissions.

The Verdict: KASER was much better than previous methods at predicting what errors a specific student would make.
The "Mode Collapse" Fix: Unlike other models that kept generating perfect code or the exact same error, KASER produced a wide variety of realistic, flawed code that matched the student's knowledge level.

What the Paper Doesn't Claim

It is important to stick to what the paper actually says:

It is not a replacement for teachers. The paper suggests this tool helps teachers understand student errors better, but it doesn't claim to replace human instruction.
It doesn't fix the errors yet. The paper focuses on predicting and simulating the errors, not on automatically fixing them or teaching the student how to fix them (though the authors mention this as a future possibility).
It's not perfect at syntax. The authors admit that while KASER is great at simulating logical mistakes (like bad math or wrong steps), it still struggles to simulate basic "syntax" errors (like missing commas or typos) because the underlying computer brain is trained to write perfect code and finds it hard to "forget" how to be perfect.

In summary: KASER is a tool that teaches a computer to "act" like a specific student by studying their report card. It uses a special reward system to ensure the computer makes the right kind of mistakes, helping educators see exactly where a student is likely to stumble before they even submit their homework.

Technical Summary: KASER (Knowledge-Aligned Student Error Simulator)

Problem Statement

Open-ended coding tasks are ubiquitous in computer science education and provide rich data regarding student knowledge. However, training Large Language Models (LLMs) to simulate student responses—specifically to predict the errors students will make—presents significant challenges. Existing approaches often suffer from mode collapse, where models generate overly uniform or correct code, failing to capture the high diversity in syntax, style, and solution approaches found in real student submissions. Furthermore, many existing methods lack an explicit link to student knowledge, making them less pedagogically useful for diagnosing specific misconceptions. Current state-of-the-art methods either rely on supervised fine-tuning (SFT) which tends to generate correct code, or use in-context learning and persona-based prompting which struggle to generalize error patterns to individual students.

Methodology: KASER

The authors propose KASER (Knowledge-Aligned Student Error Simulator), a framework designed to align simulated student errors with their underlying knowledge profiles. The methodology consists of three primary components:

1. Student Knowledge Estimator (KE)

KASER first employs a Knowledge Tracing (KT) model to estimate a student's mastery over a set of Knowledge Components (KCs).

Input: A history of a student's prior code submissions.
Process: The model compresses the student's knowledge state into a mastery vector $m_t \in [0, 1]^k$ , where each dimension represents the mastery level of a specific KC.
Output: An interpretable knowledge profile used to guide the code generation.

2. Student Code Predictor

A large language model (specifically Qwen2.5-Coder 7B Instruct) is trained to predict the next code submission ( $c_{t+1}$ ) given the problem statement ( $p$ ) and the student's mastery vector ( $m_t$ ).

Prompting: The prompt includes the problem text and the specific mastery levels for the KCs associated with that problem.
Initial Training: The model undergoes Supervised Fine-Tuning (SFT) to learn the baseline task of code generation conditioned on knowledge.

3. Reinforcement Learning with Hybrid Reward (GRPO)

To overcome mode collapse and align errors with knowledge, KASER utilizes Group Relative Policy Optimization (GRPO). Instead of standard SFT, the model is further trained using a hybrid reward function $R$ composed of three distinct components:

Code Similarity Reward ( $R_{Sim}$ ): Measures the syntactic and semantic similarity between the generated code and the ground-truth student code using CodeBLEU.
Error Match Reward ( $R_{Error}$ ): Explicitly encourages the model to predict errors that match those in the ground-truth code. This is calculated using the Intersection over Union (IoU) of the error sets (syntax, runtime, and logical errors) identified by an LLM-as-a-judge. This component prioritizes matching the type of error over mere surface similarity.
Code Diversity Reward ( $R_{Div}$ ): A group-level reward that penalizes the model if generated codes within a batch are too similar to each other. This is calculated as $1 - \max(\text{CodeBLEU})$ against other samples in the group, preventing mode collapse and encouraging diverse solution approaches.

Error Annotation Pipeline

To train and evaluate the error matching capability, the authors developed an automated error annotation pipeline:

Generation: An LLM (o4-mini) generates error annotations for student code, covering syntax, runtime, and logical errors.
Clustering: Error descriptions are clustered using Sentence-BERT embeddings and Hierarchical Agglomerative Clustering to group similar errors.
Summarization: A representative, generalized error label is generated for each cluster.
Human evaluation confirmed substantial agreement (F1 $\approx$ 0.75–0.92) between the LLM judge and human annotators.

Key Contributions

Knowledge-Aligned Simulation: The development of a GRPO-based method that explicitly aligns predicted student errors with input student knowledge profiles, moving beyond surface-level code prediction.
Hybrid Reward Design: The introduction of a three-part reward function that balances code similarity, error overlap (IoU), and generation diversity to prevent mode collapse.
Comprehensive Evaluation: Extensive quantitative evaluation at two levels:
- Per-Student-Problem Pair: Evaluating prediction accuracy and error matching for specific student instances.
- Per-Problem: Evaluating the model's ability to cover a wide range of errors and generate diverse code across a population.

Results

Experiments were conducted on two real-world datasets: CodeWorkout (Java) and FalconCode (Python).

Per-Student-Problem Level: KASER outperformed all baselines (including PersonaPrompt, In-Context Learning, ParaStudent, and Student SFT) across all metrics. Notably, it achieved a statistically significant improvement in Error Match (IoU) and Code Similarity (CodeBLEU). For instance, on CodeWorkout, KASER achieved an IoU@1 of 0.157 compared to 0.115 for the strongest baseline (Student SFT).
Per-Problem Level: KASER demonstrated superior Error Coverage and Code Diversity. It successfully predicted a wider range of unique errors and generated more diverse code solutions compared to baselines, which often converged to correct solutions or repetitive error patterns.
Qualitative Analysis: Case studies showed that KASER could simulate specific error types (e.g., incorrect comparison operators, missing return statements) corresponding to low mastery levels in specific KCs, whereas baselines like Student SFT tended to generate correct code regardless of the simulated knowledge profile.

Significance and Claims

The paper claims that KASER represents a significant step forward in educational AI by enabling the simulation of student errors that are interpretable and aligned with specific knowledge deficiencies.

Pedagogical Utility: By aligning errors with knowledge components, the system can help instructors diagnose student misconceptions more accurately and provide personalized feedback.
Overcoming Mode Collapse: The hybrid reward approach effectively mitigates the tendency of LLMs to generate only correct or uniform code, a critical limitation in simulating the "struggle" inherent in learning.
Limitations: The authors modestly note that while the model excels at logical and runtime errors, it still struggles to generate certain syntax errors (e.g., unbalanced braces), likely due to the pre-training bias of LLMs toward syntactically valid code. They also acknowledge that their error annotation relies on LLMs without access to test cases, which may limit the granularity of error categorization.

In conclusion, KASER demonstrates that integrating explicit knowledge tracing with reinforcement learning and diversity-promoting rewards allows LLMs to better simulate the nuanced and diverse error patterns of human students in open-ended coding tasks.