💬 NLP

CodeGENCAT: Generative Computerized Adaptive Testing for Open-ended Coding Problems

The paper proposes CodeGENCAT, a generative computerized adaptive testing framework that leverages a Generative Item Response Theory model to predict student code responses and select questions based on coding style diversity and response uncertainty, thereby outperforming traditional baselines in assessing programming knowledge.

Original authors: Wanyong Feng, Alexander Scarlatos, Ruochen Sun, Andrew Lan

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wanyong Feng, Alexander Scarlatos, Ruochen Sun, Andrew Lan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to figure out exactly how good a student is at coding. In a traditional test, the teacher asks a question, the student answers, and the teacher marks it simply as "Right" or "Wrong." Based on that single mark, the teacher picks the next question: if the student got it right, the next one is harder; if wrong, it's easier.

The Problem with the Old Way
The paper argues that this "Right/Wrong" system throws away a lot of useful information. Imagine two students both fail a coding problem.

Student A made a tiny typo (a missing semicolon).
Student B wrote code that was completely the wrong logic.

In a traditional test, both are just "Wrong." The teacher can't tell the difference, so they can't tailor the next question to help the specific student. It's like a doctor treating a broken arm and a headache with the exact same pill just because the patient said "I feel bad."

The Solution: CodeGENCAT
The authors propose a new system called CodeGENCAT. Instead of just waiting for a student to answer, this system uses a "Digital Twin" (an AI) to predict what the student would write before the student actually answers.

Here is how it works, step-by-step, using a cooking analogy:

1. The "Digital Twin" Chef (The GIRT Model)

Imagine you want to know how good a student is at cooking. Instead of asking them to cook a whole meal right away, you have a super-smart AI chef who knows the student's current skill level.

If the student is a beginner, the AI predicts they would chop vegetables unevenly or forget to salt the soup.
If the student is an expert, the AI predicts they would use precise knife skills and perfect seasoning.

In the paper, this AI is called the Generative Item Response Theory (GIRT) model. It takes the student's estimated knowledge and generates a piece of code that looks exactly like what that specific student would write. It doesn't just guess "Right" or "Wrong"; it generates the actual code, including the specific bugs and mistakes the student is likely to make.

2. The "Menu Selection" (Question Selection)

Once the AI has predicted what the student would write, the system has to decide: "Which question should we ask the student next to learn the most?"

The paper introduces three ways to pick the next question, like a chef choosing the next ingredient to test:

The Uncertainty Chef: Picks a question where the AI is most confused about whether the student will get it right or wrong (a 50/50 guess). This is the classic "Goldilocks" zone—not too easy, not too hard.
The Diversity Chef: Picks a question where the AI thinks the student might write many different kinds of code. If the AI is unsure how the student will solve it, that question is very informative.
The Information Chef: Picks the question that, based on the predicted code, will give the biggest "shock" to the system's understanding of the student, helping to refine the skill estimate the fastest.

3. The Training (Teaching the AI)

To make sure this "Digital Twin" is accurate, the authors trained it in two stages:

Supervised Fine-Tuning (SFT): They taught the AI to look at a student's past answers and learn how to mimic them.
Direct Preference Optimization (DPO): This is the secret sauce. They realized the AI sometimes got lazy and wrote the same "perfect" code even for beginners. So, they taught the AI to be honest: "If the student is a beginner, you must generate code with mistakes." This ensures the AI's predictions are realistic and varied.

The Results

The researchers tested this on real coding datasets (Java and Python). They found that CodeGENCAT was much better at figuring out a student's true skill level than traditional methods, especially in the first few questions of the test.

The Analogy: It's like the difference between a teacher who asks, "Did you get it right?" and a teacher who says, "Let me guess exactly how you tried to solve it, and then I'll ask the perfect follow-up question to fix your specific misunderstanding."

Key Takeaways

Don't just look at the score: The paper claims that looking at the actual code (even predicted code) gives much richer information than just a "Pass/Fail" label.
Early detection matters: The system is most powerful at the beginning of a test, helping to identify a student's strengths and weaknesses much faster than old methods.
Security: The system also manages to keep the test secure, ensuring students don't all get the exact same questions, which is a common problem in adaptive testing.

In short, CodeGENCAT uses AI to simulate a student's mind, allowing the test to adapt not just to whether they got it right, but how they think, leading to a much more accurate and personalized assessment.

Technical Summary: CodeGENCAT

Problem Statement
Existing Computerized Adaptive Testing (CAT) frameworks predominantly rely on Item Response Theory (IRT) to select questions based on the predicted probability of a student answering correctly. In this paradigm, student responses are compressed into binary correctness labels (correct/incorrect). This abstraction discards rich diagnostic information inherent in open-ended responses, a limitation particularly acute in programming education. In coding tasks, two students may both fail a question, yet their code submissions can reveal vastly different levels of understanding—one might exhibit a minor syntax error while the other demonstrates a fundamental conceptual gap. Treating both as simply "incorrect" ignores the specific nature of the error, hindering the system's ability to accurately model conceptual understanding and coding ability.

Methodology
The authors propose CodeGENCAT (Generative Computerized Adaptive Testing for Open-ended Coding Problems), a framework that leverages Large Language Models (LLMs) to select questions based on predicted, open-ended student code responses rather than binary outcomes. The methodology consists of three core components:

Generative Item Response Theory (GIRT) Model:
- Knowledge Representation: Student knowledge is represented as a latent vector $z_i$ projected into explicit mastery levels ( $\theta_i$ ) for specific Knowledge Components (KCs) via a Multi-Layer Perceptron (MLP).
- Code Generation: An LLM generates predicted code responses conditioned on the question text and the student's estimated knowledge. To bridge the gap between scalar mastery levels and LLM inputs, the authors employ a "soft prompting" technique. Mastery levels are linearly interpolated between pre-trained embeddings of "TRUE" and "FALSE" to create continuous embeddings injected into the prompt.
- Training Strategy: The model undergoes a two-stage training process:
  - Supervised Fine-Tuning (SFT): The model learns to generate code conditioned on knowledge while minimizing negative log-likelihood. An auxiliary loss aligns the average predicted mastery with the ground-truth correctness score.
  - Direct Preference Optimization (DPO): To address mode collapse (where the model generates identical code regardless of input knowledge), a reinforcement learning stage is introduced. Preference pairs are formed by comparing actual student responses against responses from other students that are less likely given the current student's knowledge. DPO refines the model to ensure generated responses vary semantically based on the input knowledge state.
Question Selection Algorithms:
CodeGENCAT utilizes the GIRT model to sample $N$ potential code responses for candidate questions. Three selection criteria are proposed to determine the next question:
- Uncertainty-Based: Selects the question where the average predicted correctness score is closest to 0.5, maximizing information gain regarding correctness probability.
- Diversity-Based: Selects the question where the sampled code responses exhibit the highest semantic diversity (measured via pairwise cosine similarity of CodeBERT embeddings), indicating high uncertainty about how the student might respond.
- Fisher-Information-Based: Selects the question that maximizes the trace of the Fisher Information matrix, aiming to most significantly reduce uncertainty about the student's latent knowledge vector.
Adaptive Testing Cycle:
The system initializes a student's latent knowledge using a population prior. At each step, it samples responses from the GIRT model for candidate questions, scores them using the selection algorithm, administers the highest-scoring question, and updates the student's latent knowledge via gradient descent based on the actual response.

Key Contributions

Generative CAT Framework: The first CAT framework specifically designed for open-ended coding problems that utilizes predicted student code responses for question selection, moving beyond binary correctness.
GIRT Model: A novel model combining latent knowledge representation with LLM-driven code generation, trained with SFT and DPO to ensure generated responses align with estimated student knowledge.
Novel Selection Algorithms: Three distinct algorithms (Uncertainty, Diversity, Information) that leverage generative capabilities to evaluate question informativeness in the context of open-ended responses.

Experimental Results
The framework was evaluated on two real-world datasets: CodeWorkout (Java submissions) and ProgFeed (Python submissions).

Performance: CodeGENCAT consistently outperformed strong baselines (1PL IRT, BOBCAT, NCAT, LACAT) in estimating student knowledge, measured by Area Under the Curve (AUC) and Accuracy.
Early-Stage Gains: The most significant improvements occurred in the early stages of testing ( $t=1$ to $t=7$ ). On CodeWorkout, the Diversity-based selection algorithm achieved a 4.32% AUC improvement over the strongest baseline at the first stage.
Algorithm Variance: The optimal selection algorithm varied by dataset and test stage. Diversity performed best in early stages on CodeWorkout, while Information-based selection was consistently superior on ProgFeed and in later stages on CodeWorkout.
Test Security: CodeGENCAT maintained competitive test security metrics (question exposure and overlap rates), demonstrating it can balance efficiency with security requirements.
Ablation Studies: The study confirmed that DPO alignment is critical for the model to generate diverse, knowledge-conditioned responses. Larger backbone models (3B vs. 1B) showed dataset-dependent benefits, and reducing the number of sampled responses ( $N$ ) from 10 to 3 offered significant time savings with minimal accuracy loss.

Significance and Claims
The paper claims that CodeGENCAT represents a significant advancement in adaptive testing for programming education by exploiting the rich information contained in open-ended code. By shifting the focus from "whether a student will fail" to "how a student might fail," the framework captures specific errors and misconceptions that binary models miss. The authors assert that this approach leads to more accurate and efficient knowledge estimation, particularly during the critical early phases of testing. They note that while the current evaluation is confined to introductory computer science, the framework is theoretically adaptable to other open-ended domains, provided suitable datasets are available. The work highlights the potential of generative models to enhance psychometric modeling without discarding the nuance of student-generated content.