Teaching Language Models How to Code Like Learners: Conversational Serialization for Student Simulation

1. Problem Statement

Current educational research relies heavily on models to simulate student behavior for evaluating tutoring strategies and feedback mechanisms at scale. However, existing approaches in programming education face significant limitations:

• Reliance on Proprietary Models: Many methods use prompting on large, closed-source Large Language Models (LLMs), raising concerns regarding privacy, cost, and data dependency.
• Superficial Simulation: Prompted LLMs often generate "expert-like" code or surface-level error patterns rather than capturing the iterative debugging process of real learners. They tend to jump to correct solutions prematurely, failing to simulate the struggle and incremental progress characteristic of students.
• Lack of Contextual Feedback: Prior work often ignores the specific feedback loop between a student's code submission and the automated assessment system (e.g., test results, error traces), which is crucial for realistic debugging simulation.

The authors aim to train open-weight, small-to-medium-sized language models that can authentically replicate how students interact with learning environments, specifically focusing on the iterative process of debugging based on automated feedback.

2. Methodology

The core innovation is a Conversational Serialization approach that transforms raw log data into a dialogue format suitable for training chat-capable LLMs.

A. Data Serialization (Logs to Dialogs)

The authors convert student interaction logs into a structured dialogue where:

• System Prompt: Defines the persona as a novice student learning Python.
• User Role: Represents the Automated Assessment System (providing assignment descriptions and feedback like test outcomes, grades, or runtime errors).
• Assistant Role: Represents the Student (providing code submissions).
• Structure: Each interaction is an alternating turn:
  1. User: Assignment description or feedback on the previous code.
  2. Assistant: Student's code submission.
  3. User: Feedback on that submission (e.g., "Tests passed: 1/8" or "Runtime error").
  4. Assistant: Revised code.

This format allows the model to learn the causal relationship between code changes and environment feedback without architectural modifications.

B. Training Pipeline

The authors propose a three-stage training pipeline using Qwen models (4B and 8B parameters):

1. Supervised Fine-Tuning (SFT):

   • The model is trained on the serialized dialogues using Negative Log Likelihood (NLL).
   • Loss is calculated only on the "Assistant" turns (student code) to teach the pattern of problem-solving.

2. Offline Preference Optimization (DPO):

   • Goal: Align the model with authentic student trajectories and prevent premature convergence to correct solutions.
   • Mechanism: Constructs preference pairs from a single student trajectory.
     • Preferred ($y_w$): The student's actual next submission ($a_{t+1}$).
     • Dispreferred ($y_l$): The next submission that resulted in a different grade (a semantically distinct attempt).
   • This teaches the model that students often make specific, non-optimal changes before finding a solution, rather than jumping straight to the answer.

3. Online Preference Optimization (DAPO/GRPO):

   • An alternative approach using Group Relative Preference Optimization (GRPO).
   • The model samples multiple candidate next steps, scores them based on functional alignment (matching the student's grade) and syntactic similarity (matching the student's code structure), and updates weights based on relative quality.
   • Rewards are tiered: +2.0 for exact AST match, +1.0 for same grade, -1.0 for compilation failure.

3. Key Contributions

1. Student-Environment Serialization: A novel method to represent student debugging as a multi-turn dialogue between a learner and an automated grader, enabling LLMs to learn iterative debugging processes.
2. Training Pipeline for Artificial Students: A framework combining SFT with both offline (DPO) and online (DAPO) preference optimization to align models with authentic student behavior.
3. Empirical Validation: Extensive evaluation on the FalconCode dataset (US Air Force Academy, Python CS1), demonstrating that incorporating environment feedback and preference optimization significantly outperforms prompting and code-only baselines.
4. Open Release: The authors released their code and methodology to support reproducibility.

4. Experimental Results

The models were evaluated on a test set of 1,000 trajectories from the FalconCode dataset using three metrics: Coverage (ability to generate a submission at every step), Grade Proximity (functional alignment with student grades), and CodeBLEU (syntactic similarity).

• Baselines: Prompted models (including GPT-5-mini and untrained Qwen) performed poorly. They frequently generated fully correct solutions immediately (low coverage) or produced code with little resemblance to student errors (low CodeBLEU).
• Impact of Feedback: Models trained with environment feedback (SFT) significantly outperformed those trained on code history alone (PARA), proving that feedback is more informative than additional code context.
• Preference Optimization:
  • DPO (Offline): Achieved the best overall performance. It yielded the highest coverage (0.982 for 4B, 0.957 for 8B) and highest grade proximity, effectively simulating the gradual improvement of real students.
  • DAPO (Online): Showed mixed results. While it matched DPO on grade proximity for the 8B model, it had lower coverage and did not consistently improve over SFT. The authors hypothesize this is due to computational constraints (sampling fewer prefixes) and the lack of explicit chain-of-thought reasoning in the training data.
• Degradation: DPO models maintained high performance across multiple rollout steps (up to 5 steps ahead), whereas baseline models degraded rapidly.

5. Significance and Future Directions

• Scalable Simulation: This work provides a practical, privacy-preserving method to create "artificial students" using open-weight models and local data, enabling large-scale evaluation of tutoring strategies without repeated classroom deployments.
• Beyond Knowledge Tracing: Unlike traditional Knowledge Tracing (which estimates what a student knows), this approach models how a student behaves (the process of debugging, failing, and revising).
• Tutoring Applications: The conversational format naturally bridges the gap between student simulation and tutoring. The same dialogue structure could be extended to include a "Tutor" turn, allowing for the simulation of student-tutor interactions to train reinforcement learning-based tutors.
• Limitations: The study was limited to Python and a single dataset (FalconCode) with a high success rate (95% of trajectories ended in perfect scores). Future work needs to address struggling students, different programming languages, and the integration of prior assignment history for personalization.

In conclusion, the paper demonstrates that conversational serialization combined with preference optimization is a superior method for training AI agents to mimic the realistic, iterative, and often imperfect debugging behaviors of human learners.