💬 NLP

Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models

This paper introduces a benchmark-oriented framework for using large language models to simulate students with controllable, partial skill mastery to support teacher education, demonstrating that while selective retention and suppression of competencies are achievable, the degree of control remains dependent on the specific model used.

Original authors: Alexander Apartsin, Omri Sason, Yehudit Aperstein

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Apartsin, Omri Sason, Yehudit Aperstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a new teacher. To get good at their job, they need to practice with students who make specific mistakes, forget specific things, and struggle with particular concepts. They can't just practice with perfect geniuses; they need to see how to handle a student who knows how to add but has forgotten how to subtract.

This paper asks a big question: Can we use a super-smart AI (a Large Language Model) to pretend to be a "flawed" student on command?

Here is the breakdown of what the researchers did, using some simple analogies.

The Goal: The "Chameleon" Student

Usually, when we ask an AI to solve math problems, we want it to be perfect. We want it to get an A+. But for teacher training, we need the AI to be a "C-minus" student who specifically forgot how to handle fractions but still remembers how to do geometry.

The researchers wanted to see if they could tell an AI: "Pretend you are a student who knows Skill A and Skill B, but you have completely forgotten Skill C."

The Challenge: Why It's Hard

Think of an AI like a library that has read every math book ever written. It knows everything.

The Problem: If you ask the library to "forget" how to do fractions, it's hard to make it stop using that knowledge. It's like trying to tell a person who loves chocolate to suddenly hate it just because you asked nicely. The knowledge is "entrenched" deep inside.
The Fear: If you tell the AI to forget fractions, will it accidentally forget how to do geometry too? Or will it just start guessing randomly? The researchers wanted to know if they could surgically remove only the specific skill they wanted the AI to "forget."

The Experiment: The "Menu" of Skills

The researchers built a test using math problems for 4th and 5th graders. They treated math topics (like "Fractions" or "Geometry") as items on a menu.

The Profile: They created a "student profile" for the AI. This was a list where they checked "Yes" for skills the AI should keep and "No" for skills the AI should forget.
The Instructions: They tried three ways to tell the AI what to do:
- Just Telling: "You are a student who forgot fractions."
- Just Showing: Giving the AI examples of a student making mistakes.
- The Hybrid (The Winner): Telling the AI what to forget AND showing it how to make those specific mistakes.

The Results: What Worked?

The researchers found that the AI could indeed pretend to be a flawed student, but it depended heavily on how they asked and which AI they used.

The "Hybrid" Approach Won: Just telling the AI to forget something wasn't enough. Just showing examples wasn't enough. But when they did both (The Hybrid method), the AI got much closer to the "flawed student" role. It was like giving a method actor both a script and a backstory; the performance became much more convincing.
Not All AIs Are Equal: They tested three different AI models (Claude, DeepSeek, and GPT-4o).
- Claude was the best "actor." It could follow the instructions to forget specific skills very well without messing up the skills it was supposed to keep.
- DeepSeek was very good at math but harder to control. It kept trying to solve the problems correctly even when told to fail.
- GPT-4o was somewhere in the middle.
No "Collateral Damage": Good news! When the AI was told to forget "Fractions," it didn't accidentally forget "Geometry." The forgetting was localized. It was like turning off the lights in the kitchen without turning off the lights in the bedroom.

The Bottom Line

This paper doesn't say we have built a perfect virtual student for a classroom yet. It says something more specific: We have proven that we can build a "benchmark" to test if an AI can selectively forget things.

They created a way to measure if an AI can be "steered" to be imperfect in a controlled way. This is a crucial first step. Before we can use AI to train teachers, we first have to prove that the AI can actually pretend to be a student with specific weaknesses, rather than just being a perfect robot that refuses to make mistakes.

In short: The researchers built a test to see if they could teach an AI to "play dumb" on specific topics. They found that with the right instructions, the AI can do it, but it's a tricky act that depends on which AI you use.

Technical Summary: Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models

1. Problem Formulation

The central problem addressed is the controllable simulation of imperfect students using Large Language Models (LLMs) for teacher deliberate practice. Current teacher training often lacks access to standardized, repeatable interactions with learners who exhibit specific, known partial mastery (e.g., retaining some skills while forgetting others).

The paper identifies a fundamental misalignment between standard LLM evaluation and educational simulation goals:

Standard Paradigm: Maximizes benchmark accuracy (performance-first).
Simulation Requirement: Specifies a behavior profile (specification-first). The goal is not a uniformly capable assistant, but a model that reliably retains specific curricular skills while suppressing others.

The authors distinguish this from "fact forgetting" (machine unlearning of discrete facts) and "persona control" (narrative consistency). Instead, they target skill-profile control: the ability to induce a coherent pattern of partial mastery across related task families (e.g., solving linear equations) while minimizing collateral damage to untargeted skills. The core research question is whether prompted LLMs can be steered to behave like students with specified skill vectors (retained vs. forgotten) with measurable fidelity.

2. Methodology

2.1 Benchmark Design: MathCAMPS

The study utilizes MathCAMPS, a curriculum-aligned dataset based on U.S. Common Core standards.

Structure: Eight grade-level datasets (Grades 1–8), each containing 100 multiple-choice questions balanced across skill domains.
Granularity: Detailed analysis focuses on Grades 4 and 5.
- Grade 4: 4 skill families (Measurement & Data; Number & Operations Base; Fractions; Operations & Algebraic).
- Grade 5: 3 skill families (Number & Operations Base; Fractions; Operations & Algebraic).
Item Construction: Open-format math problems were converted to four-option multiple-choice items with conceptually plausible distractors. Ambiguous items were replaced to ensure quality.

2.2 Controllable Generative Student Framework

The framework defines a simulated student via a binary skill vector $\mathbf{k} \in \{0,1\}^K$ , where $1$ indicates a retained skill and $0$ indicates a suppressed (forgotten) skill.

Prompting Strategies: Three conditions were tested to enforce the profile:
1. Instruction-only: Explicit constraints to answer correctly for retained skills and incorrectly for suppressed ones.
2. Example-only: Demonstrations of correct and incorrect behavior (retrieved from external files to prevent leakage).
3. Hybrid: Integration of explicit instructions with concrete demonstrations.
Models Evaluated: Three frontier model families: Claude (Anthropic), DeepSeek (Reasoning-oriented), and GPT-4o (OpenAI). All used deterministic decoding ( $T=0$ ).

2.3 Evaluation Metrics

The paper introduces metrics to separate controllability from raw capability:

Relative Loss: Measures the proportional drop in performance for forgotten skills relative to a perfect-student baseline.
Root Mean Squared Error (RMSE): Quantifies the discrepancy between the intended binary profile and the observed continuous performance profile.
Controllability Score ( $C$ ): Rewards high accuracy on retained skills and low accuracy on forgotten skills.
Cross-Skill Influence: Analyzed via Pearson correlation matrices and multivariate calibration to detect if suppressing one skill inadvertently degrades others (interference).
Prediction Score: Compares observed retained-skill accuracy against a theoretical expectation derived from a Gaussian calibration model.

3. Key Contributions

Benchmark Formulation: A novel framework for partial mastery control using explicit binary skill vectors to specify retained and forgotten skills.
Prompt-Based Mechanism: A student-profile mechanism demonstrating that LLM behavior can be steered toward selective retention and suppression.
Evaluation Framework: A suite of metrics (RMSE, retained-vs-forgotten separation, calibration diagnostics) designed to isolate controllability from baseline model capability.
Empirical Study: A comparative analysis across prompting strategies and model families in a curriculum-aligned mathematics setting.

4. Results

4.1 Prompt Strategy Efficacy

Hybrid prompting (instructions + examples) was the only strategy that achieved close alignment with target profiles.

RMSE: Hybrid prompting reduced RMSE to 0.08 (Grade 4) and 0.12 (Grade 5).
Comparison: Instruction-only and example-only strategies resulted in significantly higher error rates (0.46–0.59), indicating that neither constraints nor demonstrations alone are sufficient for precise control.

4.2 Cross-Skill Independence

Correlation: Cross-skill correlations were weak (Grade 4 $\approx$ -0.10; Grade 5 range -0.10 to -0.24).
Implication: Targeted suppression remained largely localized; forgetting one skill did not automatically cause a global collapse in performance across the skill set.

4.3 Model Performance and Controllability

Controllability was found to be model-dependent:

Claude: Demonstrated the strongest controllability, achieving the lowest forgotten-skill accuracy (near 0.00 for some skills) and the largest gap between retained and forgotten performance.
DeepSeek: Showed strong baseline math ability but higher resistance to suppression (retained substantial residual performance on forgotten skills).
GPT-4o: Exhibited mixed results, with uneven suppression across domains and larger calibration errors, particularly in Grade 4.

4.4 Selective Control Visualization

Profile-level analysis confirmed that models could selectively collapse targeted skills while preserving others. For example, a profile requiring the retention of "Measurement" and "Algebra" while suppressing "Fractions" resulted in high accuracy for the former and near-zero accuracy for the latter, rather than a uniform degradation of all skills.

5. Significance and Claims

The paper positions controllable learner simulation as a distinct research problem at the intersection of teacher education, educational simulation, and language-model control.

Primary Claim: Prompt-based control can produce measurable, answer-level partial mastery in a structured mathematics benchmark. It is possible to steer an LLM to behave like a student with a specific skill gap without collapsing into a uniformly weak or uniformly strong agent.
Educational Implication: This capability is a necessary prerequisite for simulating learners for teacher deliberate practice, where exposure to specific, repeatable misconceptions is more valuable than generic correctness.
Modesty of Scope: The authors explicitly state that this work does not validate a full classroom-facing student simulator. The evaluation is limited to multiple-choice answer selection, not tutoring dialogue, explanation quality, or misconception narratives. The framework addresses the "narrower benchmark problem" of skill-level controllability rather than the broader problem of building a fully pedagogical virtual student.
Limitations: The study acknowledges that controllability varies by model architecture and skill family, that the binary mastery assumption oversimplifies continuous learning, and that the current evaluation does not cover open-ended interaction or other subject domains.

In conclusion, the paper establishes that while LLMs are not yet perfect pedagogical agents, they can be configured via hybrid prompting to exhibit selective partial mastery, providing a foundational benchmark for future developments in teacher training simulations.