🤖 AI

From Mastery Profile to Simulated Response: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation

This paper introduces a Stochastic Student Knowledge Graph (SSKG) framework that overcomes the tendency of standard LLMs to default to high mastery by decoupling answer correctness from generation, thereby enabling faithful simulation of diverse student proficiency levels with a clear mastery gradient.

Original authors: Yuan An, Emily Wang, Benjamin Wang, Ruhma Hashmi

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuan An, Emily Wang, Benjamin Wang, Ruhma Hashmi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of education, understanding exactly what a student knows and where they struggle is the key to effective teaching. For decades, researchers have tried to build computer models that can mimic a student's mind, creating synthetic data to test new tutoring systems or to train artificial intelligence before real students ever see a lesson. The hope is to generate thousands of realistic practice scenarios without the time, cost, and ethical hurdles of recruiting actual classrooms. Recently, powerful language models have emerged as a promising tool for this task. These are the same types of artificial intelligence that can write essays, solve complex logic puzzles, and explain difficult concepts in simple terms. The idea is to simply ask the computer to "act like a struggling student" and let it generate the answers. However, a fundamental problem has persisted: these models are so smart that they cannot easily pretend to be less smart. When asked to simulate a student who knows very little, the computer often ignores the instruction and answers correctly anyway, because its internal knowledge is too strong to be suppressed by a simple request.

A team of researchers at Drexel University set out to solve this problem of "fake struggling." They wanted to create a simulation where an artificial student's performance would drop in a predictable, realistic way as their knowledge decreased, rather than staying perfect no matter what the prompt said. To do this, they moved away from asking the computer to guess how a student would behave. Instead, they built a detailed map of the subject matter itself. They took an open algebra textbook and broke it down into thousands of tiny, specific facts and steps, creating a structured network of knowledge. They then assigned a probability of success to each of these tiny steps for a simulated student. If a student had a 20 percent chance of knowing a specific fact, the simulation would randomly decide whether they knew it or not, just like a real person might forget a detail under pressure.

The researchers tested this new approach using 379 algebra questions from the SAT, a standardized exam used to assess college readiness. They first tried the traditional method, simply telling three different large language models to act as students with varying levels of skill, from near-experts to those who were severely struggling. The results were stark. Regardless of whether the computer was told to be a genius or a novice, it answered between 96.8 and 100 percent of the questions correctly. The models were effectively ignoring the instructions to be bad at math. The simulation failed to distinguish between a student who knew everything and one who knew almost nothing.

To fix this, the researchers introduced a new system that separates the decision of whether an answer is right from the explanation of why. First, the system looks at the specific chain of facts required to solve a math problem. It checks the simulated student's map of knowledge against this chain. If the student is supposed to be struggling with a specific concept, the system randomly decides that the student fails at that specific step. This decision is made by a simple mathematical process, not by the language model. Only after the system has decided whether the answer is right or wrong, and exactly which step caused the error, does it ask the language model to write the explanation. The model is then instructed to narrate a first-person story that matches the pre-determined outcome. If the system decided the student failed because they forgot how to convert units, the model writes a story about that specific confusion. If the system decided the student succeeded, the model writes a confident explanation.

The results of this new method were a complete reversal of the traditional approach. When using the new system, the simulated students' accuracy dropped in a smooth, logical gradient. The "near-expert" students got about 85 percent of the questions right, while the "struggling" students got only about 44 percent. Crucially, the errors were not random. When the researchers created a profile for a student who was good at early algebra but bad at advanced topics, the simulation showed them failing specifically on the questions that required advanced knowledge. Similarly, a student with gaps in the basics failed on the foundational questions. The system could even trace every wrong answer back to the exact tiny fact the student had missed, providing a level of diagnostic detail that is rarely available in real classroom data.

The researchers also tested different layers of their new system to see which parts were most important. They found that simply telling the computer to forget facts was not enough; the system needed to understand that some facts are more critical than others. For instance, forgetting a vocabulary definition should not necessarily cause a student to fail a whole problem, but missing a procedural step should. By weighting these different types of knowledge, the simulation became much more realistic. They also found that the system could generate specific wrong answers that matched common student mistakes, rather than just picking a random incorrect option. This allowed the simulation to produce not just a score, but a detailed map of how a student's mind works, showing exactly where the cracks in their understanding lie.

This work suggests that to truly simulate human learning, we cannot rely on the artificial intelligence to guess how a student thinks. Instead, we must build a framework that explicitly models the structure of knowledge and the randomness of human error. By grounding the simulation in a map of the curriculum and letting chance decide the outcome of each step, the researchers created a tool that can generate vast amounts of realistic, diagnostic data. This could help educators and developers test new teaching methods and build better tutoring systems, ensuring they are ready for the messy, unpredictable reality of human learning before they are ever used in a classroom.

Technical Summary: Stochastic Student Knowledge Graphs (SSKG) for Faithful LLM Student Simulation

1. Problem Statement

The paper addresses a critical limitation in using Large Language Models (LLMs) to simulate student responses for educational data generation and system stress-testing. While LLMs excel at solving curriculum questions and generating natural language explanations, they exhibit a "competency bias." When prompted to simulate a student with low mastery (e.g., a "struggling student"), the LLM tends to default to its own high-level capabilities, resulting in near-perfect accuracy regardless of the intended proficiency level.

The authors demonstrate this failure using 379 College Board-calibrated SAT Algebra items and five archetypal mastery profiles (ranging from near-expert to severely gapped). Three distinct LLMs (Gemini 3.1 Flash Lite, Claude Haiku 4.5, and GPT-5.4-mini) achieved 96.8–100% accuracy across all profiles, including those explicitly labeled as "struggling." Consequently, prompt-based approaches fail to produce a monotone mastery gradient or skill-specific error patterns, rendering them ineffective for generating faithful synthetic data where the distinction between mastery levels is crucial.

2. Methodology: The SSKG-Grounded Simulation

To overcome the limitations of prompt-level control, the authors propose a method grounded in a Stochastic Student Knowledge Graph (SSKG). This approach decouples the decision of whether a student answers correctly from the generation of how they explain their answer.

Core Components

Curriculum Knowledge Graph (CKG): A graph extracted from an open algebra textbook, consisting of 864 fine-grained triples (subject, predicate, object). These triples are categorized into three types:
- Definitional: Vocabulary and concepts (e.g., "Variable is a letter").
- Procedural: Executable steps (e.g., "Multiply by a unit fraction").
- Relational: Method-selection facts (e.g., "Uses identity property").
Triple Chain: For every assessment item, a chain of required triples is identified that leads to the correct solution.
Distractor Map: A mapping that links specific failures in the triple chain to specific incorrect answer choices (distractors). This allows the system to predict not just that a student will fail, but which wrong answer they will select based on the specific step they missed.
Stochastic Student Knowledge Graph (SSKG): A student profile is represented as a vector of mastery probabilities ( $p \in [0, 1]$ ) assigned to each triple in the CKG.

The Two-Stage Simulation Process

The simulation operates in two distinct stages:

Stage 1 (Analytic Decision): Given a student profile (SSKG) and an item's triple chain, the system performs explicit stochastic sampling to determine the outcome before invoking an LLM.
- It simulates retrieval (does the student know the triple?) and execution (does the student make a slip/error?).
- It applies a conjunctive rule (based on the DINA model): if any required procedural or relational triple fails, the item is marked incorrect. Definitional failures do not automatically cause item failure but are recorded for narration.
- If the item is incorrect, the system uses the Distractor Map to select the specific wrong answer choice corresponding to the failed triple.
Stage 2 (Narration): An LLM is invoked only after the outcome and the specific failure point are determined. The LLM is tasked with generating a first-person rationale consistent with the pre-determined outcome and the specific error type (retrieval vs. execution).

Ablation Study (Four Arms)

The authors evaluate the method through a cumulative ablation study to attribute gains to specific mechanisms:

Arm-a: Single-draw sampler (Bernoulli trial per triple).
Arm-b: Adds retrieval/execution decomposition (separating knowledge availability from careless slips).
Arm-c: Adds taxonomy-conditioned weighting (definitional failures do not sink the item; only procedural/relational failures do).
Arm-d: Adds distractor routing (selecting specific wrong answers based on the failed step rather than uniform random selection).

3. Key Contributions

Human-Guided CKG Extraction: A method for extracting and constructing a curriculum knowledge graph from textbook sources, verified by humans to ensure accuracy.
SSKG-Grounded Simulation Method: A novel framework that represents student profiles as stochastic vectors over a knowledge graph, enabling the simulation of stratified mastery levels with monotonic accuracy and skill-specific error patterns.
Theoretical and Practical Implementation: A formalization of the simulation problem, including definitions for CKG, triple chains, distractor maps, and SSKGs, alongside a working implementation.
Cumulative Ablation Study: A four-arm study that isolates the contribution of specific mechanisms (retrieval/execution, taxonomy awareness, and distractor routing) to simulation fidelity.

4. Experimental Results

The evaluation was conducted on 379 SAT Algebra items across five student profiles using three LLMs.

Monotonicity (Q1):
- Method 1 (Direct Prompting): Accuracy remained flat at 96.8–100% across all profiles, failing to distinguish between expert and struggling students.
- Method 2 (SSKG): Accuracy showed a clear monotone gradient. For the best-performing arm (Arm-d), accuracy ranged from 85.2% (near-expert) down to 44.1% (struggling), successfully recovering the mastery gradient.
Skill Specificity (Q2):
- Profiles with structured gaps (e.g., weak in early chapters but strong in later ones) failed specifically on items requiring knowledge from their weak regions. Direct prompting showed no such localization (100% accuracy everywhere).
Diagnosability (Q3):
- Under Arm-d, every incorrect answer was traceable to a specific low-mastery triple. A strong negative correlation ( $\rho = -0.459$ ) was found between triple mastery and error rates on items requiring that triple.
- The generated rationales faithfully reflected the specific error types (e.g., unit conversion mistakes) rather than generic arithmetic errors.
Attribution (Q4):
- Retrieval/Execution (Arm-a $\to$ b): This mechanism primarily reduces accuracy for high-mastery profiles (specifically decreasing P1 accuracy by 9.5% and at most 2.1% for others) by introducing realistic "slips" (execution errors). Low-mastery profiles are largely unaffected by this step because they already fail mainly due to retrieval errors, which the slip rate does not impact.
- Taxonomy Awareness (Arm-b $\to$ c): Produced the largest gain (14–19.5% increase in accuracy for lower profiles) by preventing definitional gaps from automatically causing item failure.
- Distractor Routing (Arm-c $\to$ d): Did not change overall accuracy but significantly improved the realism and diagnostic value of the wrong answers by selecting specific distractors.

5. Significance and Claims

The paper claims that architectural control (via SSKG) is superior to prompt-level control for faithful student simulation. By moving the mastery model outside the LLM prompt and into a stochastic, curriculum-grounded architecture, the method:

Restores the monotone mastery gradient that prompt-based methods collapse.
Produces skill-specific error patterns that align with the student's knowledge gaps.
Generates synthetic data with built-in diagnostic labels (failure type and specific knowledge gap), which are typically unavailable in real classroom data.
Reduces computational costs by using LLMs only for narration, allowing for the generation of large, exactly reproducible response corpora.

The authors conclude that while the method successfully simulates the structure of student errors and mastery gradients, future work is needed to validate the simulated accuracy against real student performance data and to compare it directly with other architectural approaches in the literature.