🤖 AI

BEAGLE: Behavior-Enforced Agent for Grounded Learner Emulation

BEAGLE is a neuro-symbolic framework that integrates the theory of self-regulated learning with a semi-Markov model, Bayesian Knowledge Tracing with feed-in errors, and a decoupled agent design to simulate authentic, error-prone student learning behaviors in open-ended problem-solving tasks, thereby effectively passing a human Turing test against real student data.

Original authors: Hanchen David Wang, Clayton Cohn, Zifan Xu, Siyuan Guo, Gautam Biswas, Meiyi Ma

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanchen David Wang, Clayton Cohn, Zifan Xu, Siyuan Guo, Gautam Biswas, Meiyi Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher who has developed a new way to teach programming to students. To test your new teaching method, you need to see how it works with students who struggle, get confused, and make mistakes. But you cannot simply let real children fail over and over again; that is expensive, takes too long, and raises data privacy concerns.

So you want to build a digital robot student that behaves exactly like a real human learner.

The problem is that current AI models (Large Language Models) are too intelligent. If you ask them to "play a confused beginner," they usually solve the problem perfectly anyway. They are like a brilliant actor who forgets their lines and instead improvises a perfect ending, rather than stumbling through the scene.

This work introduces BEAGLE, a new type of AI intentionally designed as a "poor" student. It is not just about pretending; it possesses a special internal mechanism that forces it to struggle, get stuck, and learn the hard way, just like a real human.

Here is how BEAGLE works, broken down into simple parts:

1. The "Brain" vs. the "Hands" (The Architect)

Most AI models try to think and write code simultaneously. BEAGLE splits this task into two distinct roles:

The Strategist (The Brain): This part decides what the student thinks. It says: "I am confused," or "I think the gravity calculation is wrong," or "I will just guess." It sets the mood and the goal.
The Executor (The Hands): This part actually writes the code. Crucially, it is not allowed to correct its own mistakes if the "Brain" has instructed it to make a specific error. If the Strategist says, "I forgot to add gravity," the Executor must write code that forgets gravity, even if the AI knows better. This prevents the AI from "cheating" by silently self-correcting.

2. The "Traffic Cop" (Symbolic Control)

Real students do not work in a straight line. They build a little, get stuck, try to fix it, get stuck even harder, then take a break and try again.
BEAGLE uses a Traffic Cop (a semi-Markov model) to control the flow. It decides:

"Okay, for the next 5 minutes you must Debug (get stuck on an error)."
"Switch now to Planning for 2 minutes."
"Now Act (just type blindly without thinking)."
This ensures the AI does not solve the problem in 6 perfect steps. It forces the AI to spend time in the chaotic, iterative loops where real students get trapped.

3. The "Blinding" (Knowledge Tracking and Error Injection)

Real beginners suffer from the "curse of incompetence": they do not just make mistakes; they do not even know what they do not know. They might try to use a mathematical tool they have never heard of.
BEAGLE uses a Blinding system:

It tracks what the student "knows" (like a knowledge map).
If the student is supposed to be a beginner in "Trigonometry," BEAGLE blocks the AI from using mathematical libraries related to it.
Instead of using the right tool, the AI must improvise. It might try to guess the answer or write a fake, broken version of the mathematical function. This creates "unknown unknowns" – errors that occur because the student actually lacks the concept, not because they made a typo.

4. The "Turing Test" (Did it work?)

The researchers tested BEAGLE by showing real humans (students, teachers, and experts) a mix of code from real students and code from BEAGLE. They asked: "Which one is the human?"

The result was startling: Humans could not tell the difference.

Guessing accuracy was about 53%, which is basically a coin flip.
Even experts with years of experience could not unmask the AI.
The AI's code looked chaotic, emotional ("Ugh, why doesn't this work?"), and full of the same repetitive mistakes real students make.

Why this matters

BEAGLE is not just a cool trick; it is a safe sandbox for educational research.

Stress Tests: Teachers can use BEAGLE to test new tutoring software. They can see if their new app helps a "confused robot student" learn without endangering the education of a real child.
Data Privacy: You do not need to collect private data from real children to train these systems.
Understanding Difficulties: It helps researchers investigate why students get stuck, because BEAGLE gets stuck for the right reasons (lack of knowledge) and not just because the AI is acting randomly.

In short: BEAGLE is the first AI that has successfully learned to be a struggling student by being forced to follow the rules of human confusion, rather than just being told to "play confused." It proves that to simulate a human learner, you cannot simply ask an intelligent AI to play; you must build a system that forces it to be imperfect.

Technical Summary: BEAGLE – Behavior-Enforced Agent for Grounded Learner Emulation

Problem Statement

Progress in generative AI for education depends heavily on authentic student data to train, evaluate, and refine pedagogical models. However, acquiring such data is hindered by strict privacy regulations (e.g., FERPA, GDPR), the high costs of longitudinal studies, and ethical concerns regarding interventions with real minors. While Large Language Models (LLMs) offer a potential solution for simulating student behavior, they suffer from a fundamental competence bias. Preference-optimized LLMs are tuned for outcome quality and efficiency, leading them to generate linear, correct solutions even when instructed to act as novices. This results in three specific types of unfaithfulness:

Behavioral Unfaithfulness: LLMs avoid the iterative trial-and-error cycles (e.g., persistent debugging loops) characteristic of Self-Regulated Learning (SRL).
Epistemic Unfaithfulness: LLMs act as "masked experts," often possessing the insight to diagnose errors they should not know about, and fail to simulate the "curse of incompetence" (unknown unknowns).
Perceptual Unfaithfulness: RLHF training smooths out output diversity and fails to capture the noise and emotional variability of authentic student expression.

Methodology: The BEAGLE Framework

BEAGLE (Behavior-Enforced Agent for Grounded Learner Emulation) is a neuro-symbolic framework designed to overcome competence biases through architectural constraints rather than relying solely on prompt engineering. It operates as a training-free system with frozen LLM backbones and integrates three coupled subsystems:

1. Symbolic Control (The "Governor")

This subsystem enforces the temporal and structural dynamics of learning based on SRL theory.

Semi-Markov Dynamics: Unlike standard Markov models that enforce geometric duration distributions, BEAGLE uses a Semi-Markov controller with Gamma-distributed durations. This captures the irregular persistence of real students (e.g., "getting stuck" in a state) and governs transitions between metacognitive behaviors (Planning, Executing, Monitoring, Reflecting) and cognitive behaviors (Constructing, Debugging, Evaluating).
Explicit Error Injection (EFI) & Bayesian Knowledge Tracing (BKT): To simulate "unknown unknowns," BEAGLE combines BKT with EFI. BKT tracks the probability of mastery for specific Knowledge Components (KCs). If a KC is not mastered, EFI blocks the agent's access to that concept in its vocabulary. The agent is forced to improvise incorrect solutions or remain confused, preventing the LLM from leveraging its latent expert knowledge to "cheat."
Observation Filtering: Feedback is gated by the agent's metacognitive state. During impulsive "Executing" states, error logs are redacted, forcing the agent to initiate realistic trial-and-error processes rather than making immediate diagnoses.

2. Neural Action (The "Actor")

This subsystem employs a Strategist/Executor architecture to decouple high-level planning from code generation, thereby preventing silent self-correction.

Strategist: Synthesizes a high-level plan consisting of a Goal, a Stance, and a Directive based on the current symbolic state. It functions as an inner monologue.
Executor: Generates the final action (code and monologue) strictly conditioned on the Strategist's directive. Crucially, the Executor is forced to implement the directive even if it is flawed; it cannot overwrite the plan with a "better" solution. This architectural separation ensures the agent follows the flawed reasoning path of a novice rather than correcting it.

3. Environment

The environment executes the generated code in a Python sandbox (IDE Oracle) and returns filtered observations to the agent. It updates BKT mastery probabilities only during conscious evaluation phases (Monitoring/Reflecting), thereby mirroring how humans consolidate knowledge.

Main Contributions

Novel Architecture: The first system that jointly enforces behavioral dynamics (via Semi-Markov control) and knowledge constraints (via BKT + EFI) to simulate novice learners.
Decoupled Agent Design: The Strategist/Executor split effectively prevents the "silent self-correction" common in LLMs and enforces epistemic faithfulness by forcing the agent to implement flawed plans.
Explicit Error Injection: A mechanism for simulating "unknown unknowns" through deterministic blocking of access to specific concepts, forcing the agent to improvise rather than access latent expert knowledge.
Comprehensive Evaluation: A rigorous evaluation across four tasks (three in-distribution physics simulations, one out-of-distribution math task), six leading LLM backbones, and nine baselines.

Results

BEAGLE was evaluated on metrics for behavioral, epistemic, and perceptual faithfulness:

Behavioral Faithfulness: BEAGLE significantly reduced the KL-divergence ( $D_{KL}$ ) between simulated and real cognitive behavior distributions (0.31 vs. baselines $\ge$ 0.53). It successfully reproduced iterative debugging loops (C $\leftrightarrow$ D transitions) that baselines systematically bypassed.
Epistemic Faithfulness: BEAGLE achieved an error repetition rate of 86.2%, corresponding to the "curse of incompetence" where novices repeatedly struggle with the same errors. In contrast, baselines like Vanilla-LLMs solved errors immediately (7.8% repetition).
Perceptual Faithfulness (Human Turing Test): In a study with N=71 participants (including students, professionals, and educators), participants could not reliably distinguish BEAGLE traces from real student data. Classification accuracy was 52.8%, statistically equivalent to chance ( $p=0.053$ ). Analysis using Signal Detection Theory yielded a discriminability index of $d' = 0.15$ , and the equivalence test (TOST) confirmed that the traces were perceptually indistinguishable from real data ( $p_{TOST} = 0.038$ ).
Generalization: The framework maintained high faithfulness across various LLM backbones (Gemini, GPT, Claude) and out-of-distribution tasks (Gradient Descent), demonstrating that the architectural constraints generalize beyond specific model capabilities.

Significance and Claims

The work claims that BEAGLE represents a significant advance in synthetic student generation by moving beyond persona-based assumptions via prompts toward the architectural enforcement of learning theories. By successfully simulating the "struggle" of learning—including persistent errors, lack of self-insight, and non-linear problem solving—BEAGLE enables scalable, ethical AI education research. It allows researchers to stress-test pedagogical interventions and train adaptive tutor systems without the risks and costs associated with experiments on real students. The authors note that while a residual competence bias remains on the strongest backbones, the framework provides a principled foundation for future work in multi-agent classrooms and capacity-adaptive symbolic control.