EvalConvoLearn: An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations

Technical Summary: EvalConvoLearn – An Open-Source Framework for Evaluating Grounded Learner Simulations in Tutoring Conversations

Problem Statement
While Large Language Models (LLMs) have enabled the creation of richer, more naturalistic conversational learner simulations for testing learning theories, evaluating instructional materials, and powering teachable agents, a critical gap remains. There is currently no open-source framework capable of rigorously evaluating whether these simulations faithfully reproduce real learner behavior. Existing approaches often focus on single-turn matching or are closed-source, hindering reproducibility and extension to diverse datasets. Furthermore, LLMs trained as helpful assistants struggle to simulate the specific conversational characteristics and learning trajectories of actual students.

Methodology: The EvalConvoLearn Framework
EvalConvoLearn is an open-source framework designed to assess learner simulations along two primary axes: learning behavior (skill-conditioned mastery outcomes) and conversational quality (talk moves, error types, question rates, and turn length). The framework operates by grounding metrics in authentic tutoring conversation datasets and anchoring generated tutor responses in existing tutor utterances.

The evaluation process follows these steps:

1. Scenario Definition: Given a dataset of tutoring dialogues with tagged skills, the framework extracts learning scenarios defined by a target skill and the learner's prior mastery state.
2. Simulation Interface: Custom learner simulations must implement a minimal interface of four functions: initializing a knowledge state, generating responses based on conversation history and knowledge, updating knowledge states, and reporting mastery status.
3. Tutor Grounding: To ensure realistic interactions, the framework employs a tutor grounded at the individual level using few-shot dialogue examples from the dataset.
4. Metric Calculation:
   • Learning Behavior Score ($LB$): The framework treats each learner turn as a skill practice attempt, capping conversations at 7 turns. The $LB$ score is calculated as the macro-averaged L1 distance between the simulated and real outcome distributions for each scenario and skill.
   • Conversational Realism Score ($CONV$): This score aggregates four metrics: the presence of specific talk moves, error type distributions (both LLM-labeled with high inter-rater reliability), the rate of questions in interrogative turns, and turn length. Distances between simulated and real distributions for these metrics are computed using Jensen–Shannon divergence or Wasserstein-1 distance.
5. Final Score ($ECL$): The final EvalConvoLearn score is a weighted average of the $CONV$ and $LB$ scores across all scenarios, accounting for the distribution of scenarios in real dialogues.

Experimental Evaluation
The framework was applied to the Eedi dataset, a large collection of student–teaching assistant tutoring dialogues. The study retained 66 conversations where students proactively sought help, assuming they had mastered prerequisites but not the target skill. Two LLM-based learner simulations were evaluated using GPT-4.1-mini with custom knowledge implementations:

• Conversation Summaries Learner: Stores summaries of past conversations to inform new responses.
• Binary Skills Learner: Uses an LLM-as-a-judge to tag items with skills and stores binary mastery outcomes.

Both simulations were tested against a tutor using 3-shot prompting (Claude's Sonnet-4.6) across three skills.

Key Results
The evaluation yielded the following findings (mean scores across 3 runs):

• Learning Behavior: Both simulations achieved similar $LB$ scores (approx. 0.556), indicating they solved roughly 90% of items. This is significantly higher than real students (a 56-point difference), suggesting the models currently overestimate student mastery.
• Conversational Realism: The simulations showed high within-learner consistency but diverged in specific metrics. The "Binary Skills" learner performed better on error types (0.385 distance) compared to the "Conversation History" learner (0.708), while the latter performed better on question rates (0.226 vs. 0.415).
• Overall Performance: The final $ECL$ scores were 0.494 for the Binary Skills learner and 0.503 for the Conversation History learner, indicating that while the framework can differentiate between simulation designs, there remains substantial room for improvement in generating realistic learner behavior.

Significance and Claims
The paper positions EvalConvoLearn as a foundational tool for benchmarking learner simulations across educational contexts. Its primary contribution is the provision of an open-source, reproducible method to evaluate simulations based on goal-aligned fidelity rather than surface-level realism. The author claims the framework provides a signal for iterating on learner designs by targeting specific conversational or learning deficits.

The paper modestly frames EvalConvoLearn as a "work in progress." It acknowledges the need for expanding the suite of scenarios and further validating metrics with human ratings. The author does not claim the current simulations are perfect; rather, they emphasize that the framework successfully identifies gaps (such as the overestimation of mastery) and offers a pathway for developing benchmarks for various educational contexts, including the integration of fine-tuned or RL-trained models in the future.