Towards Reward Modeling for AI Tutors in Math Mistake Remediation

1. Problem Statement

Evaluating the pedagogical quality of AI tutors, particularly in the context of math mistake remediation, remains a significant challenge.

• Limitations of Standard Metrics: Conventional Natural Language Generation (NLG) metrics (e.g., BLEU, ROUGE) are domain-agnostic and rely on reference texts. They fail to capture nuanced pedagogical aspects such as whether a tutor identifies a mistake, scaffolds reasoning, or avoids revealing the final answer.
• Limitations of Existing Benchmarks: While benchmarks like MRBench exist, they rely on an 8-dimensional annotation taxonomy (e.g., mistake identification, actionability, tone). Previous attempts to aggregate these dimensions into a single score (e.g., summing weighted scores) often result in ties and assume uniform importance across dimensions. This is unrealistic; for instance, "providing guidance" is pedagogically more critical than "tutor tone."
• The Core Gap: There is a lack of a robust reward model that can accurately predict human preferences for the overall quality of a tutor response, balancing competing pedagogical goals (e.g., being actionable vs. not revealing the answer).

2. Methodology

The authors propose a pipeline to create a specialized reward model trained on human-aligned preference data, utilizing a mix of weighted-sum rankings and synthetic data augmentation.

A. Deriving a Pedagogical Hierarchy

The authors first conducted human pairwise preference annotation on MRBench data to determine which pedagogical aspects humans prioritize when selecting a better response. Through discussion and analysis, they established a hierarchy of valuable aspects (in descending order of importance):

1. Factuality + Non-contradiction + No Nonsense: The response must be correct and relevant.
2. Mistake Identification + Location: Explicitly or implicitly identifying the error.
3. Scaffolding + Actionability: Guiding the student step-by-step rather than giving the solution (unless scaffolding fails).
4. Targetedness: Addressing the core misconception.
5. Not Revealing the Final Answer: Avoiding giving away the solution.
6. Clarity + Coherence: Logical flow and clear wording.

B. Synthetic Data Augmentation

Recognizing that human annotation is expensive and that preliminary weighted-sum rankings do not perfectly align with human preferences, the authors developed a synthetic data generation pipeline inspired by Contrastive Learning from AI Revisions (CLAIR).

• Process:
  1. Improving Suboptimal Responses: For responses with poor annotations in key dimensions (Guidance, Actionability, Coherence), an LLM generates revised versions that improve specific aspects (e.g., adding scaffolding or fixing clarity).
  2. Joint Improvement: Generating versions that improve all aspects simultaneously.
  3. Degrading Optimal Responses: Taking high-quality responses and intentionally degrading them along specific dimensions (e.g., making them reveal the answer or lose coherence) to create contrastive pairs.
  4. Factuality Checks: Generating factually incorrect versions to ensure the model penalizes hallucinations strictly.
• Model Used: Claude-Sonnet-4 was selected for generating these synthetic pairs due to its superior performance in preliminary preference classification tasks.
• Dataset: This process created a large-scale dataset of 11,346 preference pairs, balancing various pedagogical dimensions.

C. Reward Model Training

• Architecture: The authors trained Bradley-Terry preference models using Qwen2.5-0.5B and Qwen2.5-1.5B as backbones, along with Gemma-2-2B.
• Training Data: They experimented with three data configurations:
  1. Weighted-sum data: Pairs derived from MRBench using a weighted scoring scheme.
  2. Synthetic data: The newly generated contrastive pairs.
  3. Combined data: Merging weighted-sum data with specific subsets of synthetic data (e.g., weighted-sum + "improved Mistake ID & Location").
• Objective: Minimize negative log-likelihood on pairwise comparisons ($P(y^+ \succ y^-) = \sigma(r(x, y^+) - r(x, y^-))$).

3. Key Contributions

1. Pedagogical Hierarchy: A human-derived hierarchy of pedagogical aspects that clarifies the relative importance of different tutoring behaviors (e.g., factuality > tone).
2. Synthetic Contrastive Dataset: A novel method for generating minimally contrastive tutor response pairs that systematically vary along key pedagogical dimensions, addressing the scarcity of high-quality human preference data.
3. Specialized Reward Model: A reward model trained specifically for AI tutoring that outperforms larger, general-purpose reward models.
4. Open Resources: Release of annotated human preference data, the synthetic augmentation dataset, trained models, and code to facilitate future research.

4. Results

The models were evaluated on a human preference test set (414 pairs) and a weighted-sum test set.

• Performance:
  • The best-performing configuration was Qwen2.5-0.5B trained on a combination of Weighted-sum data + Synthetic data focusing on "Improved Mistake ID & Location".
  • This model achieved 0.74 pairwise accuracy on the human preference test set.
  • Using only synthetic data, the model reached 0.69 accuracy.
• Comparison to Baselines:
  • The proposed 0.5B-parameter model outperformed larger general-purpose reward models, including Skywork-Reward-V2-Llama-3.1-8B (0.64) and RewardAnything-8B-v1 (0.68).
  • It also outperformed the previous state-of-the-art tutor-specific model by Macina et al. (2025) (0.69) on the human test set.
• Error Analysis: Models trained on combined data made complementary errors compared to those trained only on weighted-sum data. The combined model showed a stronger preference for responses that guided reasoning and clarified misconceptions, whereas the weighted-sum-only model occasionally favored responses that simply restated problems or revealed answers.

5. Significance

• Efficiency: The study demonstrates that a small, domain-specific model (0.5B parameters) can outperform much larger general-purpose models (8B+) when trained on high-quality, pedagogically aligned data.
• Pedagogical Alignment: By moving beyond simple score summation to a hierarchy-based preference approach, the work provides a more realistic framework for evaluating AI tutors.
• Scalability: The synthetic data generation pipeline offers a scalable solution for creating preference data in educational domains where human annotation is costly.
• Application: The resulting reward model can serve as a tool for automatic evaluation of AI tutors or as a reward signal for Reinforcement Learning from Human Feedback (RLHF) to align LLMs toward more effective pedagogical behaviors.

6. Limitations

• Annotation Scale: The human test set is relatively small (414 pairs) and relies partly on single-annotator labels.
• Context Dependence: The pedagogical hierarchy is specific to math mistake remediation and may not generalize perfectly to other subjects or age groups without adaptation.
• Synthetic Bias: The synthetic data was generated by a single proprietary model (Claude-Sonnet-4), which may introduce stylistic biases not present in natural human-human tutoring interactions.