💬 NLP

Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation

This paper proposes a two-stage alignment pipeline combining supervised fine-tuning and Direct Preference Optimization on a pedagogically enriched dataset to create open-source LLM tutors that effectively remediate math mistakes while adhering to pedagogical strategies like scaffolding, achieving performance competitive with proprietary baselines.

Original authors: Kseniia Petukhova, Tien Dat Nguyen, Ekaterina Kochmar

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kseniia Petukhova, Tien Dat Nguyen, Ekaterina Kochmar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but overly eager new teaching assistant. This assistant has read every math book in the library and can solve any problem instantly. However, when a student makes a mistake, this assistant's instinct is to immediately shout out the correct answer and the full solution. While this is helpful for getting the answer right, it doesn't actually help the student learn how to solve the problem themselves.

This paper is about teaching that eager assistant to be a real tutor instead of just a "cheat sheet."

The Problem: The "Answer Key" Trap

The authors found that standard AI models (Large Language Models) are great at talking and solving math, but they are terrible at pedagogy (the art of teaching). If you ask them to help a student who got a math problem wrong, they often:

Spill the beans: They give away the final answer too quickly.
Hallucinate: They sometimes invent numbers or facts that aren't in the problem.
Miss the mark: They don't pinpoint exactly where the student went wrong.

A good tutor, like a human coach, uses a technique called scaffolding. Imagine building a house; you don't just hand the builder the finished roof. You give them a ladder, a few tools, and a hint about where the beam goes, letting them do the heavy lifting. The AI needs to learn to do the same: guide the student to the answer without handing it to them.

The Solution: A Two-Step Training Camp

The researchers built a special training pipeline to fix this, which they call a "two-stage alignment pipeline." Think of it as a two-step boot camp for the AI.

Stage 1: The Classroom Imitation (Supervised Fine-Tuning)
First, they took the AI and showed it thousands of examples of real (and simulated) conversations between human tutors and students. It's like putting the AI in a classroom where it has to watch and copy the best teachers. It learns the style of a tutor: asking questions, offering hints, and not giving away the answer.

Stage 2: The "Good vs. Bad" Taste Test (Direct Preference Optimization)
This is the clever part. The researchers created a massive dataset of "preference pairs."

The "Good" Tutor: An AI response that gently guides the student, spots the specific error, and stays factually accurate.
The "Bad" Tutor: A response that is the same topic but slightly "degraded." Maybe it accidentally gives the answer away, maybe it hallucinates a number, or maybe it misses the student's mistake entirely.

They then used a technique called Direct Preference Optimization (DPO). Imagine a judge tasting two cookies: one is perfect, and the other is slightly burnt or missing sugar. The judge tells the baker, "I prefer the first one." The AI learns from millions of these comparisons to understand exactly what makes a response "good" from a teaching perspective, not just a math perspective.

The Secret Ingredients

To make the AI even better, the researchers tested giving it different "cheat sheets" during training:

Just the conversation: The AI has to guess if the student is right or wrong.
The "Correctness Flag": A simple yes/no button telling the AI, "Hey, the student is wrong."
The Gold Solution: The AI is given the correct answer key.

They found that when the AI knew the correct answer and knew the student was wrong (the "Gold Solution" setup), it became much more factually accurate. It's like a human tutor who has the answer key in their pocket; they can spot errors much faster without accidentally making up new math rules.

The Results: Beating the Big Players

The team tested their new "Pedagogically Aligned" AI against:

Base models: The raw, untrained AI.
Existing tutoring AIs: Other models already designed for teaching.
A "Proprietary Baseline": A very strong, expensive, closed-source AI (like a top-tier commercial product).

The Verdict:

Fact Check: Their model made far fewer factual errors than the others.
Teaching Style: It was much better at spotting mistakes and guiding students without revealing the answer.
The Big Win: In human tests, their open-source model (which anyone can use and study) was rated as better than the expensive, proprietary "black box" model.

The Catch (Limitations)

The authors are honest about the flaws:

The "Judge" Problem: They used other AIs to grade the tutors. Sometimes the "judge" AI disagreed with the human evaluators. It's hard to perfectly measure "good teaching" with a computer.
Synthetic Data: The training data was mostly generated by other AIs, not real human teachers. While this is scalable, it might miss some of the messy, nuanced reality of a real classroom.
Did they learn? The study measured how good the tutor sounded, not whether the students actually learned more math. They didn't test if the students' test scores went up, only that the AI sounded like a better teacher.

Summary

In short, this paper shows that you can take a smart but clumsy AI and teach it to be a patient, accurate, and helpful math tutor. By using a two-step process of imitation and "taste-testing" good vs. bad responses, they created an open-source AI that teaches better than some of the most expensive, closed-source models currently available. It's a step toward AI that doesn't just give answers, but actually helps us understand them.

Technical Summary: Towards Pedagogically Aligned LLM Tutors for Math Mistake Remediation

Problem Statement

While Large Language Models (LLMs) demonstrate strong capabilities in reasoning and natural language generation, they are often ill-suited for direct deployment in educational settings without specific adaptation. Standard LLMs frequently fail to adhere to effective pedagogical strategies, such as guiding students through Socratic questioning or providing targeted hints without prematurely revealing final answers. Existing tutoring models often rely on supervised imitation of pedagogical behavior or holistic reinforcement learning with scalar rewards, which can limit explicit control over specific pedagogical attributes like factual accuracy, mistake identification, and the degree of solution revealing. The core challenge addressed is how to align LLMs with established human tutoring practices to create systems that are not only factually accurate but also pedagogically sound in remediating math mistakes.

Methodology

The authors propose a two-stage pedagogical alignment pipeline designed to align LLM responses with desired tutoring behaviors.

1. Data Construction

The study constructs a comprehensive dataset for math mistake remediation by integrating existing tutoring corpora with synthetic data:

Instruction Tuning Data: The authors utilize MathDial and SocraTeach, preprocessing conversations into tutor-student turns where the tutor's response serves as the gold target.
Synthetic Preference Data: To address the scarcity of high-quality pedagogical preference data, the authors adapt the RMBoost approach using MR-GSM8K and PRM800K.
- LLM-generated solutions from these datasets are treated as student responses.
- GPT-5 is used to generate high-quality tutor responses that adhere to specific pedagogical criteria (Factuality, Mistake Identification, Targetedness, Avoiding Answer Revealing, and Clarity).
- GPT-4.1 is used to generate "degraded" variants of these responses by intentionally violating specific criteria, creating preference pairs (preferred vs. non-preferred).
- The dataset includes cases with both incorrect and correct student solutions to prevent the model from always assuming errors.
- The final dataset comprises over 146,000 instances for Supervised Fine-Tuning (SFT) and 29,390 preference pairs for Direct Preference Optimization (DPO).

2. Training Pipeline

The alignment process involves two distinct stages:

Stage 1: Supervised Fine-Tuning (SFT): Base models (Qwen3-4B and Qwen3-8B) are fine-tuned on the tutoring dialog data using LoRA. This stage establishes a baseline for generating fluent, context-aware tutor responses.
Stage 2: Direct Preference Optimization (DPO): The SFT models are further aligned using the synthetic preference pairs. A weighted DPO objective is employed, where pedagogical aspects (Factuality, Mistake Identification, Targetedness, Revealing Answer) are weighted at 1.0, while Clarity is weighted at 0.5, reflecting the authors' observation that clarity degradation does not always equate to pedagogical failure.

3. Input Configurations

The study investigates four input configurations to determine how access to external information affects performance:

V1: Dialog context only.
V2: Context + a binary flag indicating student correctness.
V3: Context + the gold solution.
V4: Context + correctness flag + gold solution.
The authors note that while V4 assumes access to signals not always available in real-time deployment, it serves as an upper bound reflecting standard human tutoring practices where tutors have access to correct solutions.

Key Contributions

Dataset Construction: Creation of a large-scale dataset for math mistake remediation that combines real tutoring dialogs with synthetic preference pairs generated across multiple pedagogical dimensions.
Input Configuration Analysis: A systematic study of how providing models with correctness flags and gold solutions impacts factual accuracy and pedagogical quality.
Two-Stage Alignment: Demonstration that combining SFT with DPO effectively aligns open-source models with pedagogical principles, outperforming base models and existing specialized tutoring models.
Resource Release: The release of the synthetic dataset, trained models, and code to facilitate further research in pedagogically aligned AI tutors.

Results and Evaluation

Automatic Evaluation

The models were evaluated on 1,000 test samples across four pedagogical dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability.

Factuality: Access to the gold solution and correctness flag (V4) significantly improved factual accuracy. The best-performing model, Qwen3-8B + DPO V4, achieved a factuality score of 60.10% (improved to 70.10% after a targeted retraining to fix numerical hallucinations).
Pedagogical Quality: The Qwen3-8B + DPO V4* model outperformed existing open-source baselines (SocraticLM, TutorRL-7B) and the proprietary GPT-4.1-nano fine-tuned models on most dimensions under GPT-5 evaluation.
Evaluator Disagreement: A significant discrepancy was observed between evaluators. GPT-5 tended to penalize responses that revealed answers (lowering "Actionability" scores), while LoMTL (a smaller 2B model) often rated such responses highly. This highlights the unreliability of current automated evaluators for fine-grained tutoring dimensions.

Human Evaluation

A human evaluation involving 10 annotators (NLP students and professors) compared the best open-source model (Qwen3-8B + DPO V4*) against its SFT variant, the base model, and a strong proprietary baseline (GPT-5).

Performance: The DPO-aligned model was preferred over the SFT variant in 35.3% of cases and over the GPT-5 baseline in 54.3% of cases.
Qualitative Findings: Annotators noted that the base model often lacked coherence and factual accuracy. The SFT model improved clarity but sometimes revealed answers. The DPO model was preferred for its ability to provide effective scaffolding without revealing the final answer, whereas GPT-5 was frequently criticized for revealing the solution.
Agreement: Inter-annotator agreement (Fleiss' $\kappa$ ) was 0.55, indicating moderate agreement.

Significance and Limitations

The paper argues that preference-based pedagogical alignment is an effective strategy for improving LLM tutors, particularly when combined with explicit access to solution correctness and gold answers. The results suggest that intelligent tutoring systems may benefit from a modular design where separate components handle task solving, verification, and feedback generation, rather than relying on a single monolithic model.

The authors maintain a modest stance regarding the scope of their findings:

Evaluation Limitations: The study focuses on the quality of tutor responses rather than direct student learning outcomes. The authors acknowledge that evaluating actual educational effectiveness would require controlled user studies with pre- and post-tests, which were outside the scope of this work.
Data Limitations: The preference data is synthetic, generated by LLMs. While this offers scalability, the authors note it may encode family-specific stylistic biases and does not fully capture the diversity of natural human tutoring interactions.
Evaluator Reliability: The significant disagreement between automated evaluators (GPT-5 vs. LoMTL) underscores the current difficulty in reliably assessing tutoring quality without human oversight.

In conclusion, the work demonstrates that open-source models, when aligned via a two-stage pipeline on pedagogically structured data, can achieve competitive performance with strong proprietary baselines while offering greater transparency and reproducibility.