Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning

Technical Summary: Beyond Direct Answering: Aligning Educational LLMs as Socratic Guides via Heuristic Reinforcement Learning

Problem Statement
Large Language Models (LLMs) deployed in educational settings frequently exhibit a "Direct Answerer" failure mode. Instead of adhering to Socratic pedagogy—which prioritizes guiding students through progressive inquiry to foster durable conceptual understanding—current models default to immediately disclosing target concepts. Empirical analysis of over 10,000 student queries from a live science platform revealed that models ranging from 7B to 72B parameters leak core scientific terminology in their opening responses at rates exceeding 96%. The authors argue this is a behavioral issue, not a capability limitation, and cannot be resolved by scaling model size alone.

Methodology: The HeuristicEdu Pipeline
The paper proposes HeuristicEdu, a two-phase alignment pipeline designed to transform the Qwen2.5-7B-Instruct model into a Socratic tutor.

1. Dataset Construction (SocraticEdu):

   • The authors reconstructed a corpus of 797 multi-turn dialogues from a live Chinese educational platform ("Dr. Curious").
   • The process involved filtering 10,512 raw questions for scientific intent (F1=0.98), clustering them into seven elementary-science domains, and selecting 134 high-engagement "seeds."
   • The final corpus consists of 395 real reconstructed trajectories and 402 synthetic trajectories generated via API to cover diverse tutoring styles (guiding, exploratory, analogical).

2. Training Pipeline:

   • Phase 1 (Supervised Fine-Tuning): The model is warm-started on the 797-trajectory SocraticEdu corpus to establish basic Socratic response patterns.
   • Phase 2 (Group Relative Policy Optimization - GRPO): The model undergoes reinforcement learning using a heuristic reward function ($R_{total}$) without a learned critic. The reward is a weighted sum of three components:
     • $R_{cog}$ (Cognitive Depth): Measures the "cognitive lift" between the student's query and the model's response using Bloom's Taxonomy levels (1–6), encouraging the model to elevate the student's thinking rather than just confirming facts.
     • $R_{eng}$ (Curiosity Engagement): Scores the model's response based on an Engagement Proxy Model (EPM) that detects rhetorical questions, analogies, and open-ended hypotheses to sustain curiosity.
     • $R_{dir}$ (Directness Penalty): Penalizes the premature disclosure of target scientific terms. Crucially, this includes a $K_{query}$ correction, which only penalizes terms in the model's response that were not already introduced by the student, preventing false penalties when a student correctly uses terminology.

3. Evaluation Metrics:

   • Scaffolding Effectiveness (SE): The proportion of test cases where a simulated student successfully articulates the target concept without the tutor leaking any core keywords.
   • Conversation Depth (CD): Measures semantic progression by calculating the fraction of consecutive turn pairs with low Sentence-BERT cosine similarity (< 0.70), ensuring the dialogue advances rather than repeats.
   • Leakage Rate: The frequency of core keyword disclosure by the tutor.

Key Results
Experiments were conducted on 30 held-out questions using a simulated student loop.

• Alignment vs. Prompting: The best-performing model, PT-GRPO w/o $R_{dir}$ (GRPO trained without the directness penalty), achieved an SE of 63.3% and a leakage rate of 13.3%. This significantly outperformed the SFT-only baseline (50.0% SE) and the prompt-engineering reference (40.0% SE).
• The Counterproductive Role of $R_{dir}$: A critical finding was that including the directness penalty ($R_{dir}$) during GRPO optimization degraded performance. The "Full" GRPO variant (with $R_{dir}$) dropped to 43.3% SE and 23.3% leakage. The authors attribute this to an optimization conflict where the penalty suppresses necessary scientific vocabulary, hindering the model's ability to use terms as scaffolding cues, which conflicts with the engagement reward ($R_{eng}$).
• Scale vs. Behavior: An unaligned Qwen-72B baseline achieved 0% SE and a 96.7% leakage rate, demonstrating that increasing model scale does not inherently induce Socratic behavior.
• Depth: The best variant also achieved the highest Conversation Depth (0.722), indicating that successful scaffolding correlates with deeper semantic progression.

Significance and Claims
The paper claims three primary contributions:

1. Dataset: The introduction of SocraticEdu, a style-controlled, multi-turn corpus specifically designed for Socratic tutoring in elementary science.
2. Methodology: A demonstration that GRPO with heuristic rewards (specifically cognitive lift and engagement) is more effective than prompt engineering for behavioral alignment in education. The work highlights that explicit anti-leakage constraints can be detrimental when used as gradient-based rewards, suggesting a need for careful reward design.
3. Evaluation: The proposal of SE and CD as outcome-oriented metrics that move beyond surface fluency to assess whether a dialogue actually changes the learner's state.

The authors conclude that Socratic alignment is a targeted behavioral problem requiring explicit optimization. They assert that while parameter training (GRPO) successfully redirects a 7B model toward Socratic inquiry, simply scaling up to 72B without such alignment yields no improvement over direct answering.