Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning

1. Problem Statement

Large Language Models (LLMs) generally excel at generating coherent text but often lack pedagogical knowledge—the strategic ability to guide learners toward answers rather than simply providing them. Current commercial (proprietary) models are often opaque, expensive to fine-tune, and difficult to customize for specific educational needs. Conversely, while open-source models offer transparency and cost-efficiency, they typically underperform compared to massive proprietary systems on specialized pedagogical tasks.

The core challenge is to transform a mid-sized, open-source LLM into a domain expert capable of outperforming significantly larger proprietary models in educational settings, specifically by distinguishing between content knowledge (factual accuracy) and pedagogical knowledge (teaching strategy).

2. Methodology

The authors propose a novel multi-stage optimization pipeline (RL-SFT-RL) applied to the Qwen3-32B dense model (32 billion parameters). The process consists of three distinct phases:

A. Base Model Selection

• Model: Qwen3-32B (Dense architecture).
• Rationale: Selected for its strong baseline performance, computational efficiency, and demonstrated responsiveness to iterative optimization compared to Mixture-of-Experts (MoE) architectures.

B. Phase 1: Initial Reinforcement Learning (RL)

• Algorithm: DAPO (Decoupled Advantage Policy Optimization) was selected over GRPO. DAPO decouples policy updates from advantage estimation, providing more stable gradients for complex, multi-step reasoning chains required in pedagogy.
• Strategy:
  • Progressive Difficulty Training: The model is trained on a curriculum starting with simple questions and gradually increasing complexity.
  • Failure-Mode Identification: The base model attempts 30 times per question. Questions answered correctly every time are discarded. The remaining questions (where the model showed uncertainty or errors) form a "hard-negative" dataset of 440 items.
  • Extended Rollouts: Reasoning rollouts were increased from 5 to 8 steps to encourage longer chains of thought.
  • Reward Model: Trained to prioritize responses that guide learners (scaffolding) rather than giving direct answers.
• Outcome: Produces EduQwen 32B-RL1.

C. Phase 2: Supervised Fine-Tuning (SFT)

• Data Synthesis: The RL-optimized model (EduQwen 32B-RL1) generates 40,000 synthetic training examples.
• Difficulty-Weighted Sampling: A rigorous filtering process retains only high-quality examples where the model previously struggled.
  • High-accuracy questions: 1 representative example retained.
  • Low-accuracy (hard) questions: All high-quality examples retained.
  • This results in a curated dataset of 1,050 difficulty-ordered data points.
• Training: Standard cross-entropy loss with increased weights assigned to high-difficulty questions to act as a corrective regularizer.
• Outcome: Produces EduQwen 32B-SFT.

D. Phase 3: Optional Second Round of RL

• Process: The SFT model undergoes a second round of RL optimization using the same difficulty-aware dataset from Phase 1.
• Goal: To solidify gains and further refine the model's ability to solve previously challenging problems using its refined knowledge.
• Outcome: Produces EduQwen 32B-SFT-RL2.

3. Key Contributions

1. EduQwen Family: Development of three open-source pedagogical models (EduQwen 32B-RL1, EduQwen 32B-SFT, and EduQwen 32B-SFT-RL2) that serve as domain experts.
2. Novel Pipeline: Introduction of a RL-SFT-RL framework that combines:
   • DAPO-based RL with progressive difficulty and extended rollouts.
   • Difficulty-weighted SFT using synthetic data generated by the RL model itself (a virtuous cycle).
3. Empirical SOTA: Demonstration that a 32B parameter open-source model can outperform significantly larger proprietary systems (e.g., Gemini-3 Pro, Qwen3-235B) on specialized pedagogical benchmarks.

4. Results

The models were evaluated on the Cross-Domain Pedagogical Knowledge (CDPK) Benchmark, a dataset of 920 teacher multiple-choice exam questions designed to test strategic teaching capacity.

• Baseline (Qwen3-32B): 83.37% accuracy.
• EduQwen 32B-RL1 (Post-RL): 94.13% accuracy (+10.76 points).
• EduQwen 32B-SFT (Post-SFT): 96.20% accuracy (+2.07 points).
• EduQwen 32B-SFT-RL2 (Final Model): 96.52% accuracy (+0.32 points).

Comparative Performance:

• The final model (96.52%) surpasses the previous leaderboard leader, Gemini-3 Pro (90.55%), by 5.97 percentage points.
• It significantly outperforms much larger models, such as Qwen3-235B-A22B-Thinking (86.63%) and Qwen2.5-72B-Instruct (78.60%).
• Generalizability: Preliminary tests on the TutorBench (multimodal) benchmark showed the SFT-optimized model (EduQwen 30B-VL-SFT) achieving 61.64% accuracy, surpassing Gemini-3 Pro (58.52%).

5. Significance and Impact

• Cost-Efficiency & Accessibility: The study proves that domain-specialized, mid-sized open-source models can outperform massive general-purpose systems. This drastically reduces the cost of deploying high-quality AI tutors, making them accessible to institutions without resources for expensive proprietary APIs.
• Transparency & Customization: Unlike proprietary "black box" models, the EduQwen family allows educators to audit, customize, and adapt the AI to specific cultural contexts and pedagogical frameworks, addressing critical concerns regarding bias and data privacy.
• Pedagogical Alignment: The methodology successfully shifts model behavior from "answering" to "teaching," aligning LLMs with the "Two-Sigma Problem" of personalized tutoring (Bloom, 1984).
• Future Outlook: The authors suggest this optimization strategy is scalable to larger architectures and emphasizes that these models are designed to augment human educators, not replace them, requiring careful validation in real-world classroom settings.