Technical Summary: KT4EQG

Problem Statement

Educational Question Generation (EQG) aims to synthesize customized exercise questions to enhance student learning. While existing EQG approaches have made strides in text quality, format, and difficulty control, they largely lack fine-grained personalization. Most current systems do not effectively model a student's specific knowledge state to determine which questions would provide the greatest learning benefit. Consequently, generated questions may be too trivial, too difficult, or focused on material the student has already mastered.

The core challenge addressed in this paper is how to integrate Knowledge Tracing (KT)—a method that models a student's evolving knowledge state based on historical performance—into the EQG process. The goal is to move beyond selecting questions from a fixed pool (as in recommendation systems) to generating new, diverse questions that are specifically tailored to a student's current learning needs and knowledge gaps.

Methodology: KT4EQG Framework

The authors propose KT4EQG, a personalized EQG framework that decomposes the generation task into two distinct optimization steps: selecting the optimal knowledge concept (KC) and generating a question aligned with that concept.

1. Task Formulation and Objective

The objective is to generate a question $x_{t^*}$ that maximizes the student's overall knowledge mastery after practice. The authors define the learning progress as the sum of mastery probabilities across all knowledge concepts $C$.
$$\max_{x_{t^*}} \sum_{c \in C} p(K_c = 1 | H, X_{t^*} = x_{t^*}, A_{t^*} = 1)$$
where $H$ is the historical practice data, and $A_{t^*}=1$ assumes the student successfully answers the new question (treating exercises as learning opportunities rather than one-shot assessments).

2. Two-Step Optimization

The authors decompose the objective function into two terms: Educational Value and Alignment.
$$\max_{x_{t^*}, c_{t^*}} \left[ \underbrace{\sum_{c \in C} p(K_c = 1 | H, \psi(X_{t^*}) = c_{t^*}, A_{t^*} = 1)}_{\text{Educational Value}} \cdot \underbrace{\mathbb{1}[\psi(x_{t^*}) = c_{t^*}]}_{\text{Alignment}} \right]$$

Step 1: Maximizing Educational Value (Concept Selection)

• Mechanism: The framework uses a Knowledge Tracer (KT2) to estimate the "educational value" of practicing each candidate KC. This value represents the potential improvement in the student's overall knowledge mastery if they were to practice that specific concept.
• Process: An exhaustive search is performed over all candidate KCs. The KT2 model, based on a hidden Markov tree structure, computes the posterior distribution of knowledge states. The KC that yields the highest sum of updated mastery probabilities across all concepts is selected as the target concept $c_{t^*}$.

Step 2: Maximizing Alignment (Question Generation)

Once the optimal KC $c_{t^*}$ is selected, a Large Language Model (LLM) generator is tasked with producing a question $x_{t^*}$ that faithfully tests this concept. This is achieved through a two-phase training process:

1. Supervised Fine-Tuning (SFT): A text LLM (QWEN3-8B) is fine-tuned on a dataset of question-concept pairs to learn the basic mapping from a KC to a question.
2. Reinforcement Learning (RL) with Alignment Reward: The model is further optimized using RL. The reward signal is provided by a Question-Concept Verifier ($g_\phi$).
   • Verifier Training: The verifier is trained using an InfoNCE-style contrastive objective to map questions and concepts into a shared embedding space, ensuring aligned pairs are closer than non-aligned pairs.
   • Reward Calculation: The alignment reward $R_{align}$ is computed based on the similarity between the generated question and the target KC in the embedding space, standardized for numerical stability.

Key Contributions

1. Integration of KT and EQG: The paper proposes the first framework to explicitly integrate knowledge tracing into the generation loop of EQG, moving beyond simple recommendation or coarse difficulty adaptation.
2. Decomposed Optimization: The authors introduce a novel decomposition of the EQG objective into "Educational Value" (determined by the KT model) and "Question-Concept Alignment" (determined by a learned verifier).
3. Training Pipeline: The development of a two-stage training pipeline (SFT + RL with a contrastive alignment verifier) to ensure generated questions are not only fluent but semantically grounded in the selected knowledge concept.
4. Adaptive Policy: The framework naturally produces an adaptive policy where the selected KCs shift based on the student's current mastery level (e.g., selecting easier concepts for weaker students and more challenging ones for advanced students).

Experimental Results

The framework was evaluated on two datasets: XES3G5M (expert-annotated KC trees) and MOOCRADAR (LLM-constructed KC trees).

• Performance: KT4EQG consistently achieved the highest Exam Scores (average predicted correctness on a held-out test set) across all knowledge modules in both datasets, outperforming baselines including standard LLMs (QWEN3-8B), specialized math generators (ScaleQuest), and concept-conditioned prompt synthesis methods (PromptCoT-2.0).
• Comparison to Oracle: While providing baselines with "Oracle" concept selection (the best possible KC chosen by KT2) improved their performance significantly, KT4EQG still outperformed them. This indicates that the framework's alignment-trained generator produces higher-quality questions for the target concept than standard generators, even when the concept selection is perfect.
• Answerability: The generated questions maintained high answerability rates (comparable to the base LLM), demonstrating that the optimization for educational value did not compromise the solvability or quality of the questions.
• Ablation Studies:
  • Removing the RL alignment phase resulted in lower alignment scores ($R_{align}$), confirming the necessity of the reinforcement learning step.
  • Analysis of practice rounds showed that methods with effective concept selection (like KT4EQG) showed greater cumulative learning gains as the number of practice rounds increased, whereas methods with random or poor concept selection showed diminishing returns.

Significance and Claims

The paper claims that KT4EQG demonstrates the potential of integrating knowledge tracing with question generation to support personalization in educational systems.

• Pedagogical Benefit: The system generates questions that are both concept-aligned and beneficial for student learning, addressing the gap where existing EQG systems fail to model individual knowledge states.
• Scalability: The framework is shown to work effectively even on datasets without human-annotated tree structures (using LLM-constructed trees), suggesting broad applicability.
• Modest Scope: The authors position this work as a step toward fine-grained personalization. They acknowledge that while the framework improves question selection and generation, the ultimate measure of success is the student's learning progress, which is simulated here via knowledge tracing models. The work does not claim to replace human educators but rather to provide a tool for generating highly targeted practice materials.