Technical Summary: Towards Sustainable Learning in Online Education: A Reinforcement Learning Approach

1. Problem Statement

Online education faces a fundamental tension: while it offers unprecedented scalability, it struggles with low learner engagement, high dropout rates, and poor long-term knowledge retention. Existing approaches generally fall into two categories with distinct limitations:

• Predictive Models: These often focus on short-term objectives, such as maximizing the probability of success on the next learning module, but fail to optimize for long-term engagement or durable retention.
• Reinforcement Learning (RL) Models: While these shift focus to long-term rewards, they typically assume learners will persistently follow recommendations (ignoring dropout risks) and prioritize knowledge expansion over systematic review, neglecting the trade-off between acquiring new concepts and reinforcing prior knowledge.

The authors identify a need for a system that jointly optimizes sustained engagement (preventing dropout) and long-term effectiveness (ensuring knowledge retention), addressing the inherent trade-offs between pushing learners forward and allowing for necessary review.

2. Methodology: The AI-Tutor Framework

The paper proposes AI-Tutor, a personalized learning-path recommender system built on a Model-Based Reinforcement Learning (RL) framework. The system is designed to align with the dual goals of sustainable learning through three core technical components:

A. Knowledge Graph (KG) Representation

To handle the structural dependencies of knowledge, the authors construct a directed Knowledge Graph $G=(V, E)$ where nodes represent knowledge items and edges represent relationships.

• Edge Types: The graph integrates Curriculum-Driven Edges (expert-defined prerequisites) and Empirically-Driven Edges (inferred from historical data using association rule mining to identify latent support relationships).
• State Encoding: A Graph Neural Network (specifically Deep Graph Infomax) learns embeddings for both individual knowledge items and the learner's current knowledge state (represented as a subgraph of learned items).
• Action Space Masking: The RL agent's action space is constrained to nodes adjacent to the learner's current knowledge subgraph. This ensures recommendations are pedagogically appropriate (adhering to constructivist learning theory) and computationally efficient.

B. Learner Simulator (LearnSim)

To avoid the data inefficiency and safety risks of training RL agents directly on live users, the authors employ a Model-Based RL approach using a Learner Simulator (LearnSim).

• Architecture: LearnSim utilizes a Transformer architecture to process a learner's historical trajectory.
• Multi-Task Prediction: It simultaneously predicts three critical quantities for the RL training loop:
  1. Engagement Probability ($p_{t}^{eng}$): The likelihood the learner will persist after the current interaction.
  2. Recall Probability ($p_{t}^{rec}$): The likelihood the learner can recall the item before the interaction.
  3. Half-Life Strength ($h_{post}(t)$): The updated memory strength after the interaction, used to model the forgetting curve.
• Training: LearnSim is trained on historical data using binary cross-entropy losses for engagement/recall and a specialized loss function for half-life estimation based on future recall outcomes.

C. RL Policy Optimization with Cognitive Theory

The AI-Tutor agent uses an Actor-Critic framework with specific modifications to the standard Bellman equation to address the identified gaps:

1. Reward Function Design: The immediate reward $r_t$ is a weighted sum of two components grounded in cognitive theory:

   • Knowledge Acquisition Gain ($r_{t}^{acq}$): Based on the improvement in recall probability (1 - predicted pre-interaction recall).
   • Knowledge Retention Gain ($r_{t}^{ret}$): Based on the increase in half-life memory strength ($h_{post} - h_{pre}$), derived from the Forgetting Curve theory.
   • Formula: $r_t = r_{t}^{acq} + \beta r_{t}^{ret}$.

2. Engagement-Aware Value Estimation: The authors revise the classical Q-function to account for the stochastic nature of learner dropout. Instead of a fixed discount factor $\gamma$, they incorporate the dynamic engagement probability $p_{t}^{eng}$:

   • Modified Bellman Equation: $Q^\pi(s_t, a_t) = r_t + p_{t}^{eng} \mathbb{E}[Q^\pi(s_{t+1}, a_{t+1})]$.
   • This forces the agent to balance the potential future rewards against the risk of the learner disengaging, effectively acting as a "risk manager" for the learning path.

3. Key Contributions

The paper claims two primary contributions:

1. A Novel RL Framework for Sustainable Learning: Unlike prior work, this framework explicitly balances knowledge expansion with retention and incorporates behavioral engagement into the long-term reward estimation. It addresses the trade-off between academic progress and learner motivation.
2. Theory-Driven Model Design: The approach is among the first to embed domain theories (Constructivist learning theory for action masking; Forgetting Curve and Memory Strength theories for reward formulation) directly into the RL model formulation, rather than treating them as post-hoc heuristics.

4. Empirical Results

The model was evaluated on a dataset of 23.6 million learning records from 33,700 learners on the MaiMemo language learning platform (English vocabulary for the Graduate Entrance Examination).

• Baselines: The AI-Tutor was compared against:

  • Myopic Greedy: Prioritizes immediate difficulty.
  • FSRSv1/v2: Industry-standard spaced repetition systems (using HLR and LearnSim respectively).
  • DRL-SRS: A deep Q-learning approach for spaced repetition.
  • Ablation Variants: AI-Tutor without engagement modeling (v1), without KG masking (v2), and without retention rewards (v3).

• Performance Metrics:

  • Course Completion Rate: AI-Tutor achieved a 31.1% completion rate, significantly outperforming DRL-SRS (11.2%) and all other baselines. This represents a 177.7% increase over the best non-ablation baseline.
  • Final Exam Performance: AI-Tutor achieved a score of 55.9%, a 149.6% improvement over DRL-SRS (22.4%).
  • Learner Simulator: LearnSim outperformed standard baselines (Logistic Regression, Random Forest, Gradient Boosting) in engagement prediction (improving balanced accuracy by >35%) and memory prediction tasks.

• Ablation Insights:

  • Removing engagement modeling (AI-Tutorv1) caused the most significant performance drop, confirming the necessity of modeling dropout risk.
  • Removing KG-based action masking (AI-Tutorv2) led to higher dropout rates without learning gains.
  • Removing retention rewards (AI-Tutorv3) resulted in lower final exam scores, highlighting the importance of systematic review.

5. Significance and Claims

The paper concludes that AI-Tutor successfully advances the dual goals of sustainable learning in online education: effective long-term knowledge retention and sustained learner engagement.

• Human-Centric Adaptation: Learning path analyses reveal that AI-Tutor dynamically adapts to learner profiles (e.g., assigning lower difficulty and more review for low-performers) and balances challenge with motivation, preventing the "overload" seen in aggressive greedy strategies.
• Generalizability: The authors claim the framework is adaptable to other domains beyond language learning. They posit that any subject with a structured knowledge graph, periodic review cycles, and final assessments can utilize this framework by training a domain-specific LearnSim and AI-Tutor.
• Theoretical Alignment: The study demonstrates that integrating cognitive psychology theories (forgetting curves, constructivism) directly into RL reward structures and state representations yields superior practical outcomes compared to purely data-driven or rule-based approaches.