Cold Start Problem: An Experimental Study of Knowledge Tracing Models with New Students

Technical Summary: Cold Start Problem in Knowledge Tracing

Problem Statement
Knowledge Tracing (KT) aims to predict a student's knowledge state based on interactions within Intelligent Tutoring Systems (ITS). A critical barrier to effective personalization is the "cold start" problem, where models struggle to make accurate predictions for new students due to a lack of prior interaction data. While existing literature has addressed cold start scenarios regarding new questions, skills, or activities, a significant gap remains in evaluating how well KT models generalize to entirely new students. Traditional evaluation methods often train models on a student's initial interactions and test on their subsequent interactions, which fails to simulate the real-world scenario where a model encounters a learner with zero historical data.

Methodology
This study proposes a rigorous evaluation framework that simulates realistic cold start conditions by strictly segregating data. The researchers trained three prominent KT models exclusively on historical data from past students and evaluated their performance solely on five randomly selected sets of entirely new, unseen students. This approach ensures that the models possess no prior knowledge of the test students' performance history.

The study utilized three widely recognized datasets from the ASSISTments platform:

• ASSISTments 2009: Refined 'skill-builder' version with detailed knowledge component (KC) annotations.
• ASSISTments 2015: Lacks KC annotations and offers fewer attributes.
• ASSISTments 2017: Contains extensive metadata and 102 KCs.

The three models evaluated were:

1. Deep Knowledge Tracing (DKT): Uses Recurrent Neural Networks (LSTM) to capture temporal dependencies.
2. Dynamic Key-Value Memory Networks (DKVMN): Employs an external memory structure with keys (concepts) and values (mastery levels) for dynamic updates.
3. Self-Attentive Knowledge Tracing (SAKT): Utilizes attention mechanisms inspired by Transformers to model relationships among past interactions.

Performance was measured by tracking accuracy as the number of interactions increased (from 3 up to 20 questions, or 30 for ASSISTments 2017), allowing the authors to observe learning curves and adaptability.

Key Contributions

1. Realistic Evaluation Framework: The study introduces a methodology that trains and tests on completely disjoint student populations, providing a more accurate assessment of a model's ability to generalize to new learners compared to standard intra-student evaluation.
2. Comparative Analysis: It offers a systematic comparison of DKT, DKVMN, and SAKT specifically under cold start conditions across multiple datasets.
3. Empirical Evidence on Attention Mechanisms: The research provides data challenging the assumption that attention-based architectures inherently solve the cold start problem, highlighting that while they offer early gains, they still face significant limitations.

Results
The experimental results indicate that all three models initially struggle under cold start conditions but show progressive improvement as interaction data accumulates:

• DKT: Demonstrated steady improvement across all datasets. While initial accuracy was lower (ranging 0.45–0.60), it consistently rose to approximately 0.75 by the 20th question, showing strong long-term adaptability.
• DKVMN: Consistently performed well, achieving high initial accuracy and rapid improvement to over 0.75 in most conditions. Its memory-based architecture allowed for effective prediction even with limited data, often outperforming other models in the ASSISTments 2015 dataset.
• SAKT: Showed quick initial adaptation, often outperforming other models in the early stages (particularly in ASSISTments 2009 and 2015). However, its performance frequently plateaued in later stages, suggesting that attention mechanisms alone do not fully resolve the cold start challenge or sustain performance over extended sequences without sufficient data.

Significance and Claims
The paper claims that its findings underscore the necessity for KT models that can effectively generalize to new learners without relying on their specific historical data. The authors emphasize that while attention-based models like SAKT provide a boost in initial accuracy, they do not inherently mitigate the cold start problem. Consequently, the study highlights the importance of developing models robust in few-shot and zero-shot learning scenarios. The authors conclude that future work should focus on hybrid models that combine the rapid adaptation of memory-based or attention-based approaches with sustained learning capabilities, potentially leveraging techniques like transfer learning and federated learning to support diverse educational needs.