ConceptKT: A Benchmark for Concept-Level Deficiency Prediction in Knowledge Tracing

1. Problem Statement

Traditional Knowledge Tracing (KT) systems primarily focus on predicting the binary correctness of a student's answer (correct vs. incorrect) based on historical response data. While effective for tracking overall performance, these systems lack fine-grained diagnostic capabilities. They fail to identify the specific conceptual misunderstandings or deficiencies that lead to errors.

In real-world educational scenarios, knowing why a student is wrong is as critical as knowing that they are wrong. Without identifying specific missing concepts (e.g., confusing volume calculation with area calculation), instructional systems cannot provide targeted remediation or adaptive scaffolding. Furthermore, existing datasets often lack detailed traces of student reasoning steps and concept-level error annotations.

2. Methodology

A. Dataset Construction: ConceptKT

The authors constructed ConceptKT, a novel dataset extending the MathEDU dataset.

• Source: Derived from 4,048 solution process records from 6 students with diverse academic backgrounds (Math, Finance, Physics, etc.).
• Annotation: Three mathematics education experts annotated the data with:
  1. Associated Concepts: The specific concepts required to solve a problem (55 core concepts grouped into 13 categories).
  2. Missing Concepts: The specific concepts the student failed to demonstrate mastery of when answering incorrectly.
• Granularity: The dataset distinguishes between "conceptual deficiencies" (e.g., wrong mathematical operation) and "careless errors" (e.g., calculation slips), focusing annotation efforts on the former.
• Quality: Inter-annotator agreement (Fleiss' $\kappa$) was substantial (0.68 for associated concepts, 0.63 for missing concepts).

B. Task Formulation

The study formulates an extended KT task using In-Context Learning (ICL) with Large Language Models (LLMs) and Large Reasoning Models (LRMs).

• Input: A target question $q$ and a subset of the student's historical response history $H$.
• Output: The model must perform three steps via Chain-of-Thought (CoT) prompting:
  1. Concept Identification: Infer the concepts required for the target question.
  2. Student Modeling: Analyze the history to assess concept mastery.
  3. Prediction: Predict the answer correctness ($c$) and identify the set of missing concepts ($\gamma$) if the answer is wrong.

C. Response Selection Strategies

Since feeding entire student histories can introduce noise or redundancy, the authors investigated strategies for selecting the most informative historical records:

1. All Responses: Using the full history without filtering.
2. Same-Concept Only: Selecting only historical responses that involve the same concepts as the target question.
3. Conceptual Semantic Selection: A hybrid approach. First, filter for responses sharing concepts with the target. Then, if the count exceeds a limit ($m=30$), rank them by semantic similarity (using BERTScore) to the target question and select the top $m$.

3. Key Contributions

1. New Task Formulation: Introduced Concept-Level Deficiency Prediction, extending KT beyond binary correctness to identify specific conceptual gaps.
2. ConceptKT Dataset: The first expert-annotated dataset for KT that captures both required concepts and specific missing concepts in erroneous solutions, enabling fine-grained diagnosis.
3. Empirical Analysis of Selection Strategies: Demonstrated that conceptual alignment and semantic similarity are critical for effective in-context learning in KT, outperforming the use of raw, unfiltered history.

4. Experimental Results

Models Evaluated

The study evaluated various LLMs (e.g., Llama-4, Gemini-2.0-Flash) and LRMs (e.g., o3-mini, DeepSeek-R1) against traditional KT baselines (DKT, DKVMN, GKT, SAKT, OKT).

Key Findings

• Correctness Prediction: LLMs generally achieved high accuracy (e.g., o3-mini reached ~70.35%), often outperforming traditional deep learning baselines.
• Deficiency Prediction:
  • Filtering is Crucial: Using "All Responses" led to a significant drop in deficiency prediction performance (Macro-F1 scores as low as 2.16% for o3-mini).
  • Conceptual Alignment: Restricting input to "Same-Concept Only" responses drastically improved performance (e.g., DeepSeek-R1 improved to 17.40% Macro-F1).
  • Semantic Selection: The "Conceptual Semantic Selection" strategy yielded the best results for several models. For instance, o3-mini's deficiency prediction Macro-F1 jumped from 2.16% (All Responses) to 12.77% (Semantic Selection).
• Augmentation Analysis: When same-concept records were sparse, adding random or semantically similar records from different concepts degraded performance. This suggests that conceptual relevance is more important than input quantity; adding noise from unrelated concepts confuses the model's reasoning.

5. Significance and Implications

• Diagnostic Precision: The study proves that LLMs can effectively diagnose specific conceptual gaps if provided with the right context, moving KT from "black box" prediction to interpretable diagnosis.
• Efficiency in Context: It challenges the assumption that "more data is better." In KT, curated, conceptually aligned data is superior to massive, unfiltered histories, allowing smaller models to perform well even with limited context windows.
• Adaptive Education: By accurately predicting missing concepts, educational platforms can dynamically generate targeted remedial exercises, improving the efficacy of personalized learning interventions.
• Future Directions: The paper highlights that while promising, predicting deficiencies remains challenging due to the variability of student performance and the coarse-grained nature of current binary annotations (deficiency vs. no deficiency). Future work should explore multi-level mastery annotations and diverse prompting strategies.