Faster, Cheaper, More Accurate: Specialised Knowledge Tracing Models Outperform LLMs

1. Problem Statement

The paper addresses the critical challenge of predicting student performance in educational technology (EdTech) platforms. Specifically, the goal is to forecast whether a student will answer a future question correctly or incorrectly based on their historical interaction data.

While Knowledge Tracing (KT) models have long been the standard for this task due to their domain specificity, the rise of Large Language Models (LLMs) has prompted the question: Can general-purpose LLMs replace specialized KT models? The authors investigate whether LLMs can match the accuracy, latency, and cost-efficiency of KT models for this specific, high-volume, real-time prediction task.

2. Methodology

The authors conducted a systematic benchmark comparing specialized KT models against various LLMs across three dimensions: predictive performance, inference latency, and deployment cost.

Task Formulation

• Objective: Binary classification (Correct vs. Incorrect) for a student's next response.
• Input Data:
  • KT Models: Question IDs and student answer history.
  • LLMs: Question IDs, text, construct descriptions, and student history passed via prompt.
  • LLM KT (Hybrid): Uses a Qwen 0.6B embedding model to convert text (questions, misconceptions, explanations) into fixed vectors, which are then fed into a small, custom temporal transformer. Note: The LLM is only used for feature extraction offline; the inference model is a small transformer.
• Dataset: A real-world dataset from an online learning platform containing 12,800 training students and 1,600 validation students. Each student has a history of 50 questions, with the model tasked to predict the final 40.
• Evaluation Metrics:
  • Accuracy: Proportion of correct predictions.
  • F1 Score: Macro-averaged to handle class imbalance (correct vs. incorrect responses).

Models Evaluated

• Specialized KT Models:
  • DKT (Deep Knowledge Tracing): Recurrent Neural Network (RNN) based.
  • SAKT (Self-Attentive Knowledge Tracing): Transformer-based attention mechanism.
  • LLM KT (Ours): A custom encoder-decoder temporal transformer using pre-computed embeddings.
• General-Purpose LLMs:
  • Closed Source: GPT-4o-mini, Gemini-2.5-flash-lite.
  • Open Source: Llama-1B (zero-shot and LoRA fine-tuned), Qwen2.5-7B-Instruct.
• Baselines: Random guessing and "Dataset Bias" (predicting based on the overall correctness rate of the question).

Experimental Setup

• Infrastructure: Self-hosted models were tested on a CPU-only Azure instance (4 vCPUs, 14GiB RAM). Closed-source LLMs were accessed via API.
• Scale: Projections were made for 100,000 students, each receiving 40 predictions annually.
• Prompting: LLMs were given a constrained prompt requiring a "Yes/No" output to ensure deterministic parsing.

3. Key Contributions

1. Systematic Benchmarking: The first comprehensive comparison of specialized KT models vs. LLMs across accuracy, speed, and cost in an EdTech context.
2. Hybrid Architecture Proposal: Introduction of the "LLM KT" model, which leverages LLMs for offline text embedding but relies on a lightweight temporal transformer for online inference, avoiding the latency of running full LLMs during prediction.
3. Cost-Benefit Analysis: A detailed breakdown of the economic implications of deploying LLMs vs. KT models at scale, highlighting that LLMs are often economically unviable for high-frequency student prediction.

4. Results

A. Predictive Performance

• Specialized KT models outperformed all LLMs.
  • LLM KT (Hybrid): Achieved the highest accuracy (72.8%) and F1 score (0.674).
  • SAKT: 72.7% Accuracy, 0.669 F1.
  • DKT: 71.8% Accuracy, 0.650 F1.
• LLMs underperformed:
  • GPT-4o-mini: 58.6% Accuracy (worse than the naive dataset bias baseline of 66.5%).
  • Gemini-2.5-flash-lite: 66.5% Accuracy.
  • Qwen2.5-7B: 64.6% Accuracy.
  • Llama-1B (Zero-shot): 33.5% Accuracy (failed to follow prompt format).
  • Llama-1B (LoRA Fine-tuned): Improved to 71.0% accuracy but still lagged behind specialized KT models.
• Key Insight: General-purpose LLMs struggle to model individual student learning trajectories and misconceptions without explicit domain supervision, often failing to capture the statistical properties of student data (e.g., general improvement over time).

B. Latency (Inference Speed)

• KT Models: Extremely fast, with latencies < 0.25 seconds per student (sub-second).
• LLMs: Orders of magnitude slower.
  • GPT-4o-mini: ~3.1 seconds.
  • Gemini-2.5-flash-lite: ~128 seconds.
  • Qwen2.5-7B & Llama-1B: ~1,600 to 3,300 seconds (27 to 55 minutes) per student.
• Conclusion: LLMs are too slow for real-time, interactive educational platforms.

C. Deployment Cost

• Scale: 100,000 students × 40 predictions/year.
• KT Models: <$2.00 per year (due to low compute requirements and CPU deployment).
• LLMs:
  • GPT-4o-mini: ~$2,322.
  • Gemini-2.5-flash-lite: ~$1,230.
  • Llama-1B variants: ~$11,991.
  • Qwen2.5-7B: ~$24,741.
• Conclusion: KT models are 600x to 12,000x cheaper than LLMs for the same workload.

5. Significance and Conclusion

The paper concludes that specialized Knowledge Tracing models remain the superior choice for EdTech prediction tasks.

• Domain Specificity Matters: While LLMs excel at general reasoning and coding, they lack the fine-grained, data-driven efficiency required to model individual student knowledge states over time.
• Scalability: For platforms serving thousands of students, the latency and cost of LLMs make them impractical for real-time intervention.
• Recommendation: LLMs should not be adopted as a "universal solution" for student modeling. Instead, they should be used selectively (e.g., for generating explanations or feedback) where they demonstrably add value, while specialized KT models should remain the backbone for performance prediction and misconception detection.

The study serves as a strong caution against the "LLM for everything" trend in education, advocating for the continued development and deployment of lightweight, domain-specific architectures.