Technical Summary: Tracing Mathematical Proficiency Through Problem-Solving Processes

Problem Statement

Traditional Knowledge Tracing (KT) methods aim to model a student's knowledge state to predict future performance in Intelligent Tutoring Systems. However, existing approaches face fundamental limitations in explainability and depth. They primarily rely on outcome-centric supervision (binary correctness: correct/incorrect) and limited metadata (e.g., problem difficulty, response time, or knowledge concepts). This reliance on the final answer fails to capture the nuances of a student's actual understanding, as it treats the problem-solving process (PSP) as a "black box."

In contrast, real-world educators evaluate understanding through the problem-solving process, observing reasoning steps, intermediate computations, and strategic choices. Current KT datasets and models largely neglect this rich information, particularly in mathematical domains where capturing PSP at scale is challenging. Consequently, existing models struggle to provide interpretable insights into why a student succeeds or fails, and they often perform poorly in cold-start scenarios where little historical interaction data exists.

Methodology: KT-PSP and STATUSKT

To address these gaps, the authors propose a new task formulation and a corresponding framework.

1. Knowledge Tracing with Problem-Solving Process (KT-PSP)

The authors redefine the KT task to explicitly incorporate the student's problem-solving process into the interaction sequence.

• Traditional KT: Input sequence $S_{conv} = \{(q_t, c_t, r_t)\}$, where $r_t$ is binary correctness.
• KT-PSP: Input sequence $S_{new} = \{(q_t, c_t, r_t, p_t)\}$, where $p_t$ represents the detailed PSP (step-by-step reasoning, intermediate computations, etc.).
  The objective remains predicting future correctness $P(r_{t+1}=1 | \dots)$, but now leverages both historical performance and underlying solving strategies.

2. The KT-PSP-25 Dataset

To facilitate research in this new domain, the authors released KT-PSP-25, a novel dataset containing:

• 22,289 student-problem interactions from a tablet-based math platform.
• 1,343 students solving 2,696 problems across 490 knowledge components.
• Handwritten PSP: Unlike previous datasets, this includes the actual handwritten solution traces, which were processed via an OCR pipeline (using GPT-5) to convert images into structured text while preserving mathematical logic.
• Privacy: Student and question IDs were anonymized, and OCR outputs were refined to ensure data quality and privacy.

3. The STATUSKT Framework

The core contribution is STATUSKT, a process-aware KT framework that extracts Mathematical Proficiency (MP) signals from the raw PSP using a Teacher-Student-Teacher LLM pipeline. This pipeline converts unstructured text into structured, interpretable intermediate signals.

The pipeline operates in three stages:

1. Indicator Extraction (Teacher LLM): Given a problem text and its knowledge components, the Teacher LLM generates a set of MP indicators. These are specific questions targeting four observable dimensions of mathematical proficiency (based on Findell et al., 2001):
   • Conceptual Understanding (CU)
   • Strategic Competence (SC)
   • Procedural Fluency (PF)
   • Adaptive Reasoning (AR)
     (Note: Productive Disposition is excluded as it is difficult to assess via text.)
2. Response Generation (Student LLM): The Student LLM acts as a proxy for the actual student. It analyzes the student's handwritten PSP and generates responses to the MP indicators. Crucially, it preserves the student's original reasoning (including errors) rather than correcting them.
3. Proficiency Assessment (Teacher LLM): The Teacher LLM evaluates the Student LLM's responses against the original indicators to determine if the student demonstrated mastery of each specific dimension.

Integration:
The output of this pipeline is an MP Ratio for each dimension (the proportion of satisfied indicators). These ratios serve as auxiliary inputs to the KT backbone model. The model is trained with a composite loss function:
$$L = BCE(r_{gt}, r_{pred}) + \alpha \sum_{i \in P} MSE(m_{gt}^i, m_{pred}^i)$$
This dual-objective approach forces the model to learn both the final correctness and the underlying proficiency patterns, enhancing both predictive accuracy and interpretability.

Key Contributions

1. New Task Formulation (KT-PSP): A formal definition of KT that integrates problem-solving processes into the interaction sequence, enabling process-aware modeling.
2. KT-PSP-25 Dataset: The first large-scale mathematical KT dataset containing real-world, OCR-transcribed handwritten problem-solving processes for every interaction.
3. STATUSKT Framework: A novel architecture utilizing a three-stage LLM pipeline to distill raw PSP into interpretable MP signals, which are then integrated into standard KT models.
4. Empirical Validation: Comprehensive experiments demonstrating that STATUSKT outperforms existing Deep Learning KT (DLKT) baselines and provides fine-grained, interpretable insights into student learning.

Experimental Results

Experiments were conducted on KT-PSP-25 using various baseline architectures (DKT, DKVMN, SAKT, SAINT, AKT, etc.).

• Overall Performance (RQ1): STATUSKT consistently improved prediction performance (AUC and Accuracy) across most architectures compared to both standard baselines and a "Baseline+PSP" variant (which simply concatenated raw text embeddings). The authors note that raw text embedding alone often introduces noise or fails to improve performance, whereas the structured MP distillation in STATUSKT effectively leverages the process information.
• Cold-Start Robustness (RQ2): In scenarios with limited data (only the first 5 interactions per student), STATUSKT significantly outperformed baselines. The MP signals provided informative priors that stabilized knowledge state estimation when historical correctness data was scarce.
• MP Effects Analysis (RQ3): Case studies revealed that MP signals help correct predictions in "MP-help" cases by aligning correctness with specific underlying skills. However, the authors also identified "MP-hurt" cases where over-enriched or misaligned indicators could degrade performance, highlighting the importance of scope control in indicator generation.

Significance and Claims

The paper claims that incorporating the reasoning reflected in students' problem-solving processes is essential for advancing KT toward more accurate, interpretable, and pedagogically meaningful modeling.

• Interpretability: Unlike "black box" predictions, STATUSKT provides explicit proficiency signals (e.g., "Student has high Procedural Fluency but low Adaptive Reasoning"), offering fine-grained insights into learning progress.
• Pedagogical Alignment: The framework mimics the way human teachers evaluate students—by looking at the process, not just the answer—thereby bridging the gap between automated systems and educational theory.
• Limitations: The authors modestly acknowledge limitations, including the dataset's relatively small scale, the restriction to the mathematics domain, and the reliance on OCR which may introduce recognition errors or lose visual reasoning elements (diagrams). They do not claim the framework is universally applicable to all subjects without further adaptation.