Technical Summary: Incorporating Feedback into Elo-Based Knowledge Tracing for Iterative Problem Solving

Problem Statement

In digital learning environments, learners often solve complex tasks through iterative attempts supported by automated feedback. While standard Elo Rating Systems (ERS) are widely used for estimating learner ability and item difficulty due to their computational efficiency and scalability, they suffer from a critical limitation in these contexts: they treat all successful outcomes equivalently. Consequently, ERS fails to distinguish between independent success and success achieved with assistance. This oversight can lead to biased knowledge estimates, as a learner's performance may reflect the quality of feedback received rather than their underlying competence. The paper addresses the challenge of accurately modeling learner knowledge when automated feedback is an integral part of the iterative problem-solving process.

Methodology

Data and Context

The study utilizes a dataset of 47,224 learner-item interactions from 296 university students using the LabNBook platform. The task involves designing an acid-base titration protocol, a complex activity requiring the selection, parameterization, and ordering of actions. The system employs Constraint-Based Modeling (CBM) to evaluate submissions against 42 expert-defined constraints (items). Feedback is provided after erroneous submissions, targeting specific errors with varying levels of assistance (encoded as an ordinal variable from 0 to 5, where 0 is no feedback and 5 is the explicit correct solution).

Modeling Framework

The authors propose a hierarchy of models, moving from a standard baseline to feedback-aware extensions.

1. Baseline Models:

   • Standard ERS (ST): Models learner knowledge ($\theta_k$) and item difficulty ($d_i$) using a single global parameter per learner and a standard logistic function.
   • Enhanced ERS (EN): Serves as the primary baseline for comparison. It improves upon ST by:
     • Task-Type Specific Knowledge: Replacing global knowledge with parameters specific to task types (knowledge components).
     • Dynamic Update Rates: Introducing a decay factor for item difficulty updates to reflect decreasing uncertainty as data accumulates.
     • Prior Knowledge Estimation: Boosting the update rate during initial interactions to better estimate prior knowledge.

2. Feedback-Aware Extensions:
   The paper introduces two distinct models to incorporate feedback, both distinguishing between unassisted ($l=0$) and assisted ($l>0$) attempts:

   • New Item (NI) Model: Operates on the assumption that feedback fundamentally changes the nature of the task. It treats every $(item, feedback\_level)$ pair as a distinct item. The item space expands from $I$ to $(L+1) \times I$, where $L$ is the maximum feedback level. Each variant has its own difficulty parameter ($d_{i,l}$).
   • Probability Modifier (PM) Model: Operates on the assumption that feedback does not alter the task itself but increases the probability of success in a graded manner. It uses a binary distinction for the item space (unassisted vs. assisted) but introduces a level-dependent offset ($c_l$) to the success probability function. The probability is modeled as $\sigma(\theta_{k,t(i)} - d_{i,f(l)} + c_l)$, where $c_l$ represents the shift in success probability due to feedback level $l$.

Evaluation Strategy

The models were evaluated using Root Mean Square Error (RMSE) and Area Under the ROC Curve (AUC). To ensure robust conclusions, the authors employed three complementary evaluation strategies:

• Robustness to Learner Variation: A bootstrap procedure sampling learner trajectories to test stability under population perturbations.
• Consistency Across Learners: A Wilcoxon signed-rank test to determine if one model consistently outperforms another at the individual learner level.
• Generalization to Unseen Learners: A hold-out test where models trained on half the learners are evaluated on the other half to assess generalization capabilities.

Key Results

• Predictive Accuracy: Both feedback-aware models (NI and PM) significantly outperformed the feedback-agnostic Enhanced (EN) model and the Standard (ST) model across all evaluation metrics.
  • Compared to EN, the NI and PM models achieved RMSE gains of approximately 5.2% to 5.5% in robustness and generalization tests.
  • The confidence intervals for performance differences were strictly positive, and Wilcoxon p-values were extremely small ($< 10^{-20}$), indicating statistically significant and consistent improvements.
• Model Dynamics:
  • The NI model demonstrated that treating feedback levels as distinct items leads to more stable convergence of difficulty estimates, likely because the $(item, feedback\_level)$ pairs are more homogeneous in terms of difficulty.
  • The PM model achieved comparable performance to NI with a more compact parameterization (fewer difficulty parameters), though it required estimating additional hyper-parameters ($c_l$) for the probability shifts.
  • Both models showed that feedback-aware approaches require relatively little data to outperform baselines, with performance gaps appearing after only a few hundred predictions.
• Stability and Discrimination: Crucially, the improvements in predictive accuracy were achieved without sacrificing the interpretability of the models. Both NI and PM preserved meaningful discrimination between learner knowledge profiles and produced stable item difficulty estimates, which are essential for adaptive learning systems.

Significance and Claims

The paper claims that explicitly modeling feedback is essential for accurate knowledge tracing in iterative learning environments. The authors argue that ignoring the distinction between independent success and assisted success leads to biased estimates of learner competence.

The primary contribution is the demonstration that simple, lightweight extensions to the Elo Rating System can effectively capture the nuances of feedback-rich learning. By proposing the New Item (NI) and Probability Modifier (PM) models, the study provides computationally efficient alternatives to more complex deep learning approaches or Bayesian Knowledge Tracing (BKT) extensions. The findings suggest that:

1. Feedback should not be treated as a mere post-hoc correction but as a fundamental variable that alters the learning condition.
2. It is possible to integrate feedback into ERS while maintaining the system's scalability and real-time adaptability.
3. The choice between modeling feedback as a task transformation (NI) or a probability shift (PM) depends on the specific trade-off between model expressiveness and complexity, as both yielded highly comparable empirical results in this context.

The authors conclude that these approaches offer a compelling alternative for educational data mining, bridging the gap between the need for sophisticated modeling of instructional support and the practical constraints of scalable, real-time adaptive systems.