🤖 AI

Temporal Smoothness Doubly Robust Learning for Debiased Knowledge Tracing

This paper proposes the Temporal Smoothness Doubly Robust (TSDR) framework, which integrates a propensity model with an error imputation model and a temporal smoothness regularizer to effectively address selection bias and mitigate variance-induced training instability in Knowledge Tracing, thereby achieving unbiased and stable mastery estimation.

Original authors: Peilin Zhan, Wei Chen, Weilin Chen, Shuyi Pan, Ruichu Cai

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Peilin Zhan, Wei Chen, Weilin Chen, Shuyi Pan, Ruichu Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a student how to solve math problems. You have a notebook full of their homework, but there's a catch: the student only wrote down the answers to the questions they felt confident about. They skipped the hard ones because they were too scary, and they skipped the easy ones because they were too boring.

If you try to guess how good the student really is just by looking at that notebook, you'll get it wrong. You might think they are a genius (because they only did the easy stuff) or a failure (because they only struggled with the hard stuff). This is the core problem the paper tackles: Knowledge Tracing (the AI that guesses what a student knows) is usually trained on "skewed" data where students skip questions based on their own feelings, not randomly.

Here is how the authors fixed this, explained with simple analogies.

1. The Problem: The "Skipped Homework" Trap

In the real world, students don't answer every question. They choose.

If a question is too hard, they skip it (thinking, "I'll fail anyway").
If a question is too easy, they skip it (thinking, "This is a waste of time").

Standard AI models look at the remaining data and say, "Okay, this student is great at easy stuff and bad at hard stuff." But the AI is actually just learning which questions the student chose to skip, not what the student actually knows. This leads to a broken feedback loop: the AI recommends the wrong next questions, the student gets frustrated, skips more, and the AI gets even more confused.

2. The Solution: The "Double-Check" System (Doubly Robust)

To fix this, the authors used a method called Doubly Robust (DR) Learning. Think of this like a detective trying to solve a crime where some witnesses are missing.

The detective uses two different tools to guess what the missing witnesses would have said:

The "Propensity" Tool: This guesses why a student skipped a question. (e.g., "Oh, they skipped this because it looked too hard.")
The "Imputation" Tool: This guesses what the answer would have been if the student had actually tried. (e.g., "If they had tried, they probably would have gotten it right.")

The Magic: The "Double Robust" part means the system is safe even if one of these tools is wrong.

If the "Why they skipped" tool is perfect, the guess is accurate.
If the "What they would have answered" tool is perfect, the guess is accurate.
You only need one of them to be right to get a fair result.

3. The New Problem: The "Wobbly Bridge"

The authors realized that while this "Double-Check" system is fair, it can be unstable.

Imagine trying to cross a bridge made of rubber. If you step on it, it wobbles. If the "What they would have answered" tool makes a small mistake, that wobble gets bigger and bigger as the student answers more questions. In a long sequence of learning, these small wobbles (variances) pile up, causing the AI to panic and learn the wrong things.

4. The Fix: The "Smoothness" Constraint (TSDR)

To stop the bridge from wobbling, the authors added a rule called Temporal Smoothness.

Think of a student's knowledge like a smoothly growing tree. A tree doesn't jump from being 1 foot tall to 10 feet tall in one second; it grows gradually.

Without the rule: The AI might think a student went from "knowing nothing" to "knowing everything" instantly because of a weird data glitch.
With the rule: The AI is forced to say, "Wait, knowledge grows slowly. Even if the data looks weird, the student's ability probably didn't change that drastically in one second."

By forcing the AI to believe that knowledge changes smoothly over time, they stopped the "wobbles" from getting out of control.

5. The Result: A Better Teacher

The authors combined these ideas into a framework called TSDR (Temporal Smoothness Doubly Robust).

What they did: They took existing AI models (the "backbones") and plugged this new "Double-Check + Smoothness" system into them.
What happened: They tested it on 9 different real-world datasets (like math, coding, and language learning) and some fake data designed to be very tricky.
The outcome: The new system consistently made the AI smarter. It corrected the bias caused by students skipping questions, leading to more accurate predictions of what students actually know.

Summary

The paper says: "Students skip questions for reasons that trick our AI. We built a system that uses two different ways to guess the missing answers (so it's fair), and we added a 'smoothness' rule to stop the AI from getting confused by small errors. This makes the AI a much better teacher."

Technical Summary: Temporal Smoothness Doubly Robust Learning for Debiased Knowledge Tracing

Problem Statement
Knowledge Tracing (KT) aims to model a student's dynamic knowledge state to predict future performance. However, existing deep learning-based KT models often suffer from systematic deviations from true knowledge states. The paper identifies the root cause as selection bias arising from the Missing Not At Random (MNAR) nature of educational data. In real-world scenarios, students strategically skip questions they perceive as too difficult or too easy, and recommendation systems often filter content based on estimated ability. This results in observed logs that are skewed, causing standard empirical risk minimization to conflate genuine knowledge dynamics with interaction patterns (e.g., skipping behavior). Consequently, models learn biased mastery estimates, which trigger biased recommendations in adaptive systems, creating a feedback loop that exacerbates the bias.

Methodology
The authors propose the Temporal Smoothness Doubly Robust (TSDR) framework, which integrates causal inference principles with sequential learning constraints.

Doubly Robust (DR) Formulation:
The core of the approach is a DR estimator that combines two models to correct for selection bias:
- Propensity Model: Estimates the probability of an interaction being observed (selection probability).
- Error Imputation Model: Estimates the prediction error for unobserved (counterfactual) interactions.
  The DR estimator is theoretically guaranteed to be unbiased if either the propensity model or the imputation model is accurate. The loss function is constructed as the sum of the imputed error and a correction term weighted by the inverse propensity.
Theoretical Analysis and Variance Control:
The authors derive a generalization bound for the DR estimator in the sequential KT setting. They identify that while the bias term is controlled by the product of model errors (ensuring double robustness), the variance term is dominated by the squared error of the imputation model. In sequential settings, inaccurate imputation leads to stochastic deviations that accumulate over time, causing training instability and error amplification.
Temporal Smoothness Constraint:
To mitigate the variance and stabilize training, the authors introduce a temporal smoothness regularizer. Drawing on the Power Law of Practice, which suggests skill acquisition follows a smooth trajectory rather than abrupt changes, this constraint penalizes large deviations in the latent states of the imputation model between consecutive time steps. This effectively tightens the generalization bound by reducing the Rademacher complexity of the latent trajectory.
Joint Learning Paradigm:
The framework employs a joint learning strategy where the KT predictor, propensity model, and imputation model are co-optimized via an alternating strategy. This allows the models to mutually regularize each other, ensuring that the imputation model remains accurate enough to support the unbiased estimation of the KT backbone.

Key Contributions

Causal Perspective on KT Bias: The paper reframes KT as a counterfactual estimation problem, explicitly modeling the data generation process to address selection bias induced by non-random exercise recommendations and student choices.
TSDR Framework: A model-agnostic framework that integrates DR estimation with temporal smoothness. It is designed to be "plug-and-play," capable of enhancing various state-of-the-art KT backbones (e.g., DKT, AKT, SparseKT).
Theoretical Justification: The authors provide a generalization bound that explicitly characterizes the impact of estimator variance and identifies temporal smoothness as a critical factor for controlling stochastic deviations in sequential learning.
Extensive Empirical Validation: The framework is evaluated on 9 real-world benchmark datasets and synthetic datasets with controllable MNAR levels.

Experimental Results

Performance Enhancement: TSDR consistently improves the performance (measured by AUC, ACC, and RMSE) of six different SOTA KT baselines across nine diverse datasets. Notable improvements were observed on datasets with high sparsity or strong student autonomy (e.g., a 5.06% AUC increase for AKT on the Prob dataset).
Robustness to Bias: In synthetic experiments where the degree of MNAR bias ( $\gamma$ ) was varied from 0 to 0.999, TSDR-equipped models maintained superior performance compared to baselines, particularly in reducing error (RMSE) under high-bias conditions.
Stability and Sensitivity: The method demonstrated robustness to the hyperparameter controlling the smoothness constraint ( $\lambda$ ), achieving stable performance within a range of $[0.3, 1.0]$ .
Ablation Studies: Experiments confirmed that the combination of joint learning and temporal smoothness applied specifically to the imputation branch yields the best results, outperforming variants that apply smoothness directly to the KT backbone or omit joint learning.

Significance and Claims
The paper claims that TSDR offers a principled, causally-grounded paradigm for learning from observational educational data. By addressing the systemic selection bias inherent in MNAR data, the framework enables models to learn more reliable knowledge states rather than merely fitting observed behaviors. The authors emphasize that while the joint optimization introduces additional training overhead, this cost is confined to the offline training phase and does not compromise online inference efficiency. Ultimately, TSDR is presented as a vital step toward more reliable and fair intelligent tutoring systems, demonstrating that principled bias correction is essential for the generalization capabilities of knowledge tracing models.