Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

Technical Summary: Cross-Course Generalizability of SRL-Aligned Predictive Models Using Digital Learning Traces

1. Problem Statement

Despite the proliferation of early-warning systems in higher education, two critical gaps persist in the field of predictive learning analytics:

• Generalizability Gap: Most predictive models are trained and validated within a single course or institution. There is little empirical evidence regarding whether these models can be transferred to different courses (with different pacing/designs) or different institutions (with different student populations).
• Calibration Gap: While most research focuses on discrimination (the ability to separate high-risk from low-risk students, e.g., AUC), they often neglect calibration (whether the predicted probability of risk matches the actual observed frequency). Miscalibration can lead to inequitable or inefficient resource allocation during student interventions.

The study specifically targets the high-dropout STEM disciplines (specifically theoretical computer science) where students face intense cognitive demands and require effective self-regulation.

2. Methodology

The researchers employed a multi-university, multi-course dataset to test the robustness of predictive models.

• Data & Context: Digital trace data (LMS logs and supplemental practice system logs) were collected from three undergraduate theoretical computer science courses at two German universities.
  • Course A (TCS-A) & Course B (Logic-A): Same university, different topics.
  • Course B (Logic-A) & Course C (Logic-B): Same topic, different universities.
• Theoretical Framework: Features were constructed based on Self-Regulated Learning (SRL) theory, grouping low-level digital events (clicks, submissions, timestamps) into higher-level constructs: Task Engagement, Time Management, Post-completion Review, and Resource Access.
• Data Aggregation Strategies:
  1. Progressive Aggregation: Cumulative means and standard deviations of behaviors calculated weekly (e.g., Week 1, Weeks 1–2, etc.).
  2. Early-Reset Aggregation: A "frozen" summary of the first four weeks (critical for early intervention) followed by a new cumulative window from Week 5 onwards.
• Modeling Pipeline:
  • Algorithms: Elastic Net (regularized linear), Random Forest (non-linear ensemble), and XGBoost (gradient boosting).
  • Validation: In-sample (within-course) and out-of-sample (cross-course/cross-institution) validation.
  • Thresholding: Compared using the training course's threshold vs. recalibrating the threshold to match the target course's at-risk prevalence.
  • Calibration: Applied Platt scaling to correct predicted probabilities.

3. Key Contributions

• SRL-to-Feature Mapping: Provides a rigorous methodology for translating raw digital traces into theoretically grounded SRL indicators, moving beyond "black-box" feature engineering.
• Generalizability Framework: Establishes a systematic way to test model portability across both topical and institutional boundaries.
• Calibration Integration: Demonstrates the necessity of assessing probability calibration for the practical, ethical deployment of early-warning systems.

4. Results

• Predictive Feasibility: Early identification (within 2–4 weeks) is highly feasible. In-sample AUCs were high, particularly for Random Forest and XGBoost.
• The Generalizability Paradox:
  • Ensemble models (RF/XGBoost) showed near-perfect in-sample accuracy but failed to generalize well to different institutions, especially when the "base rate" (prevalence of at-risk students) differed.
  • Elastic Net proved to be the most robust and stable model for cross-context generalization, despite having lower peak in-sample accuracy.
• Thresholding & Base Rates: Poor generalization was often a result of "uninformed thresholding." Adjusting the classification threshold to match the target course's at-risk rate significantly improved performance.
• Feature Stability: While specific weekly predictors varied, the SRL-aligned feature families (e.g., early task initiation, consistent engagement, and effort regulation) were consistently identified as the most important predictors across all courses.
• Calibration Outcomes: Platt scaling improved probability accuracy within the same institution but could not fully correct for massive prevalence disparities when transferring across universities.

5. Significance

This study provides a cautionary but constructive roadmap for the deployment of AI in education. It concludes that:

1. Models are context-bound: High accuracy in one course does not guarantee reliability in another. Institutions should prioritize local validation before deploying transferred models.
2. Theory over Raw Data: Generalizability is more likely to be achieved by focusing on theoretically grounded behavioral patterns (SRL) rather than specific, idiosyncratic digital events.
3. Equity through Calibration: For early-warning systems to be "trustworthy" and "equitable," they must be calibrated to the specific population they serve to avoid over-identifying or under-identifying at-risk students.