Early Prediction of Student Performance Using Bayesian Updating with Informative Priors Across Cohorts

1. Problem Statement

In higher education, early warning systems (EWS) rely on predictive models to identify at-risk students using digital trace data. However, a critical limitation exists: models trained on a single cohort often suffer from performance degradation when transferred to subsequent cohorts due to shifts in student populations, instructional design, or behavioral patterns.

• The Gap: Traditional approaches typically rebuild models from scratch for each new cohort, ignoring accumulated empirical knowledge.
• The Challenge: There is a need for a modeling framework that can formally incorporate information from previous iterations (stabilizing estimates when current data is scarce) while remaining flexible enough to adapt to new observations.
• Objective: To evaluate whether Bayesian updating with informative priors derived from a source cohort can improve the accuracy and robustness of early performance predictions in a target cohort compared to models using uninformative (default) priors.

2. Methodology

2.1 Data and Context

• Setting: A blended first-year mathematics course for business administration majors at a German university.
• Cohorts: Two consecutive winter semesters:
  • Source Cohort (WS1): $N=307$ students (2022/2023).
  • Target Cohort (WS2): $N=323$ students (2023/2024).
• Data Source: Digital trace data from the Learning Management System (LMS) and online exercise platforms.
• Sample Restriction: Only students taking the first exam date were included to ensure a consistent timeline for week-by-week prediction.

2.2 Feature Variables (Predictors)

Six weekly engagement indicators aligned with Self-Regulated Learning (SRL) theory were extracted:

1. Answered Ratio: Ratio of answered MCQs in solution videos.
2. Correct Ratio: Ratio of correctly answered MCQs.
3. First Access to Deadline: Days between first access and submission deadline.
4. Points Ratio: Ratio of correctly solved problems in online exercises.
5. Total Time: Time spent on weekly exercises.
6. Worked-on Ratio: Ratio of attempted problems.

• Note: Features were not accumulated; models used data available up to the current week $t$. Missing values were imputed as zero (indicating non-engagement).

2.3 Outcome Variables

1. Binary (At-Risk): Grade $\ge 3.3$ (German scale, approx. C-). Threshold chosen to capture students with poor performance, not just those failing.
2. Continuous: Exam points (0–83).
3. Ordinal: Final grades (1.0 to 5.0, encoded as 1–11).

2.4 Modeling Approach: Bayesian Updating

The study employed a Bayesian regression framework using the brms package in R (Hamiltonian Monte Carlo).

• Source Cohort (WS1): Models were fitted using uninformative default priors to generate posterior distributions for parameters ($\beta$).
• Target Cohort (WS2):
  • Baseline: Fitted with uninformative default priors.
  • Experimental: Fitted with informative priors derived from the WS1 posterior means.
  • Prior Specification: The prior for parameter $\beta_i$ was modeled as $N(\mu_i, \eta \cdot \sigma_i)$, where $\mu_i$ and $\sigma_i$ are the mean and SD from WS1, and $\eta$ is a scaling factor ($\eta \in \{0.5, 1.0, 1.5, 3.0, 6.0\}$) to tune prior strength.
• Regression Types:
  • Linear Regression: For exam points.
  • Logistic Regression: For binary at-risk classification.
  • Ordinal Regression: For grade level prediction.

2.5 Evaluation

• Metric: Weekly evaluation (Weeks 2–10) using Accuracy (ACC), Sensitivity (SENS), and Root Mean Squared Error (RMSE).
• Goal: Identify the earliest week where reliable prediction is possible and quantify the improvement gained by informative priors.

3. Key Results

3.1 Source Cohort (WS1) Stability

• Predictive performance stabilized by Week 6.
• Logistic Regression: Accuracy plateaued >0.66; Sensitivity peaked at 0.81 in Week 13.
• Ordinal Regression: Accuracy reached 0.70 in Weeks 9 and 15.
• Linear Regression: RMSE stabilized around 13.15 points by Week 6.

3.2 Target Cohort (WS2) Performance with Informative Priors

The incorporation of priors from WS1 significantly altered early prediction capabilities:

• Logistic Regression (Binary Classification):

  • Early Gain: In Week 3, informative priors reduced the misclassification rate by 22.22% and the false negative rate by 38.46% compared to the default prior.
  • Optimal Scaling: Best results often achieved with scaling factors $\eta = 1.0$ to $1.5$.
  • Diminishing Returns: Benefits decreased in later weeks (Weeks 9–10) as current cohort data accumulated, sometimes leading to higher error rates if priors were too strong.

• Ordinal Regression (Grade Prediction):

  • Strongest Improvement: Showed the most dramatic early gains.
  • Week 2: Misclassification rate improved by 42.5% (scaling $\eta=0.5$), raising accuracy from a random guess (0.51) to 0.72.
  • Week 4: Accuracy reached 0.77 with a 20.83% reduction in misclassification.
  • Comparison: Ordinal regression outperformed logistic regression in early weeks when using informative priors, likely because it leverages the full grade structure to estimate the latent performance dimension.

• Linear Regression (Exam Points):

  • No Benefit: Informative priors provided no improvement and often degraded performance (increased RMSE).
  • Reason: The posterior distributions for the linear model parameters in WS1 and WS2 were nearly congruent (overlapping significantly). The data likelihood in WS2 was strong enough that the prior added no value, and conflicting priors introduced noise.

4. Key Contributions

1. Methodological Advancement in Learning Analytics:

   • Demonstrates that Bayesian updating is a viable strategy for cross-cohort prediction, effectively mitigating the "cold start" problem in new semesters.
   • Provides a practical blueprint for scaling prior strength ($\eta$) to balance historical knowledge with new data.

2. Early Intervention Capability:

   • Proves that reliable classification of at-risk students is possible as early as Week 2 or 3 (vs. Week 6 for standard models) when using informative priors.
   • This allows for proactive interventions before poor performance becomes irreversible on graded assessments.

3. Privacy-Preserving Knowledge Transfer:

   • The approach transfers knowledge via parameter distributions rather than individual student records, addressing data privacy concerns while retaining the predictive signal of past cohorts.

4. Model Specification Insights:

   • Highlights that Ordinal Regression benefits more from informative priors than Logistic or Linear models in this context, suggesting that modeling the full grade spectrum provides a more robust signal for early prediction.

5. Significance and Limitations

Significance:
This study shifts the paradigm from "rebuilding models from scratch" to "cumulative knowledge building." It validates that digital trace data, when processed through Bayesian frameworks, can support robust, early-warning systems that adapt to new cohorts without requiring extensive new data collection or sensitive demographic data.

Limitations & Future Work:

• Long-term Validity: The study only covers two cohorts. Future research must investigate how many iterations priors remain valid before course changes (pedagogy, instructor) render them obsolete.
• Sample Bias: The exclusion of students taking the second exam date (who may be higher risk) might lead to optimistic performance estimates.
• Linear Model Failure: The lack of benefit for linear models suggests that informative priors are most effective for classification tasks where the signal-to-noise ratio is lower in early weeks.

Conclusion:
Bayesian updating with informative priors is a powerful tool for educational data science, enabling institutions to deploy accurate, early-warning systems in the first few weeks of a semester by leveraging historical data, thereby facilitating timely and targeted pedagogical interventions.