💻 computer science

Calibrated Explainable AI for Student Risk Prediction with Resource-Constrained Intervention and Machine Unlearning

This paper presents a privacy-preserving, resource-constrained framework for early student risk prediction that integrates calibrated XGBoost models, SHAP-based explainability, and a budget-aware agentic intervention layer to ensure accurate, interpretable, and compliant decision-making while supporting machine unlearning.

Original authors: Essa E. Almazroei

Published 2026-07-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Essa E. Almazroei

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A Smart, Private, and Budget-Friendly Student Safety Net

Imagine a university is like a massive, bustling airport. Thousands of students (passengers) are trying to reach their graduation (the destination). The problem is, some students are getting lost, missing their flights, or deciding to quit the journey entirely.

Traditionally, schools have used "predictive models" to spot these struggling students. Think of these old models as weather forecasters who can tell you, "It might rain tomorrow." But they stop there. They don't tell you what to do about the rain, they don't tell you how much rain to expect (just "rain" or "no rain"), and they don't care if you only have one umbrella to share among 1,000 people.

This paper introduces a new system that acts like a Smart Airport Control Tower. It doesn't just predict the storm; it tells the tower exactly who needs an umbrella, how to hand them out when supplies are low, and how to do it all while respecting everyone's privacy.

The Core Components

1. The "Crystal Ball" (Predictive Modeling)

The researchers used data from the Open University (OULAD), which tracks things like how often students click on course materials, when they submit assignments, and their grades.

The Analogy: Imagine trying to guess if a student will fail by looking at their "footprints" in the digital sand. Did they walk past the library often? Did they stop walking halfway through the semester?
The Tool: They tested several "crystal balls" (machine learning algorithms). They found that XGBoost was the best one. It's like a super-smart detective that notices complex patterns other detectives miss. It can tell you, "This student is 85% likely to struggle," rather than just saying "Yes, they will fail."

2. The "Calibrated Compass" (Probability Calibration)

Old models often give you a number that feels right but isn't accurate. If a model says "80% risk," it might actually mean "60% risk." That's dangerous for decision-making.

The Analogy: Imagine a speedometer in a car that says you are going 60 mph, but you are actually going 40. You might drive too fast because you trust the gauge.
The Fix: This paper "calibrates" the model. It adjusts the speedometer so that when it says "80%," it truly means there is an 80% chance of trouble. This ensures the school trusts the numbers enough to act on them.

3. The "Translator" (Explainable AI / XAI)

Usually, AI is a "black box." You put data in, and a result comes out, but no one knows why.

The Analogy: If a teacher asks, "Why did the computer say this student is at risk?" a black box says, "Because the computer said so."
The Fix: This system uses SHAP (a tool that acts like a translator). It breaks down the decision: "The computer flagged this student because they haven't clicked on any videos in 10 days AND they missed their last two quizzes." Now, the school knows exactly what to fix.

4. The "Triage Nurse" (Agentic Decision Layer)

Knowing a student is at risk is useless if you don't know what to do.

The Analogy: In a hospital emergency room, a triage nurse doesn't treat everyone the same. They look at the severity of the injury and decide: "This patient needs surgery immediately," "This one needs a bandage," and "This one just needs to wait."
The Fix: The system uses rules to turn the risk score into an action.
- High Risk + No Activity: "Send a mentor immediately."
- Medium Risk: "Send a motivational email."
- Low Risk: "Just keep an eye on them."

5. The "Limited Umbrella" (Budget-Constrained Intervention)

Schools don't have infinite money or staff. They can't help every single student who is struggling.

The Analogy: Imagine you have 10 umbrellas but 100 people are getting wet. You can't give everyone one. You have to decide who gets the umbrella first.
The Fix: The system ranks students by how much they need help. If the school has a budget for 50 interventions, the system picks the top 50 students who are most likely to succeed if helped. This ensures the limited resources go to the people who need them most.

6. The "First Day" Problem (Cold Start)

What about a brand new student who just enrolled? The system has no history of their clicks or grades yet.

The Analogy: A doctor trying to diagnose a patient who walked in the door 5 minutes ago with no medical history.
The Fix: The researchers built a special "Cold Start" model. It uses very basic info (like demographics and how they signed up) to make a rough guess early on. It's not as accurate as the full model, but it's better than guessing nothing, allowing for early help.

7. The "Right to be Forgotten" (Machine Unlearning)

Privacy is huge. If a student says, "Delete my data," the school must do it. But usually, deleting data forces the AI to relearn everything from scratch, which is slow and expensive.

The Analogy: Imagine a chef who memorized a recipe using ingredients from 100 farms. If one farm says, "Remove our tomatoes from the recipe," the chef usually has to re-memorize the whole book.
The Fix: This system uses "Machine Unlearning." It's like the chef being able to surgically remove just the tomato instruction without forgetting how to cook the rest of the dish. The paper shows that even after deleting data, the system's accuracy barely dropped (by less than 0.05%). It stays smart even after "forgetting."

The Results in Plain English

The researchers tested this "Smart Control Tower" system and found:

It's Accurate: It predicts student risks better than previous methods (about 94% accuracy).
It's Honest: The probability numbers are trustworthy.
It's Fair: It helps schools spend their money wisely by targeting the students who need it most.
It's Private: It can remove student data without breaking the system.

Summary

This paper isn't just about building a better crystal ball; it's about building a complete safety system. It predicts who is in trouble, explains why, decides what to do, figures out how to do it with limited money, and respects privacy by allowing data to be deleted without crashing the whole system. It turns raw data into a practical, human-friendly plan to help students succeed.

Technical Summary: Calibrated Explainable AI for Student Risk Prediction with Resource-Constrained Intervention and Machine Unlearning

Problem Statement

Current methods for predicting at-risk students in higher education, particularly within Massive Open Online Courses (MOOCs) and Virtual Learning Environments (VLEs), suffer from significant limitations. While existing models often achieve high predictive accuracy, they frequently lack decision efficiency, ignore resource constraints, and fail to address data privacy. Specifically, the literature reveals several gaps:

Actionability: Most models focus solely on prediction without integrating decision-making or intervention strategies.
Calibration: Raw probability outputs are often poorly calibrated, making them unreliable for risk scoring and resource allocation.
Cold Start: Many models require full historical data, rendering them ineffective for newly admitted students with limited interaction history.
Privacy: Few studies address the "right to be forgotten" or the ability to remove specific data points from a trained model without full retraining (machine unlearning).
Explainability: There is a lack of transparency in how predictions are made, hindering trust and adoption by educators.

Methodology

The proposed framework integrates predictive modeling, probability calibration, explainable AI (XAI), decision intelligence, and privacy-preserving mechanisms using the Open University Learning Analytics Dataset (OULAD).

1. Data Preprocessing and Feature Engineering

The study utilizes four primary tables from OULAD: studentInfo, studentVle, assessments, and studentAssessment. The dataset (32,593 students) is split into training (26,074) and testing (6,519) sets via stratified sampling.

Imputation: Median values for numeric attributes and mode values for categorical attributes.
Feature Construction: Three distinct feature categories are engineered:
- Engagement: Total clicks, active days, activity span, clicks per day.
- Assessment: Mean score, score variance, submission patterns, late submission ratio.
- Behavioral Trends: Engagement trends, inactive day ratios, and early engagement indicators.

2. Predictive Modeling

The study evaluates Logistic Regression (LR), Random Forest (RF), and XGBoost.

Selection: XGBoost is selected as the primary algorithm due to its superior ability to capture non-linear interactions and its strong ranking capabilities.
Hyperparameters: The model uses 300 estimators, a max depth of 5, a learning rate of 0.05, and subsampling of 0.9 to balance complexity and generalization.

3. Probability Calibration and Risk Scoring

To ensure predicted probabilities accurately reflect true risk levels, Isotonic Regression is applied to the XGBoost output.

Risk Score: Defined as the calibrated probability $P(at\_risk = 1 | X_i)$ .
Benefit: This allows for flexible thresholding and prioritization rather than rigid binary classification.

4. Agentic Decision Layer

A rule-based layer converts calibrated risk scores into context-sensitive interventions.

Logic: Interventions are assigned based on risk thresholds (e.g., Risk $\ge$ 0.80 triggers immediate mentor assistance; 0.60–0.80 triggers support actions).
Integration: This layer combines predictive outputs with behavioral indicators (e.g., inactivity ratios) to determine the most effective action.

5. Budget-Aware Intervention Policy

Recognizing that institutions have limited resources, the framework implements a budget-constrained approach.

Scoring: A final intervention score is calculated: $FinalScore = 0.7 \times RiskScore + 0.3 \times PriorityWeight$ .
Selection: Students are ranked by this score, and the top $K$ students are selected for intervention based on the available budget, ensuring high precision in resource allocation.

6. Cold-Start and Machine Unlearning

Cold-Start Model: A separate model is trained using only early-stage features (demographics and initial engagement) to predict risk for new students with limited data.
Machine Unlearning: To ensure privacy compliance, the framework evaluates the removal of a subset of training data (simulating the "right to be forgotten"). The model is retrained on the remaining data to assess performance stability.

7. Explainability

SHAP (SHapley Additive exPlanations) is used to provide global and local interpretability, identifying which features (e.g., engagement, recency of activity) drive specific predictions.

Key Results

Predictive Performance

XGBoost Performance: The uncalibrated XGBoost achieved an Accuracy of 0.920, F1-score of 0.922, and ROC-AUC of 0.976.
Calibrated XGBoost: After applying isotonic regression, the model achieved an Accuracy of 0.941, F1-score of 0.943, and a ROC-AUC of 0.985.
Comparison: Ensemble methods significantly outperformed Logistic Regression, with calibrated XGBoost demonstrating the strongest discriminative power.

Calibration and Explainability

Calibration Curve: The calibrated model closely followed the diagonal line, confirming that predicted probabilities are reliable for decision-making.
SHAP Analysis: Identified engagement features (clicks, activity span) and assessment metrics as the strongest predictors of student risk.

Budget-Aware Intervention

Precision-Recall Trade-off: Under small budget constraints (5%–20% of the student population), the system maintained near-perfect precision, ensuring that the most critical cases are addressed first. As the budget increased, recall improved, capturing more at-risk students.

Cold-Start and Unlearning

Cold-Start: The early-stage model achieved a ROC-AUC of approximately 0.79, demonstrating acceptable performance for proactive intervention despite data scarcity.
Machine Unlearning: Removing 5% of the training data resulted in negligible performance degradation:
- ROC-AUC Drop: 0.0002 (0.0204% relative drop).
- F1-Score Drop: 0.0005 (0.0527% relative drop).
- This confirms the model's robustness and independence from specific individual data points.

Significance and Contributions

The paper claims to bridge the gap between predictive modeling and practical implementation in educational settings. Its primary contributions include:

Integrated Framework: Unlike prior works that focus solely on prediction, this study presents an end-to-end system combining prediction, calibration, explainability, and decision-making.
Resource Optimization: The budget-aware policy addresses the reality of limited institutional resources, prioritizing high-risk students to maximize intervention impact.
Privacy Compliance: The demonstration of machine unlearning with minimal performance loss provides a pathway for GDPR-compliant student data management in learning analytics.
Early Intervention: The cold-start capability allows for risk identification at the very beginning of a student's academic journey, a capability often missing in data-hungry deep learning models.
Actionable Insights: By using SHAP and a rule-based agent layer, the system translates complex model outputs into clear, context-aware intervention strategies for educators.

The study concludes that while traditional methods prioritize accuracy, a feasible route to effective learning analytics requires a holistic approach that balances prediction accuracy with decision intelligence, resource constraints, and privacy preservation.