💻 computer science

Target Leakage in Machine Learning Models that Predict Higher Education Graduation from Administrative Records

This study demonstrates that previously reported near-perfect predictions of higher education graduation were artifacts of target leakage from outcome-determined attributes, and establishes that when using a corrected, leakage-free protocol on administrative records from Gaza, machine learning models achieve high but realistic accuracy (up to 89.16%) with the number of completed semesters being the strongest predictor.

Original authors: Samer Yaghi, Rebhi S. Baraka

Published 2026-10-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Samer Yaghi, Rebhi S. Baraka

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern university, a vast digital trail is left behind by every student. From the moment they apply, their names, grades, and attendance are recorded in administrative systems, creating a massive archive of academic history. For decades, researchers have tried to mine this data to find patterns that could predict who will finish their degree and who might drop out. The goal is to identify struggling students early, allowing schools to offer help before it is too late. This field, known as educational data mining, relies on a simple but strict rule: a prediction must be made using only the information available at that specific moment. If a model uses a piece of information that only exists after the outcome has already happened, it is using future information. This mistake, called target leakage, creates a false sense of certainty, making a model look perfect when it is actually just reading the answer key.

A team of researchers in Gaza, Palestine, set out to test this idea using twenty years of records from the University College of Applied Sciences. They examined the files of nearly 4,700 students who had studied information technology between 2000 and 2019. Their investigation began with a startling discovery. When they built computer models to predict graduation using the data exactly as it was stored in the school's system, the results were suspiciously perfect. The models achieved 100 percent accuracy, meaning they never made a single mistake. This seemed like a triumph, but the researchers realized the models were not actually predicting the future. They were simply looking at two specific columns in the database: the year a student graduated and their final grade average. These numbers only appear in the system if a student has already finished their degree. By including them, the models were essentially being told the answer before they started the test.

Once the researchers removed these two "future information" columns, the picture changed completely. The models, which included decision trees, random forests, and gradient boosting algorithms, had to work with only the information available while the students were still studying. Under these honest conditions, the accuracy dropped to a more realistic level, hovering around 89 percent. While this is still a strong result, it is far from the impossible perfection seen before. The researchers found that the most powerful signal for whether a student would graduate was not their grade point average, but simply how many semesters they had successfully completed. A student who kept showing up and finishing terms was far more likely to graduate than one who had high grades but stopped attending. This suggests that persistence is a stronger predictor of success than momentary academic performance.

The study also looked at how early these predictions could be made. When the researchers restricted the models to use only the information available at the moment a student applied to the university—such as their high school grades and age—the accuracy fell to about 70 percent. This indicates that while schools can get a general sense of risk before a student even arrives, the true picture only emerges once the student begins their journey and their history of attendance starts to build. The researchers also checked whether the models treated male and female students fairly. They found that the models did not discriminate; in fact, because female students in this specific group had a higher natural rate of graduation, the models correctly predicted them to be more likely to succeed. The earlier version of the study had suggested a different rule based on gender, but that was an illusion created by the flawed data.

The implications of this work extend beyond just one university in Gaza. It serves as a cautionary tale for institutions worldwide that are eager to use artificial intelligence to manage student success. The researchers demonstrated that a model can look incredibly powerful while being entirely useless if it is allowed to peek at the answer. They provided a clear method for other schools to check their own data for these hidden flaws. For university advisors, the corrected findings offer a practical path forward: focus on students who stop attending classes rather than just those with low grades, and use these predictions as a prompt for human conversation rather than an automated decision. The study concludes that high accuracy is not the only measure of a good model; a model that is honest about what it knows and how it knows it is far more valuable for helping students navigate their education.

Technical Summary: Target Leakage in Machine Learning Models for Graduation Prediction

Problem Statement
This study addresses a critical methodological flaw in Educational Data Mining (EDM): target leakage. Specifically, it investigates how machine learning models trained on administrative records from higher education institutions can produce spuriously high accuracy (near 100%) when outcome-determined attributes are inadvertently included in the feature set. The authors examine a 20-year dataset (2000–2019) from the University College of Applied Sciences (UCAS) in Gaza, Palestine, involving 4,704 IT students. The core problem is that standard student information systems often contain fields like "Graduation Year" and "Graduate Grade Average" that are only populated for students who have already graduated. Including these fields allows models to rely on the presence of the outcome rather than predicting it.

Methodology
The research employs a rigorous, leakage-free protocol to re-evaluate graduation prediction using three classifiers: Decision Tree (DT), Random Forest (RF), and Gradient Boosting (GBT).

Data Preprocessing and Leakage Screening:
- The initial dataset of 65,532 records was filtered to 4,704 IT students.
- Critical Step: Two attributes, GRADUATE_YEAR and STD_GRADEAVG, were identified as deterministic functions of the outcome (graduation status) and were removed. GRADUATE_YEAR is null for non-graduates, and STD_GRADEAVG is null or distinct for non-graduates.
- Protocol Correction: The study enforces a strict ordering where the train/test split (70/30 stratified) occurs before any data scaling or augmentation. This prevents information from the test set from leaking into the training process, a flaw present in the authors' previous work.
- Augmentation: Gaussian noise augmentation was applied to the training set only as a negative control.
Experimental Configurations:
- Configuration A (Leakage Diagnostic): Retains the outcome-determined attributes to demonstrate the mechanism of spurious accuracy.
- Configuration B (Corrected Model): Uses full academic history (excluding leaked attributes) to establish realistic performance baselines.
- Configuration C (Admission-Time Prediction): Restricts features to data available at enrollment (e.g., high school stream, age, intended major) to assess early prediction capabilities.
Evaluation Metrics:
- Performance is measured via Accuracy, AUC, F1, Precision, and Recall on a hold-out test set.
- Statistical validity is confirmed using 5-fold stratified cross-validation, Wilson score intervals for confidence, and McNemar's test for pairwise model comparison.
- Fairness Audit: Recorded sex is excluded from the input features but used post-hoc to audit subgroup performance (Demographic Parity and Equal Opportunity).

Key Results

The Leakage Effect (Configuration A): When GRADUATE_YEAR and STD_GRADEAVG are included, all three classifiers achieve 100% accuracy with zero cross-validation variance. This confirms that the models are not learning patterns but simply reading the outcome label.
Corrected Performance (Configuration B): Upon removing leaked attributes, performance drops to realistic levels:
- Gradient Boosting: 89.16% accuracy (AUC = 0.955).
- Random Forest: 88.81% accuracy (AUC = 0.954).
- Decision Tree: 87.18% accuracy (AUC = 0.933).
- Statistical Significance: McNemar's test shows the two ensemble methods (RF and GBT) are statistically indistinguishable ( $p = 0.620$ ), while both significantly outperform the Decision Tree.
Feature Importance: Contrary to previous assumptions that GPA is the dominant predictor, the number of regular semesters completed ( $r = +0.546$ ) is the strongest predictor, surpassing cumulative GPA ( $r = +0.450$ ). This suggests persistence is a stronger signal than attainment.
Early Prediction (Configuration C): Using only admission-time attributes, the best model (GBT) achieves 70.47% accuracy (AUC = 0.769). This indicates that while early signals exist, they are probabilistic and insufficient for high-stakes individual decisions.
Fairness: The models exhibit a demographic parity difference of approximately -0.11 (favoring females), which aligns with the observed base rate of female graduation (45.0% vs. 36.6% for males). The models do not introduce bias; they reflect existing institutional disparities.

Key Contributions

Diagnostic of Target Leakage: The paper provides a reproducible demonstration that outcome-determined attributes produce perfect, spurious accuracy. It offers a specific diagnostic signature (100% accuracy across heterogeneous models with zero CV variance) for institutions to detect similar errors.
Corrected Baselines: It establishes defensible performance baselines for graduation prediction in a resource-constrained, conflict-affected context (Gaza), correcting previously inflated claims.
Methodological Rigor: The study highlights the necessity of splitting data before scaling/augmentation and reports the negative impact of Gaussian noise augmentation on this specific dataset.
Fairness and Interpretability: It provides a subgroup audit by sex (held out of the model) and interpretable decision rules, showing that persistence (semester count) is the primary driver of prediction, not demographic factors.

Significance and Claims
The authors explicitly state that the study's contribution is not a higher accuracy, but a defensible one. The significance lies in:

Reliability: Exposing a common methodological failure that renders many "near-perfect" educational prediction studies unreliable.
Contextual Representation: Providing rare longitudinal data analysis from a Palestinian institution, addressing a gap in Middle Eastern representation in EDM literature.
Actionable Policy: Shifting the focus from "who will graduate" to "who is persisting." The authors argue that advising interventions should be triggered by interruptions in semester progression (persistence) rather than low grades alone.
Deployment Guidelines: The paper advocates for using these models as prompts for human review (due to a ~12% false-negative rate) rather than automated decision-making tools, and emphasizes the need for ongoing subgroup monitoring to ensure equitable deployment.

The paper concludes that high prediction accuracy achieved too easily is likely an artifact of leakage, and that responsible AI in education requires transparency, methodological rigor, and an understanding that predictive models must serve human oversight, not replace it.

Technical Summary: Target Leakage in Machine Learning Models for Graduation Prediction

More like this