🤖 AI

When Can We Trust Early Warnings? Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs

This paper introduces LEAP, a protocol designed to eliminate temporal leakage in early-warning models by enforcing strict cutoff-based data truncation, demonstrating through OULAD experiments that such rigor prevents inflated performance estimates while revealing that Random Forest and Gradient Boosting are the most effective predictors at different stages of a course.

Original authors: Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ngoc Luyen Le, Marie-Hélène Abel, Bertrand Laforge

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a coach trying to predict which players on a sports team will win the championship. You want to make this prediction as early as possible—maybe after just the first two weeks of practice—so you can give extra help to those who are struggling.

This paper is about building a system to make those predictions using data from a school's online learning platform (called an LMS). However, the authors discovered a major problem: many previous studies were cheating, even if they didn't mean to. They were accidentally looking at the "answer key" before the test was even finished.

Here is a breakdown of the paper's story using simple analogies:

1. The Problem: The "Time-Traveling" Coach

In the past, researchers tried to predict if a student would pass or fail by looking at their online activity (clicking links, posting on forums, taking quizzes).

The problem was Data Leakage. Imagine a coach trying to predict a player's performance in Week 2, but the coach is secretly peeking at the player's final exam score from Week 10.

The Cheat: If the coach sees the player got an "A" on the final exam, they will predict the player will win. But that score didn't exist yet in Week 2!
The Result: The coach looks like a genius because their predictions are perfect, but in the real world, they would fail miserably because they can't see the future.

The authors found that many "early warning" systems were doing exactly this. They were mixing in information that wouldn't be available until after the prediction was supposed to be made.

2. The Solution: The "LEAP" Protocol

To fix this, the authors created a strict set of rules called LEAP (Leakage-Excluded Early-Availability Protocol).

Think of LEAP as a strict referee who blows a whistle at a specific moment (the "cutoff").

The Rule: At the moment the whistle blows (e.g., Day 14 of the course), the referee locks the door. No data from Day 15 or later can enter the room.
The Process: The system must first throw away every record that happened after Day 14. Only then can it count clicks, sum up forum posts, or check quiz scores.
The Check: Before the model makes a prediction, the referee double-checks: "Did you use any data from the future?" If the answer is yes, the prediction is thrown out.

3. The Experiment: Testing the Rules

The authors tested this on a huge dataset from the Open University (OULAD), which contains records from over 32,000 students. They ran the experiment like a series of checkpoints:

Week 1: Can we predict the outcome?
Week 2: Can we predict it better?
...all the way to Week 8.

They used standard computer models (like Random Forests and Gradient Boosting) to see how well they could guess the final result (Pass or Fail) at each of these checkpoints, strictly following the LEAP rules.

4. What They Found

The results were honest and revealed some interesting truths:

Time is the Best Teacher: The longer you wait, the better the prediction gets. This makes sense; you need more evidence to be sure.
The "Week 3" Jump: There was a noticeable jump in accuracy around Week 3. Before this, the models were guessing based mostly on "how busy" the student was. After Week 3, they started getting actual graded quiz scores, which are much stronger clues.
Different Tools for Different Times:
- Early on (Weeks 1-2): A model called Random Forest was the best detective. It was good at spotting patterns in simple activity (like "Did they click anything?").
- Later on (Week 3+): A model called Gradient Boosting took the lead. It was better at combining complex clues, like "They clicked a lot, but they also got a low score on the first quiz."
The "Leakage" Shock: When the authors intentionally broke the rules (letting the models peek at future grades), the accuracy skyrocketed to nearly 100%. This proved that without strict rules, early warning systems look amazing on paper but are useless in reality.

5. The Big Takeaway

The paper concludes that we can trust early warnings, but only if we are very strict about what data we are allowed to use.

Early predictions rely on "behavioral proxies" (like how often a student logs in or clicks).
Later predictions rely on "assessment evidence" (actual grades).

If you want to know if a student is in trouble early, you have to look at their activity habits, not their grades (because the grades don't exist yet). And if you want your system to be trustworthy, you must use a protocol like LEAP to ensure you aren't accidentally cheating by looking at the future.

In short: You can predict the future, but you can't peek at the answer key before the test is over. LEAP is the rulebook that makes sure no one looks at the answer key too soon.

Technical Summary: Leakage-Excluded Early Outcome Prediction from LMS Interaction Logs

Problem Statement
Early-warning systems in Learning Management Systems (LMS) aim to predict end-of-course outcomes (e.g., success or withdrawal) early enough to enable timely learner support. However, reported "early" performance in existing literature is frequently inflated by temporal leakage. This occurs when the modeling pipeline inadvertently incorporates information that would not be available at the specific prediction time (cutoff). Common sources of leakage include aggregating data over the full course before truncating, joining assessment tables that contain future submissions, or using preprocessing statistics derived from the entire dataset. These violations result in overly optimistic benchmarks that do not reflect real-time deployment conditions, thereby undermining the trustworthiness of early-warning interventions.

Methodology: The LEAP Protocol
To address temporal validity, the authors introduce LEAP (Leakage-Excluded Early-Availability Protocol). LEAP is not a new predictive model but a rigorous preprocessing and validation protocol designed to enforce strict temporal constraints.

Core Constraint: For any prediction cutoff time $t$ , the model must use only information observable up to $t$ .
Operational Rules:
1. Cutoff-First Truncation: Time truncation is applied to raw data sources before any joins, aggregations, or feature construction. This prevents future events from influencing early features indirectly.
2. Provenance Auditing: The protocol explicitly records the maximum timestamp of records accessed to compute each feature group. A run is halted if any feature relies on data with a timestamp $> t$ .
3. Source Handling: Interaction logs and assessment logs are truncated individually before being joined.
4. Observability Assumptions: The protocol distinguishes between behavioral traces (immediately observable) and assessment scores. For the latter, the authors adopt an explicit assumption that if a submission occurs by cutoff $t$ , the score is observable at $t$ (acknowledging that OULAD does not explicitly record grade release times).

Experimental Setup
The protocol was instantiated using the Open University Learning Analytics Dataset (OULAD), comprising 32,593 instances across 22 course runs.

Cutoffs: Evaluations were conducted at weekly intervals ( $t = 7, 14, \dots, 56$ days).
Models: A diverse set of standard classifiers (Logistic Regression, Random Forest, Gradient Boosting, SVM, kNN, etc.) was benchmarked. Models were trained independently for each cutoff to respect the information available at that specific time.
Metrics: Performance was measured using ROC-AUC, PR-AUC, Brier score (calibration), and F1@0.5.
Ablation: Intentionally "leaky" variants were created to quantify the inflation of performance when temporal constraints are violated.

Key Results

Earliness–Performance Trade-off: Under strict LEAP constraints, predictive performance improves steadily as the observation window expands.
- ROC-AUC increased from 0.7151 at week 1 ( $t=7$ ) to 0.8602 at week 8 ( $t=56$ ).
- A significant performance jump occurred between week 2 and week 3 ( $t=14$ to $t=21$ ), suggesting that week 3 provides a critical accumulation of predictive evidence.
Model Performance:
- Random Forest (RF) performed best at the earliest cutoffs ( $t=7, 14$ ).
- Gradient Boosting (GBDT) became the dominant model from week 3 onward, achieving the highest overall performance at $t=56$ .
Impact of Leakage: Leakage ablation studies revealed massive performance inflation when temporal validity was ignored.
- At $t=7$ , allowing assessment leakage (Leaky-All) inflated the RF ROC-AUC from 0.7151 to 0.9669.
- This demonstrates that post-cutoff assessment data can make early-warning systems appear nearly perfect, rendering such benchmarks unrealistic for real-time intervention.
Signal Evolution: The nature of predictive evidence shifts over time. Early predictions ( $t=7$ ) rely heavily on behavioral engagement metrics (e.g., total clicks, activity regularity). As the window expands, assessment performance (e.g., submission scores) becomes the primary driver of prediction accuracy.

Significance and Claims
The paper claims that the reliability of early-warning systems depends on the explicit enforcement of temporal validity.

Protocol over Architecture: The primary contribution is the definition of LEAP as a standard for leakage-controlled benchmarking, arguing that "early" performance claims are invalid without strict provenance checks.
Realism: The authors assert that without protocols like LEAP, reported early performance is often an artifact of retrospective data access rather than a reflection of actionable early signals.
Actionable Insight: The study identifies that meaningful predictive signals emerge early (within the first two weeks) based on engagement, but substantial gains require the accumulation of assessment evidence by week 3.
Future Directions: The authors suggest that future work should move beyond aggregate LMS traces toward richer pedagogical representations (e.g., competency structures) and integrate context-aware factors (e.g., learner circumstances) while maintaining ethical constraints. They also propose extending LEAP to sequence-based models and real-world LMS platforms like Moodle.