How Much of the Engagement–Performance Association Survives Adjustment? A Double Machine Learning Study in Online Higher Education

In the modern university, a vast amount of student life leaves a digital footprint. Every time a learner logs into a course website, clicks a link, or downloads a document, a record is created. Educators and researchers have long treated these digital traces as a window into student success. The logic seems straightforward: if a student is active online, they are likely engaged, and if they are engaged, they are likely to do well in their class. This belief has driven the creation of dashboards and early-warning systems designed to spot struggling students by counting their clicks. However, a critical question has remained unanswered: does the activity itself cause the good grades, or are the good grades simply a sign of a student who was already prepared and motivated? The difference matters immensely. If activity is merely a symptom of success, then forcing a student to click more buttons will not help them learn. If, however, the activity is a genuine driver of learning, then encouraging more interaction could be a powerful tool for improvement.

A team of researchers set out to untangle this relationship using a massive collection of data from the Open University in the United Kingdom. They examined the records of over twenty-one thousand students who took part in twenty-two different online course presentations. The dataset was rich, containing details about the students' backgrounds, their past academic performance, and over ten million interactions with the virtual learning environment. The researchers focused specifically on the period leading up to when a student submitted a piece of coursework. They counted the number of clicks and activities that occurred in the days before the submission deadline, ensuring that any activity happening after the work was turned in was not counted. This careful timing was crucial to ensure they were looking at behavior that happened before the result was known.

The study began by looking at the raw numbers, the kind of simple association that often appears in standard reports. They found a strong link: for every standard increase in the volume of online activity, the assessment score rose by about three and a quarter points. This number is what a typical dashboard might highlight to suggest that more activity equals better performance. However, the researchers knew that this raw number likely included the influence of many other factors. Students who had done well in the past might naturally click more because they are confident, or they might click more because they are already good at the subject. To find the true effect of the activity itself, the researchers used a sophisticated statistical method designed to strip away these pre-existing differences. They adjusted the data to account for everything they knew about the students before the activity window opened, including their previous grades, their demographic background, and their history of using the platform.

When the researchers applied this adjustment, the story changed significantly. The link between activity and grades did not disappear, but it shrank considerably. After accounting for prior history, the increase in the score dropped from three and a quarter points to just under one and three-quarter points for the same increase in activity. This means that nearly half of the original connection between clicking and scoring was actually due to the students' backgrounds and past performance, not the activity itself. The remaining link, while still real and statistically significant, was much smaller than the raw numbers suggested. It indicates that while online activity does carry useful information about how a student is doing, relying on the unadjusted numbers would lead institutions to overestimate how much simply increasing clicks would improve a student's grade.

The researchers also looked at whether this adjusted relationship looked different for different types of students or different kinds of assignments. They found that the link was stronger for assignments that required written, open-ended answers graded by a human tutor, compared to multiple-choice questions graded by a computer. This suggests that the type of work matters; the digital activity might be more closely tied to the deep thinking required for essay-style tasks than for quick, automated quizzes. They also discovered that the connection was stronger for students who had lower grades in the past. For these learners, the online activity seemed to be a more reliable signal of their current performance. In contrast, the link did not show a clear difference based on whether a student came from a disadvantaged background or had declared a disability, though the data was not precise enough to rule out small differences entirely.

Despite these findings, the researchers remained cautious about claiming that the activity causes the grades. To test for hidden factors they could not measure, they performed a specific check. They asked whether the current online activity could predict a grade from a previous assignment that had already been completed before the current activity even started. Logically, future clicks cannot change a past grade. Yet, the data showed that students who clicked more in the current window did indeed have higher scores on that previous assignment, even after all the adjustments. This result signals that there are still unseen forces at play. Perhaps some students are naturally more motivated or have better study habits that exist outside the digital system, driving both their online activity and their grades. Because of this, the researchers concluded that while the adjusted link is real, it is not a guarantee that an intervention to force more clicks would automatically raise scores by the full amount of the adjusted number.

The study offers a clear lesson for how universities use data to support students. The digital traces left by learners are valuable, but they are not a simple switch that turns on learning. The raw association between activity and success is strong, but it is heavily influenced by who the student is and what they have done before. When educators use these numbers to decide who needs help, they must understand that the numbers tell a story of correlation, not necessarily of cause and effect. A student with low activity might need support, but simply telling them to click more is unlikely to fix the problem if the underlying issue is a lack of prior knowledge or motivation. The most effective approach is to use these behavioral signals as a prompt for conversation and diagnosis, recognizing that the activity is a signpost pointing toward a student's state, rather than a lever that can be pulled to force a better outcome.