Engagement and Equity as Evidence for AI-Informed Instructional Design: A Learning Analytics Study of Two Large-Scale Datasets
By analyzing two large-scale learning datasets, this study demonstrates that while behavioral engagement is the strongest predictor of academic success, its unequal distribution across socio-economic and disability groups necessitates that AI-informed instructional design prioritize equity and early retention strategies over mere automation to effectively advance inclusive education.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of education, a quiet revolution has been underway for some time. Schools and universities are increasingly turning to technology to help students learn, hoping that digital tools can tailor lessons to individual needs, grade work instantly, and keep learners on track. This field, known as learning analytics, involves collecting vast amounts of data about how students interact with their courses—what they click on, how long they spend on a page, and when they submit assignments. The goal is to use this information to understand what works and what does not. However, as artificial intelligence promises to take these tools to the next level, there is a growing concern among educators. They worry that the rush to adopt new, flashy technologies is moving faster than the evidence needed to prove they actually help. The central question is no longer just what technology can do, but what specific design choices in a course reliably lead to student success, and whether these choices help all students or only a lucky few.
A new study by researchers at KCA University tackles this question by looking at two massive collections of real-world data. Instead of testing a new gadget in a small classroom, the team examined records from nearly 29,000 students in a large British distance-learning university and a separate sample of 4,000 users interacting with a commercial AI tutoring system for English language tests. By analyzing these huge datasets, the researchers sought to find the most powerful signals of success and to see if the promise of artificial intelligence is being built on a solid foundation or if it risks widening the gap between those who succeed and those who fall behind.
The investigation began by looking at the structure of the courses themselves. The researchers examined how the number of tests, the length of the course, and the types of assignments affected whether students passed or failed. They found that the specific design of a course matters more than the instructor's reputation. Longer courses that spread out assessments over time tended to have higher pass rates, while courses packed with many tests often saw more students struggle or drop out. Interestingly, the type of grading mattered less than the timing. Computer-graded assignments, which can be submitted late and often resubmitted, were used by students as flexible practice, whereas strict, timed exams were treated as final hurdles. This suggests that the rhythm and flexibility of a course are more important than the sheer volume of work or the specific technology used to grade it.
However, the most striking discovery was about student behavior. The researchers found that the single strongest predictor of whether a student would pass was simply how much they engaged with the course materials. Students who clicked on course pages, read forum posts, and took practice quizzes were far more likely to succeed than those who did not. This connection was so strong that it outweighed every structural feature of the course design they examined. In fact, the difference in activity between students who passed and those who failed was massive. Students who achieved the highest grades clicked on course materials more than twenty times as often as those who withdrew from the course. This finding suggests that keeping students interested and active is the most critical job for any educational system, whether it is run by a human teacher or an artificial intelligence.
The study also revealed a troubling reality about who gets to participate in this engagement. The data showed that success was not evenly distributed. Students from wealthier neighborhoods passed their courses at a much higher rate than those from the poorest areas. The gap was even wider when disability was added to the mix; students with disabilities who also came from deprived backgrounds were the least likely to succeed. The researchers found that these disadvantages stacked on top of each other, creating a barrier that was much higher than either factor alone. This means that simply adding more technology to a course does not automatically fix these problems. If the design does not account for these inequalities, new tools might end up helping only the students who are already well-positioned to succeed, leaving others further behind.
The picture became even clearer when the researchers looked at the AI tutoring system. They found that the engagement with this AI tool was extremely unequal. A small group of highly active users generated the vast majority of the interactions, while half of all users stopped using the system after just one session. In many cases, these users logged in for a single day and never returned. This is a critical problem for artificial intelligence because these systems need time and repeated interactions to learn what a student knows and to adapt their lessons accordingly. If a student leaves after the first try, the AI never gets the chance to do its job. The data showed that the most significant challenge for designers is not making the AI smarter or more adaptive, but rather figuring out how to get students to stay long enough for the system to work.
These findings offer a clear path forward for anyone trying to integrate artificial intelligence into education. The study argues that the priority should not be to automate grading or generate content first. Instead, the focus must be on three things: keeping students engaged from the very first session, designing the course structure to be flexible and supportive, and ensuring that the system works for students from all backgrounds, not just the average one. The researchers suggest that before adding any new AI feature, designers should ask if the course will keep a diverse group of students interested, if the workload is spread out reasonably, and if the system helps those who are struggling the most.
Ultimately, the study concludes that the promise of artificial intelligence in education will only be realized if it is built on these proven principles. The technology itself is not a magic solution that can fix a broken course or a disengaged student. The evidence shows that the most powerful tool for success is a design that encourages students to keep showing up and participating. For artificial intelligence to truly help, it must be used to support these human behaviors, ensuring that every student, regardless of their background or ability, has the chance to stay engaged and learn. The future of educational technology depends not on how advanced the algorithms are, but on how well they serve the fundamental needs of the learners they are meant to support.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.