Leveraging Large Language Models for Assessments Grading and Personalised Feedback Generation: A Case of Software Engineering

In the classrooms of modern universities, a quiet revolution is taking place, driven by a new kind of artificial intelligence known as a large language model. These are not simple search engines that retrieve facts; they are systems trained on vast amounts of human writing, capable of understanding context, reasoning through complex problems, and generating text that mimics human conversation. For years, educators have wondered if these tools could do more than just chat; could they actually teach, or at least help teachers grade the mountains of work students submit? This question is particularly tricky in fields like software engineering, where students do not just write essays or solve math problems. Instead, they create complex blueprints for software systems, drawing diagrams that show how different parts of a program should connect and function. These visual designs require a human eye to spot subtle errors in logic and structure. If a computer could grade these assignments accurately and offer helpful advice on how to improve, it would free up teachers to focus on deeper learning, but only if the computer is reliable enough to be trusted with a student's grade.

A team of researchers from universities in Pakistan and the United Kingdom set out to test this very idea. They wanted to see if two of the most advanced artificial intelligence systems available could grade complex software design assignments and provide personalized feedback to students. To do this, they gathered 54 real student submissions from a course on software design and architecture. Each student had created three specific types of diagrams: a use case diagram showing how users interact with a system, a class diagram outlining the building blocks of the software, and a three-tier architecture design mapping out the system's layers. An experienced human instructor had already graded every single one of these assignments using a strict set of rules, or rubric, that defined exactly what was required for full marks. The researchers then fed these same assignments, along with the instructor's rules, into two different artificial intelligence models: one from a company called OpenAI and another from Google.

The researchers did not simply ask the computers to give a score. They designed a careful process to ensure the models thought through the grading step-by-step, much like a human teacher would. They asked the models to first check if the student had even addressed the correct topic, then to examine every part of the rubric to see if the required elements were present, and finally to calculate a score based on specific errors found. To make sure the results were stable and not just a lucky guess, the researchers asked each model to grade every assignment three separate times and then averaged the results. They also built a custom web tool to manage this entire process, allowing the models to read the student's PDF files, which contained both text and the complex diagrams, and generate a detailed report for each student.

When the researchers compared the computer-generated grades to the human instructor's grades, the results were a mix of promise and limitation. The artificial intelligence systems were generally close to the human teacher's average score, with the computer grades differing by about five marks out of a total of fifty on average. However, the computers struggled to match the human teacher's ability to distinguish between the very best and the very worst students. The human instructor gave a wide range of scores, from a low of 19 to a high of 49, reflecting a clear understanding of who had mastered the material and who had not. The computers, on the other hand, tended to compress these scores into a narrower band, often giving mediocre work a slightly higher grade and excellent work a slightly lower one. This meant that while the computers were good at giving a rough estimate of performance, they were not yet reliable enough to replace a human teacher in deciding a final grade, as they lost the ability to rank students accurately.

Despite the imperfections in grading, the feedback the computers generated was surprisingly well-received. The students who received the AI-generated comments found them clear, useful, and helpful for understanding their mistakes. In fact, when asked to choose which computer's feedback they preferred, the vast majority of students picked the model from OpenAI, noting that its comments felt more personal and constructive, and closer to what a human teacher would say. Interestingly, this preference did not always align with which computer gave the mathematically closer grade; the model that students liked best for its writing style was not necessarily the one that was most accurate in its scoring. This suggests that for students, the quality of the advice matters more than the exact number assigned to their work.

The study concludes that these artificial intelligence tools are powerful assistants but not yet ready to take the wheel entirely. They can handle the heavy lifting of reviewing hundreds of assignments, spotting missing elements in diagrams, and drafting initial feedback, which saves teachers a tremendous amount of time. However, the research strongly suggests that a human teacher must remain in the loop to review the final grades and ensure fairness, especially for high-stakes assessments where a student's future depends on the result. The ideal future, according to these findings, is a hybrid approach where the computer acts as a tireless first reader, offering a preliminary evaluation and detailed suggestions, while the human instructor provides the final judgment and the nuanced understanding that only a person can offer. This partnership could transform software engineering education, making personalized feedback available to every student without overburdening the teachers.