A systematic comparison of Large Language Models for automated assignment assessment in programming education: Exploring the importance of architecture and vendor

Imagine you are a teacher with a mountain of homework to grade. You have 6,000 programming assignments to check, and your eyes are tired. Enter the "Digital Grader": a type of super-smart computer brain called a Large Language Model (LLM). Think of these models as incredibly well-read robots that have read almost everything on the internet, including millions of lines of code. They are designed to understand language and logic, so it seems like a perfect idea to ask them to grade student work instantly. But here is the catch: just like humans, these robots have different personalities. Some are strict, some are easy-going, and some are just plain weird. The big question for schools is: if we let these robots take over grading, will they be fair? Will they agree with human teachers? Or will they turn the grading system into a chaotic game where one robot gives you an A+ and another gives you an F for the exact same code?

This paper is like a massive "taste test" for these digital graders. The author, Marcin Jukiewicz, gathered 6,000 real student programming assignments from a university course and asked 18 different versions of these AI robots to grade them all. He didn't just ask them to give a score; he watched how they thought, how strict they were, and whether they agreed with each other or with the human teachers. The study reveals that these AI models are not a uniform block of intelligence; they are a family of very different cousins. Some are "liberal" and hand out high scores like candy, while others are "restrictive" and fail students for tiny mistakes. Even more surprisingly, the robots from the same company (like OpenAI or Google) tend to think and grade in very similar ways, almost like they were raised in the same house with the same rules.

The most important finding, however, is a bit of a reality check. Even the "best" AI robot in the study only showed a moderate level of statistical alignment with human teachers, far from a perfect match. In the world of grading, that's like two judges in a talent show giving completely different scores to the same singer. The study suggests that while AI can be consistent with itself, it still struggles to match the nuanced, forgiving, and context-aware way human teachers grade. Furthermore, the study found that the "mini" or "nano" versions of these models (the smaller, cheaper ones) were generally worse at grading than their big, full-sized brothers. So, if a school wants to use AI to grade homework, they can't just pick any robot; they have to choose carefully, because the choice of robot will actually change the grades students get.

The Great Robot Grading Showdown

Imagine you walk into a classroom where 18 different robots are sitting at desks, each holding a red pen. They are about to grade the same stack of 6,000 programming assignments. These aren't just any robots; they are the latest and greatest "Large Language Models" (LLMs) from four major tech giants: OpenAI, Google, Anthropic, and DeepSeek. The goal of this study was to see if these robots would all agree on who gets an A, who gets a B, and who fails.

The results were a bit like a reality TV show where the judges can't agree on anything. The author discovered that these robots have distinct "grading philosophies."

The "Nice" Robots vs. The "Mean" Robots
Some robots were incredibly generous. Models like GPT-4o and Claude-haiku-3.5 were the "liberal" graders. They tended to give out high scores (a "1" or full credit) very often. If a student's code was mostly right, these robots were happy to give them a passing grade. On the other end of the spectrum, you had the "restrictive" robots like DeepSeek-Reasoner and GPT-4.1-nano. These were the strict disciplinarians. They were much more likely to give a "0" (a failing grade) if they spotted even a small error. They were the robots that would fail you for a missing semicolon while the nice robots would just give you a gentle nudge.

Then there were the "balanced" robots, like Claude-sonnet-4 and Gemini-2.0-flash-lite. These were the middle-ground graders. They didn't just give you a perfect score or a failing one; they loved the "0.5" score. They were the ones saying, "You got the main idea, but you missed a few details, so here is a partial credit."

The Family Resemblance
One of the coolest discoveries was that robots from the same company acted like siblings. If you looked at the grading patterns of the different OpenAI models (like GPT-4o, GPT-5, and their "mini" versions), they clustered together. They thought alike. The same happened with Google's Gemini models and Anthropic's Claude models. It's as if OpenAI taught all its robots to grade in a specific style, and Google taught its robots a different style. The study found six distinct groups of robots: five main "clans" with shared styles (GPT-5, GPT-4, Gemini, DeepSeek, and Claude) plus one special "Outlier cluster" that grouped two unique models together because they didn't fit anywhere else.

The "Mini" vs. The "Mega"
The study also looked at the size of the robots. Tech companies often release "mini" or "nano" versions of their models that are smaller, faster, and cheaper to run. The paper found that these smaller robots were generally worse at grading. The full-size "flagship" models were more consistent and reliable. It's like comparing a professional chef to a kid who just watched a cooking show; the kid might get the dish done, but the professional is more likely to get the taste right every time. The "mini" and "nano" models had lower agreement with human teachers, suggesting that if a school is trying to save money by using the cheaper, smaller models, they might be getting lower-quality grading.

The Human Factor
Here is the twist that might surprise you. When the author compared all these robots to the actual human teachers who graded the same assignments, the robots didn't match up well. In fact, the human teachers were the most generous of all! They gave the highest average scores (0.726) and were much more likely to give a full "1" than any of the robots.

The best robot, Claude-haiku-3.5, showed a statistical agreement score of 0.470 with the human teachers. In the world of statistics, this is considered "moderate" agreement, not a "good" one. To be considered "good," you usually need a score higher than 0.75. This means that if you used this robot to grade your homework, it would likely give you a different grade than your teacher would, and it would be stricter.

The "Crowd" Paradox
The study did something clever to test the robots against each other. Instead of comparing them to humans, the researchers asked: "If we take the most common grade given by all 18 robots, is that the 'true' grade?" When they did this, the robots agreed with each other much better (over 80% agreement for the top models). This suggests that the robots are very consistent with their own kind, but they are all consistently different from the human teachers.

This raises a big question: Are the robots right and the teachers too lenient? Or are the robots all making the same mistake because they were trained on similar data? The paper suggests that while the robots are consistent, they might be missing the "human touch"—the ability to understand that a student tried hard, or that a small mistake doesn't mean they didn't learn the concept.

What This Means for the Future

The paper doesn't say that AI grading is broken or that we should stop using it. Instead, it warns schools to be careful. Choosing a robot to grade your homework isn't a neutral decision; it's a choice that changes the outcome. If you pick a "liberal" robot, students might get higher grades. If you pick a "restrictive" one, they might get lower grades.

The study concludes that we can't just let AI take over grading completely. We still need human teachers to keep an eye on things. The robots are great tools, but they are like different types of calculators: some round up, some round down, and some have different buttons. Until we figure out how to make them agree with human teachers, we have to remember that the grade a robot gives you might not be the same grade your teacher would give you. And in the world of education, that difference matters a lot.