Revisiting the Performance of Generative Artificial Intelligence on Introductory Object-Oriented Programming Assessments: Insights from 2026

In the classrooms where young people first learn to build software, a new kind of helper has arrived. These are not human tutors, but large language models: computer programs trained on vast libraries of text and code that can read a question and write a program in response. For years, educators have watched these tools evolve, wondering if they could truly understand the logic of programming or if they were merely mimicking patterns. The question matters because programming is often taught through a specific style called object-oriented programming, where code is organized into self-contained units that hold data and perform actions, much like how a real-world object has both properties and behaviors. If these artificial systems can solve the same problems as students, it changes how we teach, how we test, and how we think about the future of learning to code.

Researchers at the University of Tartu in Estonia decided to put five of the most popular artificial intelligence systems to the test in a real university setting. They did not ask the machines to solve abstract puzzles or write code in a vacuum. Instead, they took the exact same programming tests and final exam questions used in an introductory course for first-year students and fed them directly to the machines. The tasks were written in Estonian, the local language, and the researchers gave the systems no special instructions or hints. They simply asked the machines to solve the problems as a student would, using the same grading rules that teachers apply to human work. The goal was to see if these tools could keep up with the average student, and more importantly, to see where they still stumble.

The results showed that the artificial intelligence systems are remarkably capable, often outperforming the human students they were compared against. In the first programming test, which required building a complete program with multiple parts, every system except one scored higher than the average student. One system, a tool integrated into Microsoft Office, failed to produce a working program in one instance due to a small syntax error, but the others consistently earned top marks. In a second, slightly harder test involving more complex data handling, the machines again dominated, with most achieving perfect or near-perfect scores. Even on the final exam, which covered the entire course and included tricky conceptual questions, the machines generally scored well above the class average.

However, the machines are not perfect, and their mistakes reveal where their understanding is still shallow. While they could write long, complex programs that ran without crashing, they occasionally missed specific details that a human teacher would catch. For instance, some systems failed to properly format text or forgot to include a necessary instruction to make the code work, leading to errors that stopped the program from running. They also struggled with certain advanced concepts, such as how different parts of a program should talk to each other through specific rules called interfaces. In these areas, the machines sometimes produced code that looked correct but did not actually do what the question asked. Additionally, the study noted that performance was limited on graphics-related questions involving image interpretation, reflecting a broader challenge AI faces in handling non-textual information.

The researchers found that these systems have improved significantly compared to evaluations from just one year prior. Where older versions of these tools might have failed to pass the course, the newer models are now consistently scoring in the top range of student performance. Yet, a clear pattern emerged: the machines excel at following instructions to build standard structures but can falter when a task requires deep reasoning about how different pieces of a system fit together conceptually. They often add extra features that were not requested or use advanced techniques that the students had not yet learned, suggesting they are drawing from a much wider pool of knowledge than the course curriculum.

Ultimately, the study paints a picture of a tool that is powerful but not yet fully reliable for every educational need. The artificial intelligence systems can generate working code that often surpasses the average student, proving they have mastered the mechanics of programming. But they still make specific, recurring errors that indicate a lack of true understanding of the underlying logic. For educators, this means that while these tools can be used to support learning, they cannot yet replace the need for human oversight or the careful design of assessments that test deep conceptual understanding rather than just the ability to produce code. The machines are getting better every year, but they still have a long way to go before they can truly think like a programmer.