What Bugs Do Prolog Students Write? An Empirical Taxonomy and Data-Driven Mutation Framework
This paper presents LogMorph, a data-driven mutation framework for Prolog that leverages an empirical taxonomy of 7,201 student submissions to generate realistic synthetic faults with error distributions closely matching actual classroom mistakes, thereby improving the effectiveness of automated feedback tools in logic programming education.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are trying to teach a robot how to think like a human detective. You don't just tell it "solve the crime"; you have to teach it the specific, quirky rules of logic that govern how clues fit together. This is the world of Logic Programming, a way of writing computer code where you describe what is true rather than how to do it step-by-step. It's like giving a map of a city instead of turn-by-turn driving directions. But here's the catch: humans are messy. When we try to learn these rules, we make very specific, predictable mistakes. We might forget a rule, mix up two clues, or add a "stop sign" in the wrong place.
To build a helpful robot tutor that can fix our code, we need to know exactly what kind of mistakes we make. If the robot only practices on random, made-up errors, it will be confused when it meets a real student. It's like a driving instructor who only practices with cars that have flat tires, only to be surprised when a student actually forgets to put on their seatbelt. This paper dives into the messy reality of student errors to build a better training ground for these AI tutors.
The Great Prolog Bug Hunt
In a recent study, researchers decided to play detective, but instead of looking for criminals, they were hunting for bugs in computer code. They looked at 7,201 code submissions from 265 undergraduate students learning a language called Prolog. Think of Prolog as a language where you write a list of facts and rules, and the computer figures out the answers. The students were solving puzzles, from simple logic games to a final project where they built a solver for a complex board game called "Star Battle."
The team didn't just count how many programs failed; they wanted to know why. They picked 200 submissions where students had fixed a mistake and manually sorted the errors into a "taxonomy," which is just a fancy word for a detailed filing system. They found that the most common mistake wasn't a tricky logic error, but simply incomplete work. About 37.5% of the time, students forgot to write a whole piece of the puzzle, like missing a chapter in a story. The next most common errors were using the wrong ingredients (20.5%) or getting the goals in a rule mixed up (13.0%). Interestingly, they found that students rarely made the same kind of "careless" typos that people make in other languages; their mistakes were often deep misunderstandings of how the logic worked.
Building the "Bug Factory" (LOGMORPH)
Knowing what mistakes students make is great, but how do you teach a computer to recognize them? The researchers built a tool called LOGMORPH. Imagine a factory that takes a perfect, working piece of code and intentionally breaks it.
Old factories tried to break things randomly, like throwing darts at a board. They assumed every type of break was equally likely. But LOGMORPH is different. It's a data-driven factory. It looks at the "filing cabinet" of real student mistakes the researchers built earlier and says, "Okay, since students forget to finish their code 37.5% of the time, let's break the code that way 37.5% of the time."
The tool works in four steps:
- Scanning: It reads the perfect code and finds every spot where a mistake could happen.
- Sampling: It picks a spot to break, but it chooses based on the real student statistics. If "forgetting a clause" is common, it picks that often.
- Injecting: It actually breaks the code. Sometimes this is easy, like swapping two numbers. Other times, it needs to invent a new piece of code to insert. For this, it uses a smart "synthesizer" (a type of AI) to generate a new line of code that fits the rules.
- Testing: It checks if the broken code actually fails the tests. If the "broken" code still works perfectly, it throws it away and tries again.
The Results: A Near-Perfect Mirror
The team ran this factory to create 16,000 fake, buggy programs. They then compared the "bug profile" of these fake programs to the real student data. The results were surprisingly close. For most categories of errors, the fake programs matched the real ones within two percentage points. It was like looking in a mirror and seeing your reflection move exactly the same way.
However, the mirror wasn't perfect. The researchers noticed two main glitches:
- The "Cut" Problem: In Prolog, there is a special symbol called a "cut" (written as
!) that tells the computer to stop looking for other answers. Students often mess this up. But in the fake programs, these mistakes were rare. Why? Because the "testing" step of the factory was too strict. If a fake cut didn't change the test results, the factory threw it away. Real students might make a cut mistake that doesn't break the test but still confuses the logic, but the factory filtered those out. - The "Robot" Code: When the factory had to invent new code (like adding a random goal), the "synthesizer" sometimes wrote things that were technically correct but sounded like nonsense. For example, it might compare an empty list to a variable using a math symbol. No human student would ever write that; it looked like a robot trying to speak human. The researchers suspect that if they swapped the synthesizer for a more advanced AI trained on student writing, the fake code would sound much more natural.
What This Means
The paper doesn't claim to have solved all the problems of teaching logic programming. Instead, it offers a new, much better way to simulate student errors. By using real data to weight the mistakes, LOGMORPH creates a realistic training ground for automated tutoring systems. The researchers suggest that in the future, combining this data-driven approach with smarter AI language models could make these training tools even more lifelike, helping robots understand not just that a student is wrong, but why they made that specific, human mistake.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.