PROBE: Benchmarking Code Generation in Large Language Models
The paper introduces PROBE, a comprehensive benchmark framework that evaluates code generation in Large Language Models across functional correctness, solution proximity, and code quality, revealing that while models show promise, they frequently struggle with complex problems, under-resourced languages, and fundamental errors.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are teaching a robot to cook. You don't just want the robot to follow a recipe; you want it to understand why the ingredients go together, to taste the dish and know if it's salty enough, and to make sure it doesn't accidentally set the kitchen on fire. This is the world of Large Language Models (LLMs) in software engineering. Think of these models as super-smart robots that have read almost every cookbook (code) ever written. They can look at a description like "make a sandwich" and instantly write the instructions (code) to do it. But here's the catch: just because a robot can write instructions doesn't mean the sandwich will taste good, or that the instructions won't tell you to use a chainsaw instead of a knife. Scientists care deeply about this because as we let these robots write more of our software, we need to know if they are actually reliable or if they are just guessing and hoping for the best.
Enter PROBE, a new, super-organized "taste-test" for these code-writing robots. Before this, most tests were a bit like asking a robot, "Did you make the sandwich?" and only checking if the robot said "Yes." If the sandwich was burnt or had no bread, the test didn't care as long as the robot claimed it was done. The researchers behind PROBE realized this wasn't fair. They built a much stricter, more colorful evaluation system that checks three things: Did the code actually work (Functional Correctness)? How close was the robot's recipe to a perfect human recipe (Proximity)? And was the code messy or elegant (Code Quality)?
The team put six different robots—some small and open-source, some huge and proprietary—through their paces across five different "languages" (Python, C++, Java, C, and Rust). They tried three different ways of talking to the robots: just giving a command, showing one example first, or letting the robot try, fail, and then fixing its mistakes based on error messages.
Here is what they found, and it's a mix of exciting progress and some very funny, very human-like mistakes. First, the bigger robots generally did better, but even the smartest ones weren't perfect. They solved about 70% of the easy problems but struggled mightily with the hard ones. Second, showing the robot an example first (a technique called "In-Context Learning") barely helped at all. It was like showing a chef a picture of a sandwich before asking them to make one; they already knew how to do it, so the picture didn't change much. However, letting the robot try, fail, and then giving it the error message to fix it (Feedback Incorporation) was a game-changer. It helped the robots fix simple mistakes, like forgetting to import a tool, and boosted their success rate by about 5%.
But the real story is in the mistakes. The robots often failed in ways that were surprisingly basic. They tried to build houses with too many bricks (memory errors), forgot to bring the keys to the door (missing imports), or got stuck in a loop trying to count every grain of sand on a beach (inefficient algorithms). One robot even tried to calculate a number so huge it caused the system to panic, just like a calculator breaking when you divide by zero. Interestingly, the robots were great at Python but terrible at Rust, a language that is very strict about safety, suggesting they haven't read enough "Rust cookbooks" yet.
Most importantly, the researchers found that even when the robots got the code to "work," the code was often simpler and shorter than what a human would write. While that sounds good, it sometimes meant the robots were taking shortcuts that wouldn't hold up in the real world. The study concludes that while these AI tools are getting better, they are still prone to making silly, avoidable errors. They aren't ready to be left alone in the kitchen just yet; they need a human chef to double-check the recipe before serving it to the world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.