Programmers Are Poor and Overconfident Judges of LLM-Generated Assertions
This study reveals that programmers are significantly overconfident yet inaccurate when evaluating incorrect AI-generated assertions, a problem that is not mitigated—and may even be worsened—by natural-language explanations, suggesting that current AI tools may not reliably enhance code comprehension or review.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you're a junior detective in a high-tech city where a super-smart robot assistant (an AI) is helping you solve crimes. The robot doesn't just write the story of the crime; it also writes a "rulebook" of what should have happened, called an assertion. Your job is to look at the robot's rulebook and say, "Yep, that matches the story," or "Whoa, that's wrong!"
You'd think having the robot explain its rulebook in plain English would make your job super easy, right? Like having a friendly guide whispering, "Hey, look here!" while you read.
Well, a team of researchers put 86 real-life programmers (our detectives) to the test to see how good they are at this. They found some surprising, almost funny, truths about how our brains work when AI is involved.
The "Yes-Man" Problem
Here's the big discovery: Our detectives are great at spotting when the robot gets it right, but they are terrible at spotting when it gets it wrong.
When the robot wrote a correct rulebook, the programmers got it right about 74% of the time. But when the robot wrote a wrong rulebook, the programmers only caught the mistake 49% of the time. That's barely better than flipping a coin!
Even weirder? The detectives felt just as confident in their wrong answers as they did in their right ones. Whether they were right or wrong, they rated their confidence at about 4 out of 5. It's like a student who is 100% sure they got a math problem right, even when they actually got it wrong. The researchers found this difference was statistically significant (meaning it wasn't just a fluke), with the odds of catching a correct rulebook being nearly three times higher than catching a wrong one.
The "Confusing Guide" Trap
Now, let's talk about the robot's explanations. The researchers thought, "Maybe if the robot explains why it wrote the rule, the detectives will do better." They tested this by giving the programmers four types of explanations:
- Perfect: The explanation matches the rulebook exactly.
- Too Picky: The explanation adds extra rules the rulebook didn't have.
- Too Vague: The explanation leaves out important rules.
- Totally Wrong: The explanation says something completely different.
The result? The explanations didn't help at all. In fact, they sometimes made things worse.
When the explanation was too vague (under-specified), the programmers actually became less accurate than if they had no explanation at all. But here's the kicker: those vague explanations made the programmers feel more confident (rating their confidence 4.25/5 compared to 3.99/5 with no note). It's like a tour guide giving you a fuzzy, half-true map that makes you feel like you know the city perfectly, even though you're walking in circles. The researchers suggest that these bad explanations might actually interfere with your thinking rather than just failing to help.
How the Detectives Solved It (and Failed)
To understand why this happened, the researchers interviewed 10 programmers while they solved these puzzles out loud (a "think-aloud" study). They found that most detectives used a "top-down" approach: they read the robot's English explanation first, then checked the code.
They used five main tricks:
- Clause Comparison: Checking if the words in the rule matched the code.
- Logic Walk-through: Tracing the steps in their head.
- Positive Examples: Trying a "good" example to see if it worked.
- Negative Examples: Trying a "bad" example to see if it broke the rule.
- Intuition: Just guessing based on a "gut feeling."
The study found that when a rulebook was easy to check word-for-word (clause comparison), the detectives did much better. But when they relied on intuition, they were wrong more often than right (7 out of 12 times). Also, catching a wrong rulebook took significantly more time (about 123 seconds) than accepting a wrong one as correct (about 97 seconds). It seems our brains are lazy; it's easier to nod along than to dig for errors.
What This Means for the Future
The researchers suggest that we can't just rely on AI to write these rulebooks and expect us to catch the mistakes easily. We are surprisingly bad at spotting AI errors, and bad explanations can trick us into feeling confident about our mistakes.
They also found that while we struggle to find logical errors, we can tell the difference between a "strong" rulebook and a "weak" one. If a rulebook catches more potential bugs, we rate it as more complete, even if we can't spot the specific logic error.
So, the next time an AI assistant writes a rule for you, don't just nod and say "Great job!" The researchers warn that you might be missing the errors, and a confusing explanation might just make you feel too sure about it. The real challenge isn't just generating the rules; it's helping humans actually verify them.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.