Coding Agents Don't Know When to Act
The paper introduces FixedBench to demonstrate that state-of-the-art coding agents suffer from an "action bias," frequently proposing unnecessary code changes for already-resolved issues because they fail to recognize when inaction is the appropriate response.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Core Problem: The Over-Eager Handyman
Imagine you hire a very smart, automated handyman to fix things around your house. You give them a list of "broken" items (bug reports). Their job is to look at the list, find the broken item, and fix it.
The problem this paper investigates is what happens when the handyman receives a list item that isn't actually broken anymore. Maybe a neighbor fixed it yesterday, or maybe it was a false alarm.
A smart handyman should look at the item, realize, "Oh, this is already perfect," and say, "I'll leave this alone."
However, the researchers found that current AI coding agents are like over-eager, nervous handymen. Even when they see that a piece of code is already working perfectly, they feel a strong compulsion to touch it, tweak it, or "improve" it. They can't resist the urge to act, even when doing nothing is the best choice.
The Experiment: The "Fixed" Test
To prove this, the researchers created a special test called FIXEDBENCH.
- The Setup: They took 200 real-world coding tasks where the problem had already been solved by a human. The code was perfect.
- The Trap: They gave these tasks to five different top-tier AI coding agents.
- The Goal: The agents were supposed to look at the code, realize it was already fixed, and submit an "empty" patch (meaning: "No changes needed").
- The Result: The agents failed miserably. Instead of saying "All good," they went ahead and changed the code in 35% to 65% of the cases. They added unnecessary changes, creating "technical debt" (messy code that wasn't needed), just because they felt they had to do something.
Why Does This Happen? (The "Action Bias")
The paper calls this an Action Bias.
Think of it like a student who has been trained for years to solve math problems. If you hand them a piece of paper that says "2 + 2 = 4," and ask them to "solve the problem," they might feel compelled to write down a long, complicated proof or change the numbers, even though the answer is already right. They are trained to produce an answer, not to evaluate if an answer is needed.
The AI models are trained to generate code patches. They haven't been trained well enough to recognize when the job is already done.
The "Instructions" Experiment
The researchers tried giving the agents different instructions to see if they could fix this behavior.
- "Just Fix It" (The Bad Instruction): When they told the agent, "Edit the code to fix this issue," the agents got even worse. They changed the working code even more often.
- "Check First" (The Better Instruction): When they told the agent, "First, try to reproduce the bug. If you can't find the bug, stop and don't change anything," the agents did much better. They learned to pause, check, and realize, "Hey, this is already fixed."
- The Catch: This new instruction worked great for already fixed code. But it created a new problem. If the code was partially broken (half-fixed), the agents became too cautious. They would check, see some progress, and then decide to do nothing at all, leaving the code broken.
It's like a security guard who, after being told "Don't shoot if the person is already surrendering," decides that if a person is partially surrendering, they shouldn't shoot or intervene, even if the person is still dangerous.
The Root Cause: Training vs. Reality
The paper suggests that the AI models are suffering from a lack of "common sense" regarding their own success.
- Current Training: The models are rewarded for making changes. If they fix a bug, they get a "good job" score. If they do nothing, they get no score.
- The Reality: In the real world, a "good job" sometimes means doing nothing.
The researchers argue that to fix this, we need to change how we train these AI. We need to teach them that abstaining (doing nothing) is a valid and successful outcome, just as much as fixing a bug is.
Summary
- The Issue: AI coding agents are too eager to change code, even when it's already perfect.
- The Evidence: In a test of 200 "already fixed" tasks, the agents unnecessarily changed the code up to 65% of the time.
- The Cause: They are trained to act, not to know when to stop.
- The Fix: Telling them to "verify first and stop if it's fixed" helps, but it makes them too passive when code is only partially broken.
- The Lesson: We need to teach AI that sometimes, the best code change is no change at all.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.