Don't Blame the Large Language Model: How Agent Harness Evolution Shapes Coding Agent Quality
This paper presents the first controlled longitudinal study demonstrating that the rapid evolution of agent harnesses, rather than the underlying large language models, is a primary driver of quality fluctuations in coding agents, as evidenced by a deep dive into 35 sequential releases of the Qwen Code CLI.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a world where your computer doesn't just wait for you to type commands, but actually does the work for you. This is the era of "coding agents"—smart, autonomous programs that can read code, fix bugs, and even write new software all on their own. But here's the secret sauce: these agents aren't just the brain (the giant AI model); they also need a body and a nervous system to function. This body is called the "agent harness." Think of the AI model as a brilliant but lazy genius who needs a manager to hand them the right tools, remind them of the rules, and organize their thoughts. The harness is that manager. It decides what tools the AI can use, how much information to show it, and how to loop back and try again if the first attempt fails. Everyone assumes that if you keep hiring better managers and upgrading the office, the genius will get smarter and work faster. But what if the manager is actually making things worse by adding too many rules or confusing instructions? That's the big question this paper asks: as these "manager" systems get updated constantly, are they actually helping the AI do a better job, or are they just making it slower and more expensive?
The authors of this paper decided to play detective to solve a mystery that software developers have been complaining about. They noticed that whenever these coding agents got an update, they often started making more mistakes or taking way longer to finish tasks. People usually blamed the AI brain itself, thinking the model had gotten dumber. But the researchers suspected the real culprit was the "manager" (the harness) changing its mind too often. To prove this, they set up a very strict experiment. They picked one specific AI model and locked it in a cage so it couldn't change at all. Then, they took the "manager" software (specifically the Qwen Code CLI) and ran it through 35 different versions, from the very first release to the latest one. For each version, they gave the agent 50 real-world coding puzzles to solve, using the exact same AI brain every time.
The results were surprising and a bit funny. Despite the "manager" software getting updated constantly—sometimes releasing new versions more than twice a day—the AI's ability to actually solve the puzzles didn't get any better. In fact, the success rate stayed roughly the same, hovering around 30%, no matter how many times they updated the software. The early versions were just as good at fixing bugs as the fancy, new, complicated versions. However, there was a huge catch: the newer versions were incredibly wasteful. The latest versions of the manager made the AI use nearly double the amount of "tokens" (which are like the fuel or currency the AI burns to think) and made twice as many tool calls as the old versions. It was like upgrading a car engine to a super-complex version that uses twice as much gas but doesn't make the car go any faster.
The researchers dug deeper to find out why this was happening. They looked at the code changes in the manager software and found that when developers added lots of new features or tried to fix many small bugs at once, the AI got confused and wasted more energy. They also discovered that certain parts of the manager's "body" were dangerous to touch. Changing how the AI talks to its brain (the "LLM Provider" layer) or how it remembers things (the "Context Management" layer) often caused the agent to stumble and fail. On the other hand, fixing security holes or adding simple extensions seemed safe. The biggest shocker? The software team had no way of knowing these problems were happening. Their automatic tests only checked if the software crashed, not if the AI was actually doing a good job or wasting money. It was like a factory that only checks if the assembly line is moving, but never checks if the cars coming off the line actually have wheels.
In the end, the paper suggests that we need to stop blaming the AI brain for these mistakes. The real issue is that the "manager" software is evolving too fast without a proper quality check. The developers are adding so many features and changes that the AI gets bogged down, using more resources to achieve the exact same results as before. The authors call for a new kind of "quality assurance" that specifically checks if the agent is still efficient and effective after every update, rather than just checking if it still turns on. Until then, upgrading these coding agents might just be paying more for the same performance.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.