Prompt-Induced Waste in Large Reasoning Models: A Preregistered Two-Harness Benchmark of Coding Agents
This preregistered benchmark study demonstrates that prompt wording and harness design significantly inflate the reasoning costs of large coding agents—often by 2.4 to 30 times—without improving task success, with specific instructions like "develop multiple approaches" and certain harness architectures being the primary drivers of this inefficiency.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are hiring a super-smart robot assistant to fix a broken toaster. You don't just want the robot to fix it; you want it to think about how to fix it first. In the world of artificial intelligence, these "thinking" steps are called reasoning tokens. Just like a human might scribble notes, cross things out, and try different ideas before solving a puzzle, these AI models spend time and money "deliberating" before they give you an answer.
But here's the catch: you pay for every single thought the robot has, even the ones it throws away. If you ask the robot to "think really hard" or "try five different ways," it might do exactly that, burning through your budget while fixing the toaster no better than if you had just asked it to "fix the toaster." This paper explores a strange corner of computer science where researchers act like detectives, trying to figure out which words in your instructions make the robot waste money and which words keep it on a tight budget. They aren't just guessing; they are running thousands of controlled experiments to see exactly how much different phrases cost.
The Great Prompt Experiment: Why Your Words Cost Money
Imagine you have a fleet of six different super-intelligent robot chefs. You want them to write code (which is like a recipe for a computer) to solve 24 specific cooking challenges. But there's a twist: you have to pay for every single thought the chefs have while they are cooking. The researchers wanted to know: Does the way you write your order change how much the chefs think, and does that thinking actually help the dish taste better?
To find out, they set up a massive, super-organized kitchen lab. They didn't just guess; they wrote down their rules before they started cooking (this is called "preregistration," like writing a menu before you go to the grocery store). They used two different types of kitchen managers (called "harnesses") to talk to the chefs: one was a direct, no-nonsense manager, and the other was a very chatty, detailed manager who sent a huge amount of background info with every order.
Here is what they discovered after running over 4,600 cooking sessions:
1. The "Try Everything" Trap
The biggest money-waster? Telling the robot to "develop several approaches and compare them."
It sounds smart, right? Like a good chef tasting three sauces before picking one. But for these AI robots, this instruction was a disaster. It made them think 2.4 to 7.4 times more than usual. And the worst part? The final dish (the code) was no better. They just burned extra cash to think about options they never used. It's like paying a chef to write three different recipes for a sandwich, taste them all, and then serve you the same sandwich they would have made if you just asked for "a sandwich."
2. The Magic Words That Do Nothing
People often say "think step by step" or "think deeply" to get better results. The researchers found that for these coding robots, these "deep thinking" magic words were just pure waste. They made the robots think 1.6 to 2.2 times longer without improving the result. It's like telling a calculator to "think really hard" before adding 2 + 2; the answer is the same, but you paid for the extra thinking time.
3. The "Bounded Efficiency" Hack
On the flip side, there was a secret sauce that saved money. If you gave the robot a strict box to work in—telling it exactly what the goal was, what the rules were, and when to stop—it actually thought less. One robot even cut its thinking time in half! This "bounded efficiency" template was free to use and made the robots faster and cheaper without messing up the code.
4. The Manager Matters More Than the Chef
One of the most surprising findings was that who you hire to manage the robot matters more than the robot itself.
They tested the same robot with two different managers. One manager (PI.DEV) was efficient. The other (Claude Code) was like a micromanager who sent a massive 12-to-15-times-larger "introductory speech" with every single message. Even when the robots solved the problem perfectly, the micromanager version cost 5 to 30 times more just because of how the conversation was handled. It turns out, the "manager" (the software wrapping the AI) is often the biggest part of the bill, not the AI itself.
5. The "Hidden Discount" Illusion
The researchers also looked at how the companies charge for these robots. They found that the companies automatically save money by remembering the "introductory speech" they send every time. This saved about 61% of the bill. But the researchers warned: Don't call this "efficiency." It's just a discount on the same old work. The robot didn't actually work faster or smarter; the company just didn't charge you for the parts it remembered.
6. New Robots, Same Old Tricks
When a brand new, super-fast robot (Kimi-K3) was released, the researchers tested it immediately. Even though this new robot was naturally cheaper to run, the same bad instructions (like "try multiple approaches") made it waste money 15 times more than usual. The lesson? No matter how smart or cheap the robot is, bad instructions will always make it overspend.
The Bottom Line
This paper proves that how you ask a question changes how much it costs, even if the answer is the same.
- Don't tell the robot to "try many approaches" or "think deeply" unless you actually need it to do those things.
- Do give it clear boundaries and a stop condition.
- Remember that the software managing the robot might be costing you more than the robot itself.
The researchers made all their data, code, and test recipes public. They want everyone to know that in the world of AI agents, less thinking theater and more clear instructions is the best way to save money and get the job done.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.