Can your AI agent be cheaper? Investigating the effects of task specifications on token spend in agentic coding tasks
This paper investigates how task specifications impact token consumption in agentic coding workflows, demonstrating that overly brief prompts significantly increase costs while introducing a predictive method to estimate token spend across different configurations with high accuracy.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the modern world of software development, a new kind of worker has emerged: the artificial intelligence agent. Unlike a simple chatbot that answers a single question, these agents are designed to tackle complex, multi-step jobs, such as fixing a bug in a computer program or building a new feature from scratch. They do this by reading a task description, thinking through the problem, using digital tools to write and test code, and repeating the process until the job is done. However, this autonomy comes with a price tag. Every time the AI thinks, reads, or writes, it consumes digital resources measured in "tokens," which are the basic units of text the model processes. These tokens translate directly into money. As these agents become more common in real-world systems, engineers face a pressing question: how much will it cost to let an AI solve a problem, and can that cost be controlled?
The answer is not as simple as picking a cheaper computer model. Recent studies have shown that even when an AI is given the exact same instructions, the amount it spends can vary wildly from one attempt to the next. Sometimes, two identical runs of the same task can differ in cost by a factor of thirty. This unpredictability makes budgeting difficult. Furthermore, while researchers have looked at how different AI models perform, they have largely ignored how the way a human describes the task itself changes the price. A senior engineer might write a detailed, structured specification for a software fix, while a junior engineer might paste a raw error message. Do these different descriptions lead to different costs, or does the AI just do the same amount of work regardless of how the request is phrased?
To find out, a researcher at Stanford University conducted a large-scale experiment using a specific AI model known as Kimi K3. The team selected five distinct software repair tasks from a standard testing suite used by developers. For each task, they created a wide variety of instructions, ranging from a complete, highly structured document containing every possible detail to a bare-bones description with almost no information. They also tested three different levels of "thinking effort," which essentially controls how hard the AI tries to reason through the problem before acting. In total, they ran the system 2,700 times, carefully tracking how much money was spent on each attempt and how many times the agent had to turn to the computer to get things done.
The results revealed a clear and surprising pattern. The way a task is described has a massive impact on the average cost. When the researchers stripped a full, detailed specification down to a simple, plain-language story of what the user wants, the cost of solving the problem jumped by nearly thirty percent. This happened consistently across every task they tested. It seems that when the AI is not given enough specific details, it has to spend more time and money figuring out the missing pieces on its own. However, the study also found that the instructions did not change the unpredictability of the cost. No matter how detailed or vague the prompt was, the difference between one run and the next remained roughly the same. The cost of a single attempt is inherently unstable, and the only way to make the total bill more predictable is to make the individual attempts cheaper.
Another key discovery was that the level of detail in the instructions matters most when the AI is given a limited budget for thinking. When the AI is allowed to think deeply and extensively, the difference in cost between a detailed prompt and a vague one shrinks. This suggests that if you give the AI enough time to reason, it can fill in the gaps of a poor description on its own, but if you restrict its thinking time, a clear and detailed prompt becomes essential to save money. The researchers also found that removing specific sections of a task description, like the list of edge cases or the success criteria, had a small effect on its own, but removing the entire user story or the testing scenarios caused a dramatic spike in cost.
Perhaps the most practical finding for anyone using these tools is a method for predicting costs on new tasks. The team showed that you do not need to run a task dozens of times to know how much it will cost. By running a single, cheap test on a new problem, they could predict the cost of that same problem under many different instruction styles and thinking settings with reasonable accuracy. Without this single test, any guess about the cost of a new task was likely to be off by more than one hundred percent. With just one small measurement, the error dropped to about thirty-six percent. This means that engineers can quickly gauge the financial impact of their choices without wasting resources on extensive trial and error.
The study concludes that while the AI's behavior is naturally unpredictable, the cost of its work is heavily influenced by how humans frame the request. A well-structured, detailed task description can significantly reduce the money spent, especially when the AI is not given unlimited time to think. The research suggests that the most effective way to manage these costs is not to try to eliminate the randomness of the AI, but to design the task instructions carefully and use a single, low-cost test to calibrate expectations for new jobs. As these agents become more integrated into daily work, understanding the relationship between the words we use and the money we spend will be just as important as the code they write.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.