💬 NLP

How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

This paper presents the first systematic study of token consumption in agentic coding tasks, revealing that agentic workflows are exceptionally expensive and stochastic, models vary significantly in efficiency, and frontier LLMs struggle to accurately predict their own token costs.

Original authors: Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Longju Bai, Zhemin Huang, Xingyao Wang, Jiao Sun, Rada Mihalcea, Erik Brynjolfsson, Alex Pentland, Jiaxin Pei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a highly skilled, but slightly unpredictable, digital handyman to fix a leak in your house. You don't know if he’ll finish in ten minutes or spend three days tearing up your entire floor, and you only find out the total bill once he’s finished and the tools are put away.

This research paper is essentially a "financial audit" of these digital handymen (called AI Agents). The researchers wanted to know: How much do these agents actually cost, where does the money go, and can they tell you the price before they start working?

Here is the breakdown of what they found, using some everyday analogies:

1. The "Black Hole" of Costs (Input vs. Output)

When you chat with a standard AI (like a quick text message), it’s like buying a coffee—cheap and predictable. But when you give an AI an "agentic task" (like "fix this entire software project"), it’s like hiring a contractor.

The researchers found that these tasks are 1,000 times more expensive than a simple chat. Most importantly, the cost isn't coming from the AI "talking" (the output); it’s coming from the AI "reading" (the input).

The Analogy: Imagine you hire a researcher. You aren't paying them for the three sentences they write in their report; you are paying them for the 500 books they had to read and keep on their desk just to understand your question. In AI terms, the "books" are the massive amounts of code the agent has to re-read over and over again to keep track of what it's doing.

2. The "Lost in the Woods" Problem (Efficiency)

You might think that if an agent spends more time and "brainpower" (tokens) on a problem, it will eventually solve it. The researchers found this isn't true.

Sometimes, an agent gets stuck in a loop. It reads a file, tries to fix it, fails, reads the same file again, tries something else, and fails again.

The Analogy: It’s like a person trying to find their way out of a forest. Instead of walking in a straight line, they keep walking in circles. They are burning calories (money) and getting more tired, but they aren't actually getting any closer to the exit. In fact, the more "energy" they burn, the less likely they are to succeed because they are just spinning their wheels.

3. The "Expert vs. Robot" Gap

If you asked a human plumber how long a job would take, they’d give you a pretty good estimate. But the researchers found that human experts are actually bad at predicting how much an AI will spend.

The Analogy: A human might look at a leaky faucet and say, "That's a 10-minute job." But the AI agent might look at that same faucet and decide it needs to inspect every single pipe in the entire house first. The "difficulty" a human sees is totally different from the "complexity" the robot experiences.

4. Can the AI give you a Quote? (The Prediction Test)

The most important question was: Can we ask the AI, "Hey, before you start, how much is this going to cost me?"

The researchers let the AI "scout" the job first and then give a quote. The result? The AI is a terrible accountant.

The Analogy: It’s like asking a contractor, "How much will this renovation cost?" and they say, "$500," only for you to receive a bill for $5,000. The AI almost always underestimates the cost. It’s "optimistic" to a fault—it thinks it can solve the problem easily, but it doesn't realize how many "books" it will end up having to read once it actually gets into the messy details.

The Bottom Line

The paper concludes that as we move from "AI you talk to" to "AI that does work for you," the economics change completely. We are moving from a world of "pay per word" to a world of "pay per project," and right now, those projects are expensive, unpredictable, and the "contractors" (the AIs) aren't very good at telling us what the final bill will be.

Technical Summary: "How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks"

1. Problem Statement

As AI agents transition from simple chat interfaces to autonomous systems capable of long-horizon tasks (like software engineering), their token consumption is growing exponentially. This creates a significant economic challenge:

Lack of Transparency: Users cannot predict the final cost of a task until it is completed.
Financial Risk: Users may pay for massive token expenditures even if the agent fails to solve the problem.
Pricing Complexity: Current pricing models struggle to account for the stochastic (unpredictable) nature of agentic workflows.

The paper seeks to answer three fundamental questions: Where do agents spend tokens? Which models are most efficient? and Can agents predict their own costs before they start working?

2. Methodology

The researchers conducted the first systematic empirical study of token consumption in agentic coding.

Framework & Benchmark: They used the OpenHands agent framework to solve tasks on SWE-bench Verified, a benchmark consisting of real-world GitHub issues.
Models Evaluated: Eight frontier LLMs were tested, including various versions of Claude (Sonnet 3.7, 4, 4.5), GPT (5, 5.2), Qwen3-Coder, Kimi-K2, and Gemini-3-Pro.
Data Collection: They extracted fine-grained metrics from full execution trajectories, including per-type token costs (input, output, cached input, cache creation, and cache read), monetary costs, and specific tool-use actions (file views, modifications).
Prediction Task: To test predictability, they implemented a "self-prediction" task where the agent is asked to inspect the repository and estimate its own input/output token usage and total cost before executing the actual fix.

3. Key Contributions

First Large-Scale Empirical Study: Provides a comprehensive dataset and analysis of token dynamics in agentic workflows.
Identification of "Inverse Scaling" in Agents: Demonstrates that more tokens do not necessarily equal higher accuracy.
Behavioral Analysis of Inefficiency: Connects high token costs to specific redundant behaviors (e.g., repeated file viewing/editing).
Benchmarking Self-Prediction: Establishes a new task for evaluating an agent's ability to perform behavioral self-modeling.

4. Key Results

Agentic Tasks are Uniquely Expensive: Agentic coding consumes 1,000x to 3,500x more tokens than standard code reasoning or chat. Crucially, input tokens (driven by context accumulation) dominate the cost, not output tokens.
Stochasticity and Redundancy: Token usage is highly variable; the same task can differ by up to 30x in cost across different runs. High-cost runs are often characterized by "unproductive exploration"—repeatedly viewing and editing the same files without making progress.
The Accuracy-Cost Paradox: Accuracy often peaks at intermediate cost levels and saturates or even declines at higher costs. This suggests that excessive token spending often reflects "looping" or inefficient reasoning rather than deeper problem-solving.
Model Efficiency Variance: Models differ wildly in efficiency. For example, Kimi-K2 and Claude-Sonnet-4.5 consumed over 1.5 million more tokens on average than GPT-5 for the same tasks.
Weak Predictability: Frontier models fail to accurately predict their own costs. They show only weak-to-moderate correlations (max Pearson $r \approx 0.39$ ) and systematically underestimate the actual token usage, particularly for input tokens.
Human vs. Agent Complexity: Human-rated task difficulty is a poor predictor of agent token cost (Kendall $\tau_b = 0.32$ ), revealing a gap between human perception of "hard" tasks and the computational effort agents actually expend.

5. Significance

This research has profound implications for the economics of AI:

For Users: It highlights the need for "budget-aware" agents and early-warning systems to prevent unexpected bills.
For Providers: It suggests that subscription models may be unsustainable for agentic workflows, and that providers should focus on improving "token efficiency" (reducing redundant actions) as a competitive advantage.
For Researchers: It identifies a new frontier in AI development: moving beyond just "accuracy" to optimize for "accuracy-per-token" and developing models capable of better self-modeling and resource planning.