💬 NLP

Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

This paper introduces InflationAgent, a novel routing framework that addresses the "token inflation" gap in agentic LLM systems by predicting workflow-level costs via CoT Branching Entropy and optimizing model selection through a Semantic Exchange Rate, thereby achieving higher accuracy with fewer tokens than existing methods like FrugalGPT.

Original authors: Heming Fu, Shan Lin, Qianqian Xie, Guojun Xiong

Published 2026-08-17

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Heming Fu, Shan Lin, Qianqian Xie, Guojun Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a delivery service for a fleet of robots. Some robots are tiny, fast, and cheap to run, but they sometimes get confused and drop packages. Others are giant, super-smart, and expensive, but they almost never make a mistake. In the world of Artificial Intelligence, these robots are "Large Language Models" (LLMs). They are the brains behind chatbots and coding assistants. Usually, when we ask a question, we just send it to one robot and wait for an answer. But in the real world, things get messy. If the tiny robot gets it wrong, a smart system doesn't just give up; it asks the robot to try again. It might even ask the giant robot to step in. This is called an "agentic system"—a team of AI agents working together to solve a problem.

The problem is that we've been counting the cost of these deliveries all wrong. We've been paying for the price of a single trip, forgetting that if the robot crashes and has to drive back to the warehouse to try again, we are paying for the whole round trip, not just the one-way ticket. This paper is about a new way to manage these AI teams so we don't accidentally burn through our budget by sending the wrong robot on a job that requires too many retries.

The Hidden Cost of "Try Again"

Imagine you are at a carnival with a game booth. You have a limited amount of money (tokens) to play. You can choose between a cheap, rickety machine that costs $1 per try, or a fancy, high-tech machine that costs $10 per try.

The old way of thinking was simple: "The cheap machine is $1, so I'll pick that one!" But here's the catch: the cheap machine is tricky. If you ask it a hard question, it might get it wrong. So, you have to pay $1 again to try a second time. And a third. And a fourth. By the time you finally get the right answer, you've spent $5 on the cheap machine. Meanwhile, the fancy machine would have gotten it right the first time for $10.

This paper calls this hidden extra cost "Token Inflation." It's the gap between what you think you are paying (one try) and what you actually pay (five tries). The authors found that for difficult tasks, this inflation can be huge. A small, cheap model might end up costing 4.25 times more than expected because it keeps failing and needing retries.

The New Strategy: The "InflationAgent"

The researchers built a new system called InflationAgent to fix this. Instead of just looking at the price tag of a single try, this system acts like a savvy carnival manager who knows exactly which machines are prone to breaking down.

Here is how it works, step-by-step:

1. The "Gut Check" (CoT Branching Entropy)
Before the system even asks a robot to solve a problem, it does a quick, free test. It asks a small, local robot to think about the problem three different times.

If the robot gives the same answer all three times, the system knows the problem is easy.
If the robot gives three totally different, confused answers, the system knows the problem is hard and the robot is likely to get stuck in a loop of retries.
The authors call this measurement CoT Branching Entropy. It's like checking if a car engine is sputtering before you even start the trip. This costs nothing extra because it happens on a local computer, not on the expensive cloud servers.

2. The "Price Tag" Calculator
Using that "gut check," the system predicts how much the "inflation" will be. It doesn't just guess; it uses a small, smart calculator (a machine learning model) trained on past data. It predicts: "If we send this hard question to the small robot, it will probably fail 3 times, so the real cost is 3x the price."

3. The "Best Deal" Selector
Now the system calculates the Semantic Exchange Rate (SER). This is a fancy way of asking: "How much accuracy do I get for every dollar I actually spend?"

If the small robot is cheap but will fail 4 times, its "deal" is terrible.
If the big robot is expensive but will get it right the first time, its "deal" might actually be better.
The system picks the robot that gives the best value, not just the cheapest sticker price.

4. The "Fresh Start" Rule
This is the most important trick. If the system sends a question to a robot and it fails, and then it decides to escalate to a bigger, smarter robot, it throws away the old failed attempt.
The researchers discovered something surprising: if you show the big robot the messy, confused notes from the small robot's failed attempt, the big robot gets confused too! It's like showing a genius student a page of scribbles from a struggling student; the genius might start doubting themselves.
By giving the big robot a fresh prompt (a clean slate), the system keeps the accuracy high. If they had just passed the failed notes along, the big robot's accuracy dropped by 34.8 percentage points on hard questions.

What They Found

The team tested this on math problems (GSM8K) and tricky trivia questions (HotpotQA). Here is what happened:

The Inflation is Real: On hard trivia questions, the small model (Qwen2.5-7B) had an inflation rate of 4.25×. It meant that for every dollar you thought you were spending, you were actually spending $4.25 because the robot kept failing.
Better Results for Less Money: When they set a fixed budget (a strict limit on how many tokens they could spend), their new system (InflationAgent) got 94.7% of the answers right. The old popular method (FrugalGPT) only got 91.0% right.
Saving Tokens: To get the same 91.0% accuracy that the old method achieved, the new system used 31% fewer tokens. That's a massive saving.
The Danger of "More Retries": They found that just giving the small robot more chances to try (up to 10 times) didn't work forever. After a certain point, the robot got confused by its own past failures, and its accuracy actually went down. Sometimes, it's better to switch to a smarter robot than to keep banging your head against the wall.

The Bottom Line

This paper shows that in the world of AI, "cheap" isn't always cheap. If a model is likely to fail and need to try again and again, it becomes expensive and inefficient. By measuring how likely a model is to get stuck (using the "gut check" entropy) and by starting fresh when switching to a smarter model, we can save money and get better answers.

The authors are confident in these results based on their tests with specific math and trivia datasets. They suggest that while this works great for reasoning tasks, we still need to figure out how to apply it to more open-ended conversations or tools. But for now, they've proven that a little bit of smart planning goes a long way in keeping the AI budget from inflating out of control.

Technical Summary: Not All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM Systems

Problem Statement

Modern agentic systems built on Large Language Models (LLMs) frequently employ retry mechanisms when a model fails to answer a query correctly or produces an incomplete reasoning chain. While existing routing systems (e.g., FrugalGPT, RouteLLM) optimize costs based on the price per token of a single call, they fail to account for the cumulative cost of retries.

The authors define Token Inflation as the ratio between the true workflow cost (including all retries) and the estimated single-call cost.
$\text{Inflation}(m, x) = \frac{C_{\text{true}}(m, x)}{C_{\text{direct}}(m, x)}$
Where $C_{\text{true}}$ is the sum of tokens consumed across $k^*$ attempts until success (or exhaustion of retries), and $C_{\text{direct}}$ is the token count of a single attempt.

The paper demonstrates that this inflation is non-negligible and highly variable. For difficult multi-hop tasks, a "cheap" small model may fail repeatedly, resulting in a true cost 4.25 times higher than the single-call estimate. Consequently, routing decisions based solely on single-call estimates can systematically underestimate real costs by more than 2×, leading to suboptimal model selection.

Methodology: InflationAgent

To address this, the authors propose InflationAgent, a four-stage routing pipeline designed to maximize the Semantic Exchange Rate (SER), defined as expected accuracy divided by predicted true cost:
$\text{SER}(m, x) = \frac{U(m)}{\hat{C}_{\text{true}}(m, x)}$

The pipeline operates as follows:

Feature Extraction (Local): The system extracts structural features from the query (token count, vocabulary diversity, multi-hop cues) without API calls.
CoT Branching Entropy (CBE) Calculation (Local): The system samples $K$ independent reasoning chains from a local model. It calculates the Shannon entropy of the resulting answer distribution:
$\text{CBE}(x) = -\sum_{c} p_c \log_2 p_c$
High entropy indicates divergent reasoning paths and high query difficulty, serving as a zero-cost proxy for retry likelihood.
Inflation Prediction & Routing: A lightweight Multi-Layer Perceptron (MLP) predicts the inflation factor ( $\hat{\text{Inflation}}$ ) using four features: CBE, agreement score, average chain length, and prompt token count. The system estimates the true cost ( $\hat{C}_{\text{true}} = C_{\text{direct}} \times \hat{\text{Inflation}}$ ) and selects the model tier with the highest SER.
Execution with Fresh-Escalation: The selected model executes the query. If the observed inflation exceeds 2× the prediction, the system escalates to a stronger model tier. Crucially, this escalation uses a fresh prompt, discarding the failed reasoning chain to prevent context contamination.

Key Contributions

Systematic Measurement of Token Inflation: The authors instrumented an agentic retry harness across three model tiers (Qwen2.5-7B, GPT-4o-mini, GPT-4o) and two datasets (GSM8K, HotpotQA). They found that inflation varies significantly by task type, reaching 4.25× for a 7B model on multi-hop QA, whereas GPT-4o on the same task inflated by only 2.92×.
CoT Branching Entropy (CBE): A novel pre-execution difficulty signal computed entirely via local inference. It predicts high inflation with an AUROC of 0.887, enabling the system to anticipate retry overhead without incurring API costs.
Semantic Exchange Rate (SER) Routing: A routing objective that balances accuracy against predicted true cost, rather than just single-call cost.
Fresh-Escalation Policy: An empirical finding that forwarding failed reasoning chains to a stronger model degrades performance. The authors show that contaminating GPT-4o with failed chains from a smaller model reduces its accuracy by 34.8 percentage points on difficult queries, validating the necessity of discarding failed context during escalation.

Experimental Results

The evaluation was conducted on GSM8K and HotpotQA with a fixed token budget and a maximum of 5 retries.

Accuracy vs. Cost: On GSM8K, InflationAgent achieved 94.7% accuracy compared to 91.0% for the baseline FrugalGPT. To achieve FrugalGPT's accuracy level, InflationAgent required 31% fewer tokens.
Inflation Prediction: The MLP predictor achieved a Pearson correlation of 0.714 and an AUROC of 0.887 on the held-out test set for detecting high-inflation queries (threshold ≥2×).
Marginal Returns: Increasing retries for a small model (Qwen2.5-7B) showed diminishing returns; accuracy peaked at 5× the budget (92.5%) and dropped to 82.0% at 10× due to context window interference from failed chains.
Context Contamination: In a controlled experiment, GPT-4o accuracy dropped from 73.9% to 39.1% when fed failed reasoning chains from Qwen2.5-7B, a statistically significant drop ( $p=0.0078$ ).

Significance and Claims

The paper claims that token inflation is a critical, previously overlooked factor in agentic system design. By ignoring retry overhead, existing routing strategies can inadvertently select models that are "cheap" per token but expensive per successful task.

The authors assert that InflationAgent resolves this by:

Explicitly modeling the gap between single-call estimates and true workflow costs.
Using local inference (CBE) to predict this gap without additional API costs.
Implementing a "fresh-escalation" strategy that prevents the degradation of downstream model performance caused by context contamination.

The work establishes that for difficult, multi-hop tasks, a nominally cheap model with high inflation is neither cost-effective nor effective. The proposed framework offers a path to more efficient agentic inference by aligning routing decisions with the true economic cost of task completion.

Limitations

The authors note that their evaluation is limited to two reasoning benchmarks (GSM8K, HotpotQA) and does not extend to open-ended generation, multi-turn dialogue, or tool-use agents. The inflation predictor is trained on specific task distributions, and cross-task generalization remains an open question. Additionally, the compute unit ratios used are fixed and do not account for dynamic API pricing.