💻 computer science

The Token Efficiency Index: A Peer-Benchmarked Composite Indicator for AI Token Efficiency

This paper introduces the Token Efficiency Index (TEI), a peer-benchmarked composite indicator that aggregates cache performance and model usage metrics into a standardized 0–100 score to enable organizations to transparently benchmark AI token efficiency and identify cost optimization opportunities.

Original authors: Caden Wong, Vikram Das, Himanshu Dhami

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Caden Wong, Vikram Das, Himanshu Dhami

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a lemonade stand, but instead of lemons and sugar, you are selling "brain power" to robots. In the world of Artificial Intelligence, companies don't pay by the hour or by the number of employees; they pay by the "token." Think of a token as a tiny grain of sand. Some tasks need a handful of sand, while others need a whole bucket. The problem is that sand comes in different colors and sizes. Some grains are cheap and easy to find, while others are rare, expensive, and require special handling.

For a long time, big companies and startups have been buying buckets of this sand without a clear way to tell if they are getting a good deal. They know how much they spent, but they don't know if they are wasting it. It's like knowing you spent $50 on lemonade ingredients but having no idea if you made 100 cups or just 10. To fix this, we need a way to measure "efficiency"—not just how much you spent, but how smartly you used what you bought. This is where a new tool called the Token Efficiency Index (TEI) comes in. It's like a report card for how well a company uses its AI sand, turning complex spending habits into a simple score from 0 to 100.

The Great AI Lemonade Stand Audit

So, how do you grade a company on how well they use their AI? The authors of this paper, Caden Wong, Vikram Das, and Himanshu Dhami, realized that simply looking at the total bill doesn't tell the whole story. A company might spend a fortune because they are huge, not because they are wasteful. To solve this, they invented the Token Efficiency Index (TEI).

Think of the TEI as a "smart mirror" for AI spending. Instead of just showing you the total cost, it looks at three specific habits to see if you are being a good steward of your resources:

The "Reuse" Habit (Cache Hit Rate): Imagine you are writing a story. If you have to look up the definition of a word every single time you use it, that's slow and annoying. But if you write the definition down once and just point to it later, you save time. In AI, this is called "caching." The TEI checks how often a company reuses information they've already asked for, rather than asking the AI to re-calculate it from scratch.
The "Payoff" Habit (Cache Amortization): This is about getting your money's worth. If you write down a definition (a "cache write"), you want to use it many times (many "reads") before you forget it. The TEI measures if companies are actually using what they saved, or if they are just hoarding notes they never look at.
The "Expensive Choice" Habit (Premium Model Share): Not every question needs a super-genius AI. Sometimes a regular AI is just fine. But some companies use the most expensive, powerful AI for everything, even for simple tasks like "what's the weather?" The TEI checks if a company is using the "Ferrari" for a trip to the grocery store when a "bicycle" would have done the job.

The Scorecard: From Chaos to a 0-100 Grade

The paper explains that they took data from 39 different organizations (ranging from single developers to research institutions) and fed it into a special calculator. They didn't just average the numbers, because that can be misleading. Instead, they used a clever method called "Benefit of the Doubt" (BoD).

Here is how that works in plain English: Imagine a student who is great at math but terrible at art. If you average their grades, they might look average. But if you say, "Okay, let's give them a break on art and see how they look if we focus on their math," they might shine. The TEI does this for companies. It asks, "What is the best possible way to look at this company's data so they look as good as they can?" If a company is still doing poorly even when you give them the "benefit of the doubt," then they really are inefficient.

To make sure the score isn't thrown off by one weird company that happens to be an outlier (like a lemonade stand that accidentally bought a million lemons), they added a "robust" twist. They ran the calculation 500 times, each time picking a random group of peers to compare against. This smooths out the bumps and gives a fairer score.

What the Numbers Say

When they ran this test on their group of 39 organizations, they found some interesting things:

The Score Range: The scores ranged from a low of 34.7 to a high of 100.
The "Super-Efficient" Club: A surprising number of companies—13 out of 39—were so efficient that they actually scored above the perfect score of 100 (technically, their raw score was over 1.0). This means they were performing better than the "best practice" frontier the authors set up.
The Big Winners: The companies that scored highest were the ones who reused their information (high cache hit rates) and didn't waste money on expensive models for simple tasks.
The Big Losers: The company with the lowest score (Org 33) had a score of 34.7. The analysis showed they were using expensive "premium" models for 90.8% of their tasks, when the best-performing peers were only using them for about 14.3%. The paper suggests that if this company fixed this one habit, they could save an estimated $72,150.

What This Paper Is Not Saying

It is important to know what this paper doesn't claim. The authors are very careful to say that this is a simulation and a benchmark, not a magic wand that fixes everything instantly.

It's not a "One Size Fits All" rule: The paper explicitly rules out the idea that there is one single "perfect" way to spend on AI. The TEI changes its weights depending on what each company is good at.
It's not perfect yet: The authors admit their data is limited. They only looked at 39 organizations, and they couldn't see the "outside" world (like how customers use AI products). They also couldn't tell if a company needed a powerful model for a hard task or if they were simply prioritizing performance over cost. So, the "Premium Model Share" metric might sometimes punish a company for being smart, not wasteful.
It's not a crystal ball: The savings numbers (like the $72,150) are "directional estimates." They are educated guesses based on averages, not precise forecasts.

The Takeaway

The Token Efficiency Index is a new way to look at AI spending that moves beyond "how much did we spend?" to "how smartly did we spend?" By turning complex data into a simple 0-100 score, it helps companies see where they are wasting money and where they are doing a great job.

The authors suggest that while this tool is a great start, it needs more data to get better. They hope that in the future, more companies will share their data (anonymously) so the "peer group" gets bigger and the scores get even more accurate. For now, it's a powerful flashlight in a dark room, showing us that with a few small changes—like reusing information and picking the right tool for the job—companies can save a lot of money without losing any brainpower.

Technical Summary: The Token Efficiency Index (TEI)

1. Problem Statement

As enterprise spending on generative AI accelerates—reaching an estimated $37 billion in 2025—organizations face a critical challenge in determining whether their consumption is efficient. Unlike traditional seat-based SaaS costs, AI spending is usage-driven, priced by tokens (input, output, reasoning), and varies significantly by model type. Existing cost-visibility tools are primarily attribution mechanisms; they track what was spent and where, but they fail to measure how efficiently resources were used relative to comparable peers. There is a lack of a standardized, peer-benchmarked framework to condense complex token usage data into an interpretable efficiency score.

2. Methodology

The paper introduces the Token Efficiency Index (TEI), a composite indicator that aggregates multiple dimensions of token spend into a single 0–100 score. The methodology follows the six-stage COOPER framework for non-parametric frontier studies and is grounded in Data Envelopment Analysis (DEA) literature.

2.1. Data and Metrics

The TEI utilizes a dataset of 75 developer accounts grouped into 39 organizations, derived from the open-source Tokscale platform. The index is constructed from three direction-aware metrics representing controllable efficiency levers:

Cache Hit Rate: The ratio of cache reads to total input tokens (higher is better).
Cache Amortization Ratio: The logarithm of the ratio of cache reads to cache writes (higher is better). This log-transformation is applied to handle the wide variance in read/write volumes.
Premium Model Share: The proportion of tokens consumed by "premium-tier" models (lower is better). Tiers are defined relatively based on the price distribution of observed models rather than fixed vendor lists.

2.2. Scoring Framework

The TEI employs a Benefit-of-the-Doubt (BoD) approach, a specific application of Data Envelopment Analysis (DEA).

Endogenous Weighting: Instead of imposing fixed weights, the BoD method solves a linear program for each organization to find the weighting scheme most favorable to that specific organization, subject to the constraint that no organization scores above 1.0 under those weights. This allows organizations to be evaluated on their comparative advantages.
Robust Order-m Extension: To mitigate the sensitivity of standard BoD to outliers (where a single best performer defines the frontier), the TEI uses a robust order-m estimator. Each organization is benchmarked against 500 random subsamples of 15 peers rather than the entire population. This dilutes the influence of extreme outliers and preserves discrimination among top performers (allowing for "super-efficiency" scores > 1.0).
Normalization: Metrics are normalized to a [0, 1] scale using min-max scaling, with direction inversion applied where lower raw values indicate better efficiency. Missing metrics are handled by scoring based on available data while retaining the organization as a peer in others' reference sets.

2.3. Aggregation and Output

The final output includes:

Headline Score: A 0–100 score mapped from the robust BoD score (where 1.0 = 100).
Peer Percentile: The percentage of peers scoring at or below the organization's score.
Frontier-Gap Recommendations: Actionable insights identifying gaps between an organization's performance and a high-performing peer percentile. These include estimated cost savings for model-tier shifts and cache improvements, or qualitative severity tiers for metrics like cache amortization where dollar conversion is infeasible.

3. Key Contributions

First Peer-Benchmarked Composite Indicator for AI: The TEI is presented as the first framework to apply composite indicator methodology specifically to AI token efficiency, filling a gap where no standardized efficiency score previously existed.
Application of Robust DEA to Tokenomics: The paper adapts established BoD and robust order-m methods (typically used for macroeconomic or macro-performance benchmarking) to the unique constraints of AI usage data, which lacks a natural "output" variable for regression-based efficiency scoring.
Transparent and Interpretable Scoring: By using endogenous weights, the TEI avoids arbitrary weighting schemes. It provides organization-specific explanations of why a score was achieved and where improvements can be made.
Implementation and Validation: The authors provide a fully reproducible Python implementation. Evaluation on the 39-organization dataset demonstrates that the robust order-m method produces rankings that are more stable and balanced than standard BoD, while still rewarding genuine strengths. Perturbation analysis confirms the engine responds proportionally and correctly to controlled input changes.

4. Results and Evaluation

Ranking Stability: The robust order-m approach correlates more closely with an equal-weights baseline (Spearman $\rho = 0.54$ ) than standard BoD does ( $\rho = 0.14$ ), suggesting it tempers the aggressive reweighting of standard BoD while maintaining discrimination.
Metric Independence: Diagnostics (pairwise correlations and Cronbach's alpha) confirm that the three selected metrics capture distinct information rather than redundant constructs.
Sensitivity Analysis: Sobol indices indicate that all three metrics contribute meaningfully to score variation, with cache amortization ratio being the largest driver under the robust order-m model.
Actionable Insights: In the pilot dataset, every organization received at least one recommendation. The most common opportunities identified were shifting workloads to lower-cost model tiers and improving cache reuse.

5. Significance and Limitations

The paper claims the TEI offers a transparent, interpretable approach to benchmarking AI token efficiency, moving beyond simple cost attribution to actionable efficiency management. It serves as a foundation for "Tokenomics," extending FinOps principles to AI-specific consumption.

However, the authors maintain a modest scope regarding the current release:

Data Scope: The current index is based on internal developer tool usage (e.g., coding assistants) and does not yet cover externally facing AI applications or end-user token consumption, which would require production API billing data.
Peer Set Size: The benchmark relies on a thin cohort of 39 organizations. While the methodology is robust, the precision of the peer benchmark is limited by sample size.
Metric Limitations: The "Premium Model Share" metric treats all premium usage as inefficient, unable to distinguish between necessary complex tasks and default over-provisioning. The "Cache Amortization Ratio" cannot be converted to a precise dollar saving due to high variance in usage volumes.
Weight Interpretation: The averaged order-m weights are descriptive summaries of the resampling procedure, not the solution to a single optimization problem.

The authors position the TEI not as a finished product, but as a starting point for a broader organizational benchmark, inviting contributions of anonymized usage data to refine the peer set and expand the index's applicability.