💬 NLP

Cost-of-Pass: An Economic Framework for Evaluating Language Models

This paper introduces "Cost-of-Pass," an economic framework that evaluates language model productivity by combining accuracy and inference costs, revealing that complementary model-level innovations are the primary drivers of cost-efficiency and providing a principled tool for guiding deployment decisions.

Original authors: Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou

Published 2026-02-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mehmet Hamza Erol, Batu El, Mirac Suzgun, Mert Yuksekgonul, James Zou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive construction company. You have a huge pile of blueprints (tasks) that need to be built. You have two main ways to get the job done:

Hire a human master builder: They are incredibly skilled, rarely make mistakes, and can solve almost any problem. But they are expensive. You have to pay them a high hourly wage, and they work at a human pace.
Use a fleet of AI robots: Some robots are small and cheap but might get confused easily. Others are giant, super-smart robots that cost a fortune to run but rarely fail. There are also "thinking" robots that take extra time to ponder a problem before answering, which costs more but increases their accuracy.

For a long time, the tech world has only asked: "Which robot builds the best house?" (Accuracy). But this paper asks a much more practical question: "Which robot builds the house for the least amount of money, considering both the robot's price tag and how often it makes mistakes?"

The authors call this new metric "Cost-of-Pass."

Here is a simple breakdown of their findings using everyday analogies:

1. The "Cost-of-Pass" Concept: The Price of a Correct Answer

Imagine you are trying to buy a correct answer to a math problem.

If a cheap robot costs $0.01 to run but only gets the answer right 1 out of 10 times, you have to run it 10 times to get one correct answer. That costs you $0.10.
If an expensive robot costs $1.00 to run but gets it right 100% of the time, you only run it once. That costs you $1.00.

In this scenario, the cheap robot is actually the better deal, even though it's "dumber." The "Cost-of-Pass" is simply the expected total cash you need to spend to get one single correct solution.

2. The "Frontier": The Best Deal Available

The authors look at the entire market of AI models and humans to find the absolute cheapest way to get a correct answer for any given task. They call this the "Frontier Cost-of-Pass."

Think of this like a "Best Price Guarantee" on a shopping site. It tells you: "No matter what, this is the lowest amount of money you will ever need to spend to solve this specific problem today."

3. The Big Discoveries: Who is the Best Bargain?

The paper found that different types of AI models are the "best bargain" for different types of jobs. It's not just about buying the most expensive, powerful model.

For Simple Tasks (like 2+2): The Lightweight Models (the small, cheap robots) are the winners. They are so fast and cheap that even if they make a few mistakes, it's still cheaper than hiring a human or using a giant super-computer.
- Analogy: If you just need to move a single box, you don't need a forklift; a strong person with a dolly is the most cost-effective.
For Knowledge Tasks (like "Who wrote Pride and Prejudice?"): The Large Models (the big, general-purpose robots) are the best. They are smart enough to know the answer without needing to "think" too hard, and they aren't as expensive as the specialized "thinking" models.
For Hard Math & Logic (like solving a complex puzzle): The Reasoning Models (the "thinking" robots) win. Even though they are expensive to run, they are so good at solving these hard problems that they don't need to try 100 times. They solve it on the first try, saving you money in the long run.
- Analogy: If you are trying to defuse a bomb, you don't want the cheapest, fastest person. You want the expensive expert who gets it right the first time, because failing costs you everything.

4. The "Human Baseline": The Price of a Human Expert

To make sure they aren't just comparing robots to robots, the authors calculated how much it would cost to hire a real human expert to solve these problems.

For simple math, a human is way too expensive.
For super-hard math, a human is still expensive, but the new "Reasoning" AI models are now so good that they are actually cheaper than hiring a human tutor to solve the same problem.

5. The "Magic Tricks" (Inference Techniques)

People often try to make AI smarter by using "magic tricks" during the process, like asking the AI to check its own work (Self-Refinement) or asking it to answer the same question three times and picking the most common answer (Majority Voting).

The paper found that most of these tricks are a waste of money.

Analogy: It's like paying a mechanic $50 to double-check a $10 oil change. The extra check might make the car run 1% better, but it costs you $50. The "Cost-of-Pass" goes up, not down.
The only exception was a new, smart technique called TALE-EP, which is like a mechanic who knows exactly when to stop checking and move on. That one actually saved money.

6. The Bottom Line: Progress is Real

The authors tracked this "cheapest possible price" over the last year. They found that for the hardest math problems, the cost to get a correct answer has been cut in half every few months.

This means AI is getting not just smarter, but much more affordable very quickly. The paper concludes that the biggest driver of this progress isn't "magic tricks" or "trying harder," but rather building better, more efficient models in the first place.

In summary: Don't just look at how smart an AI is. Look at the price tag of a correct answer. Sometimes the "dumb" cheap robot is the best deal, and sometimes the expensive genius robot is the only way to save money. The key is matching the right tool to the right job.

1. Problem Statement

The widespread adoption of Large Language Models (LMs) in the economy depends on whether the economic value they generate outweighs their inference costs. Current evaluation metrics focus primarily on accuracy or raw capability, often ignoring the monetary cost of inference. This leads to an incomplete picture of progress, where a model might be highly accurate but economically unviable compared to cheaper alternatives or human labor. Furthermore, existing attempts to incorporate cost often rely on arbitrary fixed budgets or non-monetary heuristics, lacking economic interpretability.

The core problem is the lack of a principled, economically grounded framework to evaluate the trade-off between performance (probability of a correct solution) and cost (monetary inference expense) across different model families and task types.

2. Methodology

The authors adapt Farrell's theory of productive efficiency and stochastic frontier production functions from economics to create a quantitative evaluation framework for LMs.

Core Concepts

Cost-of-Pass ( $v(m, p)$ ): Defined as the expected monetary cost to generate a single correct solution for a specific problem $p$ $p$ using model $m$ $m$ .
- Formula: $v(m, p) = \frac{C_m(p)}{R_m(p)}$
- Where $C_m(p)$ is the expected cost of one inference attempt (tokens $\times$ price), and $R_m(p)$ is the probability of success (accuracy).
- If a model cannot solve a problem ( $R_m=0$ ), the cost-of-pass is infinite.
Frontier Cost-of-Pass ( $V_p$ ): The minimum achievable cost-of-pass across all available strategies (a set of models $M$ $M$ and a human expert baseline $M_0$ $M_{0}$ ) for a given problem.
- $V_p(M \cup M_0) = \min(V_p(M), v(\text{expert}, p))$
- This represents the true economic lower bound for solving a problem.
Human-Expert Baseline: To ground the evaluation, the authors estimate the cost of hiring a qualified human expert to solve a problem ( $v(\text{expert}, p)$ ). This is calculated based on required expertise, time per problem, and market hourly rates.
Gain ( $G_p$ ): The reduction in frontier cost-of-pass achieved by a new model release, measuring its economic contribution relative to the previous best option.

Experimental Setup

Model Categories:
1. Lightweight: Low per-token cost (<$1/M tokens), e.g., Llama-3.1-8B, GPT-4o mini.
2. Large: High-capacity general models, e.g., Llama-3.1-405B, GPT-4o, Claude 3.5 Sonnet.
3. Reasoning: Models with specialized post-training for reasoning, e.g., OpenAI o1, o3-mini, DeepSeek-R1.
Task Categories:
1. Basic Quantitative: Arithmetic (Two-Digit Addition), Grade-school math (GSM8K).
2. Knowledge-Based: Scientific QA (GPQA-Diamond), Bias detection (BBQ).
3. Complex Quantitative: Competition math (MATH-500, AIME-2024).
Inference-Time Techniques: The framework evaluates the economic impact of techniques like Majority Voting, Self-Refinement, and the budget-aware technique TALE-EP.

3. Key Contributions

Theoretical Framework: Introduced Cost-of-Pass, a metric that integrates accuracy and cost into a single economically interpretable unit (expected dollars per correct answer), moving beyond separate reporting of accuracy and cost.
Frontier Analysis: Defined the Frontier Cost-of-Pass relative to a human-expert baseline, allowing for the tracking of economic progress over time and the identification of the most cost-effective strategy for any given task.
Counterfactual Essentialness: Developed a method to quantify the "essentialness" of different model families by measuring how much the frontier cost increases if a specific family (e.g., reasoning models) is removed from the ecosystem.
Empirical Insights: Provided a comprehensive analysis of the LM ecosystem from May 2024 to February 2025, revealing distinct economic niches for different model types.

4. Key Results

A. Distinct Economic Niches for Model Families

Lightweight Models: Most cost-effective for basic quantitative tasks (e.g., Two-Digit Addition, GSM8K). Since accuracy is high across all models for these tasks, the lowest-cost models dominate.
Large Models: Essential for knowledge-intensive tasks (e.g., GPQA-Diamond). They offer the best balance of high accuracy and reasonable cost compared to lightweight models which fail frequently here.
Reasoning Models: Dominate complex quantitative tasks (e.g., AIME, MATH-500). Despite significantly higher per-token costs, their superior accuracy drastically reduces the expected cost-of-pass (fewer retries needed), making them the most economical choice for hard problems.

B. Progress Tracking

The frontier cost-of-pass has decreased exponentially over the past year.
For complex quantitative tasks, the cost of generating a correct solution has roughly halved every 2.6 months (MATH-500) and 7.1 months (AIME-2024).
This progress is driven primarily by model-level innovations (new releases) rather than inference-time techniques.

C. Essentialness of Model Families (Counterfactual Analysis)

Removing lightweight models causes a massive increase in cost for basic tasks.
Removing large models significantly impacts knowledge-based tasks.
Removing reasoning models causes the frontier cost for complex math tasks to skyrocket, confirming their critical role in solving high-difficulty problems.

D. Impact of Inference-Time Techniques

Performance-oriented methods (Self-Refinement, Majority Voting) generally fail to provide net economic benefits. The marginal accuracy gains do not justify the increased token costs and latency.
Budget-aware techniques (TALE-EP) show promise, offering meaningful cost reductions in specific tasks by dynamically managing token usage.

5. Significance and Implications

Rational Model Selection: The framework provides a principled tool for practitioners to select models based on economic viability rather than just raw accuracy. It highlights that "bigger" or "smarter" is not always "cheaper" or "better" depending on the task complexity.
Guiding AI Development: The results suggest that the primary driver of cost-efficiency is complementary model-level innovation (e.g., better reasoning architectures, more efficient lightweight models) rather than post-hoc inference tricks.
Economic Benchmarking: By incorporating a human-expert baseline, the framework quantifies when AI becomes economically superior to human labor, a critical metric for automation adoption.
Limitations & Future Work: The authors acknowledge limitations such as the binary success metric (ignoring partial correctness), potential data contamination in benchmarks, and the exclusion of hybrid human-AI workflows. They propose future extensions to include risk-adjusted metrics and more complex cost structures (e.g., tool usage, verification costs).

In summary, Cost-of-Pass shifts the paradigm of LM evaluation from "How smart is the model?" to "How much does it cost to get a correct answer?" providing a rigorous economic lens for understanding the trajectory of AI productivity.