🤖 AI

OckBench: Measuring the Efficiency of LLM Reasoning

This paper introduces OckBench, the first benchmark to jointly evaluate the accuracy and token efficiency of large language models in reasoning and coding tasks, revealing significant inefficiencies in current models and advocating for a paradigm shift toward optimizing token usage.

Original authors: Zheng Du, Hao Kang, Song Han, Tushar Krishna, Ligeng Zhu

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zheng Du, Hao Kang, Song Han, Tushar Krishna, Ligeng Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring two people to solve a complex puzzle.

Person A looks at the puzzle, thinks for a moment, and says, “The answer is 42.”
Person B spends three hours writing a 50-page essay about the history of numbers, the philosophy of logic, and their childhood memories before finally whispering, “The answer is 42.”

Both people got the right answer. But if you had to pay them by the word they spoke, Person B would cost you a fortune, take forever, and waste a lot of energy.

This paper, OckBench, is essentially a report card that stops judging AI models only on whether they get the right answer. Instead, it judges them on how efficiently they get there. It applies "Occam’s Razor" to AI: Entities must not be multiplied beyond necessity. In other words, don’t use 1,000 words when 10 will do.

Here is the breakdown of what the researchers found, explained simply:

1. The Problem: The "Chatty" AI Tax

Current AI benchmarks are like a multiple-choice test that only cares if you bubble in the right letter. They don’t care if you wrote a novel in the margin to get there.

The researchers noticed that modern AI models (like GPT-5.5 or Claude) have become very good at reasoning, but they’ve also become incredibly verbose. They generate massive amounts of text ("reasoning traces") to solve problems. This is expensive. Every extra word costs money, increases wait times, and burns more electricity.

2. The Solution: OckBench and OckScore

To fix this blind spot, the team created OckBench. It’s a specific set of 200 difficult questions in math, coding, and science.

But here’s the trick: They didn’t just pick random questions. They used a "Differentiation Filter." They specifically chose questions where some AIs give short, smart answers and others give long, rambling ones. This exposes the "efficiency gap."

They also invented a new score called OckScore.

Old Way: Score = Accuracy.
OckScore: Score = Accuracy minus a penalty for being too wordy.

Think of it like a golf score. You want the lowest number (highest accuracy), but if you take 20 swings to sink the putt (too many tokens), your score gets worse. The best players sink it in one or two swings.

3. The Three Big Discoveries

Discovery 1: The "Overthinking Tax"

You might assume that smaller, cheaper AI models are always better for your wallet. OckBench shows this is often false.

Smaller models often suffer from an "Overthinking Tax." Because they aren’t as smart, they try to compensate by talking a lot. They ramble, repeat themselves, and generate huge walls of text to try to find the answer.

Example: A small model might cost pennies per word, but because it writes 100,000 words to solve a problem, it ends up costing more than a smarter model that solves it in 5,000 words.

Discovery 2: The Open vs. Closed Gap

There is a huge difference between "closed" models (like those from OpenAI or Anthropic) and "open-weight" models (like DeepSeek or Qwen, which are available to the public).

Closed Models: These are like efficient surgeons. They cut straight to the heart of the problem.
Open Models: These are like enthusiastic interns. They are getting better at getting the right answer, but they are still very chatty.

The paper shows that an open model might have the same accuracy as a closed model, but it might use 26 times more words to get there. That’s a massive waste of resources.

Discovery 3: "Per-Token Intelligence"

The researchers coined a new term: Per-Token Intelligence. This measures how much "smarts" are packed into each word the AI generates.

High Per-Token Intelligence: The AI says only what is necessary. Dense, precise, efficient.
Low Per-Token Intelligence: The AI fills space with fluff. Sparse, repetitive, inefficient.

Surprisingly, the smartest models tend to have higher Per-Token Intelligence. As models get smarter, they actually get shorter in their responses because they don’t need to ramble to find the answer.

4. Can We Fix It?

Yes. The paper shows that we can teach models to be less chatty without making them dumber.

Mixing Models: By blending a "thinking" model with a standard model, they could cut the word count in half while keeping the accuracy steady.
Training: By training models to penalize themselves for being too long (like a teacher saying, "Be concise!"), they can reduce the "Overthinking Tax."

The Bottom Line

The paper argues that the AI community needs to change its mindset. We shouldn't just ask, "Did the AI get the right answer?" We must also ask, "Did it get there without wasting our time, money, and electricity?"

In the future, the best AI won’t just be the smartest; it will be the most concise. As the title suggests, we need to apply Occam’s Razor: Don’t multiply tokens beyond necessity.

Technical Summary: OckBench: Measuring the Efficiency of LLM Reasoning

Problem Statement
Current evaluations of Large Language Models (LLMs) predominantly focus on accuracy and output quality, neglecting a critical dimension: token efficiency. As state-of-the-art models increasingly rely on long reasoning traces to solve complex tasks, inference has become a "long decoding marathon," leading to significant latency, cost, and energy consumption. The authors argue that this creates a blind spot in evaluation, as models with similar accuracy can exhibit massive variance in output length. For instance, within the same model family, accuracy can remain stable while token usage drops significantly, or conversely, open-weight models may achieve comparable accuracy to proprietary models but at a vastly higher token cost. The paper posits that "Per-Token Intelligence"—the ability to solve problems correctly and concisely—is a fundamental aspect of model capability that current benchmarks fail to capture.

Methodology
To address this gap, the authors introduce OckBench, a hardware-agnostic benchmark designed to jointly evaluate accuracy and token efficiency. The methodology consists of three core components:

Benchmark Composition: OckBench covers three reasoning-heavy domains: Mathematics (e.g., AIME, MATH500), Software Engineering (e.g., MBPP, LiveCodeBench), and Scientific Reasoning (e.g., GPQA, ScienceQA).
Differentiation Filter: Rather than random sampling, the benchmark employs a selection strategy to expose efficiency gaps. It filters for problems where:
- Difficulty Banding: The average accuracy across a diverse set of models falls between 10% and 90%, focusing on the "reasoning frontier."
- Maximizing Token Variance: The top-k instances with the highest variance in token consumption are selected. This ensures the benchmark distinguishes between models that reason concisely and those that produce verbose, redundant traces. The final v1.0 split contains 200 problems (100 Math, 60 Coding, 40 Science).
OckScore ( $S_{Ock}$ ): A unified metric that prioritizes correctness while penalizing verbosity. It is defined as:
$S_{Ock} = \text{Accuracy} - \lambda \cdot \log\left(\frac{T}{C} + 1\right)$
Where Accuracy is the pass rate, T is the average output tokens (including reasoning and answer), $\lambda$ is a penalty coefficient (set to 10), and C is a normalization constant (set to 10,000). The logarithmic penalty compresses the wide range of token lengths, ensuring that efficiency is treated as a cost rather than a multiplier, while maintaining accuracy as the primary utility.

Key Contributions

Per-Token Intelligence: The paper formalizes this concept as a critical evaluation dimension, arguing that stronger models should be accurate and avoid treating tokens as free resources.
OckBench and OckScore: The introduction of an open benchmark and a unified metric that rewards concise, correct reasoning. The "Differentiation Filter" specifically selects tasks that reveal efficiency gaps between models.
Empirical Insights ("Overthinking Tax"): The identification of a systematic efficiency gap between open-weight and proprietary models. The paper defines the "Overthinking Tax," where smaller or less efficient models incur higher deployment costs due to verbose reasoning, despite potentially lower per-token prices.
Optimization Pathways: Demonstration that reasoning efficiency can be improved via training-free model interpolation and difficulty-aware reinforcement learning (RL).

Results
The authors evaluated 37 model/setting entries, including frontier proprietary models (GPT-5.4/5.5, Claude Opus 4.5–4.7, Gemini 2.5–3.1) and open-weight models (DeepSeek, Qwen, Kimi, etc.). Key findings include:

Efficiency Gaps: Models with similar accuracy show large variances in token usage. For example, DeepSeek-V4-Flash (max) achieved 82.5% accuracy but used 26.3× more tokens than GPT-5.4 (medium), which achieved 82.0% accuracy.
Intra-Family Efficiency: Within the Claude Opus family, Opus 4.7 reduced average output length from 28.6k to 7.5k tokens while maintaining nearly the same accuracy as Opus 4.6. Similarly, scaling the Qwen3.5 family from 9B to 397B-A17B improved accuracy from 21.5% to 67.5% while reducing output length by 4.4×.
Leaderboard Performance: GPT-5.5 (medium) ranked highest in OckScore (82.2), balancing high accuracy (86.0%) with moderate token usage (4,692 tokens). Increasing test-time effort (GPT-5.5 xhigh) improved raw accuracy to 90.0% but increased token usage by 2.8×, resulting in a lower OckScore (81.6).
Open vs. Closed Models: Open-weight models are closing the accuracy gap but lag significantly in per-token reasoning density. For instance, Kimi-K2.6 reached 75.0% accuracy but required 42,243 tokens, compared to Gemini-3.1-Pro’s 3,263 tokens for 76.0% accuracy.
Token Distribution: Frontier proprietary models concentrate their token usage in the low-thousand range, whereas open-weight models like DeepSeek-V4-Pro and Kimi-K2.6 exhibit heavy high-token tails.
Optimization Efficacy: Training-free interpolation (Qwen3-Mix-6:4) reduced average output length by ~55% while retaining accuracy. RL-based optimization (DIET) improved accuracy and reduced token usage, raising the OckScore.

Significance
The paper argues for a shift in evaluation paradigms from accuracy-centric metrics to those that account for token efficiency. By introducing OckBench and OckScore, the authors provide a concrete roadmap for improving reasoning ability and token efficiency. They contend that "tokens should not be multiplied beyond necessity," and that future model development should optimize for reasoning density. The findings highlight that token efficiency is not merely a system-level concern but a reflection of model intelligence, with significant implications for deployment costs and energy consumption.