🤖 AI

Who Thinks Best Depends on How Long You Let Them: Budget-Dependent Rankings in LLM Evaluation

This paper challenges the assumption of stable large language model rankings by demonstrating that varying inference token budgets significantly alters model performance and order, revealing non-monotonic behaviors, ranking reversals, and model complementarity that necessitate budget-conditioned evaluation protocols.

Original authors: Rodrigo Guedes de Souza, Alison R. Panisson

Published 2026-08-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rodrigo Guedes de Souza, Alison R. Panisson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are in a giant library where the books are written by super-smart robots called Large Language Models (LLMs). These robots can read almost everything and answer questions about math, science, and history. For a long time, people trying to pick the "best" robot just asked them a question and saw who got the right answer first. They assumed that if Robot A was better than Robot B on one test, it would always be better, no matter what. It was like judging a race by only looking at who crossed the finish line first, without caring how much time they were allowed to run.

But here is the twist: these robots don't just "think" instantly. They write their answers one word at a time, like a long chain of thoughts. The more words they are allowed to write, the more they can "think" before giving an answer. This limit on how many words they can write is called a "token budget." Think of it like giving a student a different amount of time to take a test. Some students might need a lot of time to solve a hard puzzle, while others might get confused if they have too much time and start overthinking. The big question for scientists is: Does the "best" robot stay the best if we change the time limit? Or does the winner change depending on how long we let them think? This is exactly what the researchers wanted to find out.

The "Overthinking" Problem: Why More Time Isn't Always Better

In this study, the researchers set up a massive experiment to see how four different AI models performed on three tricky tests: grade-school math, competition-level math, and graduate-level science questions. They didn't just let the models run; they forced them to stop after specific amounts of "thinking" (measured in tokens). They tested seven different limits, ranging from a tiny 64 tokens (very little thinking time) up to a huge 4,096 tokens (a lot of thinking time).

What they discovered is that the idea of a single "best" model is a myth. The rankings of the models flipped and changed depending entirely on how much time they were given.

The "Overthinking" Surprise
You might think that giving a robot more time to think would always help it get the answer right. But the researchers found something weird: sometimes, giving a model more tokens actually made it get the answer wrong. They call this "overthinking."

Imagine a student who knows the answer to a math problem. If you give them 10 minutes, they write it down correctly. But if you give them 30 minutes, they might start doubting themselves, change their answer, and end up with the wrong one. The study found that this happened for about 3% to 19% of the questions. Even worse, this wasn't a problem with the questions themselves; it was specific to each robot. A question that made one model "overthink" and fail might be perfectly fine for another model. It's like how one person might get nervous if they have too much time to prepare a speech, while another person thrives on it.

The Great Ranking Swaps
Because of this, the "winner" of the race kept changing.

On the easier math test (GSM8K), a huge model (LLaMA-3.3 70B) was the champion when given a medium amount of time (256 tokens). But as soon as they gave the models more time (up to 4,096 tokens), a smaller model (GPT-OSS 20B) took the lead and stayed there.
On the hardest science test (GPQA), the smallest model (LLaMA-3 8B) was actually the best when time was tight (256–512 tokens). But as time increased, the bigger models caught up and eventually tied for the top spot.

This means that if you look at a leaderboard today that says "Model X is the best," that statement is only true for a specific amount of time. If you change the time limit, Model X might suddenly be the worst one.

The "Oracle" Gap: Why We Need a Smart Switch
The researchers also asked: "If we could magically pick the perfect model for every single question, how much better would we do?" They built a theoretical "Oracle" (a perfect selector) that chose the best model for each specific question and time limit.

The results were shocking. On the hard science test, this perfect selector could get 27.8 percentage points more correct answers than the single best model running alone. Even on the easier math tests, the gap was significant (up to 16.9 points). This proves that no single robot is good at everything. Sometimes the small robot is best, sometimes the big one, and sometimes they are all wrong.

Can We Build a Smart Switch?
The team tried to build a "router"—a smart system that looks at a question and the time limit, then decides which robot should answer it.

Inside the same type of test: When they trained the router on math problems and tested it on math problems, it worked great. Using the "time limit" as a clue helped it pick the right robot, improving accuracy by 1.6 to 5.7 points.
Across different tests: But when they tried to use the same router to switch from math to science questions, it got confused. The "time limit" clues that worked for math didn't work for science. In fact, using those clues actually made the router perform worse by 1.2 points when switching domains. This suggests that the relationship between "how much time you have" and "getting the right answer" is different for every subject.

The Takeaway

The main lesson from this paper is simple but powerful: There is no single "best" AI model. The answer to "which model is best?" depends entirely on "how long you let them think."

If you are building an app for a phone where you can only give the AI a tiny bit of time, a small, fast model might be your champion. If you are running a server where you can give the AI hours to think, a different, larger model might win. The old way of ranking models—just picking one winner for everyone—is broken. Instead, we need to say, "Model A is best for short answers, Model B is best for long answers, and sometimes, if you give them too much time, they might just mess it up."

The researchers suggest that in the future, we should stop asking for a single score and start reporting how models perform at different time limits. And if we want to use these models effectively, we need smart systems that know when to call the small, quick thinker and when to call the big, slow thinker—because the winner changes the moment the clock starts ticking.

Technical Summary: Budget-Dependent Rankings in LLM Evaluation

Problem Statement

Standard Large Language Model (LLM) evaluation assumes that model rankings are stable and invariant to inference-time conditions. Specifically, it presumes that a model declared "state-of-the-art" on a benchmark (e.g., GSM8K, GPQA) maintains its superiority regardless of the token generation budget—the maximum number of tokens a model is permitted to produce before truncation.

This paper challenges that assumption. It posits that the evaluation landscape is fundamentally budget-dependent. Recent literature on test-time compute scaling suggests that increased token allowances can improve performance, while "overthinking" literature suggests additional tokens can degrade accuracy. However, the interplay between these effects across different models and tasks remains unquantified. The core problem is whether model rankings, item difficulty, and model complementarity are static properties or functions of the token budget.

Methodology

The authors conducted a systematic investigation involving four open-weight reasoning models and three reasoning benchmarks across seven distinct token budgets.

Models: Four models spanning an order of magnitude in parameter count: LLaMA-3 8B, Qwen-3 32B, LLaMA-3.3 70B, and GPT-OSS 20B.
Benchmarks: Three datasets of increasing difficulty:
- GSM8K (1,319 grade-school math problems).
- MATH-500 (500 competition-level math problems).
- GPQA-Diamond (198 graduate-level science questions).
Token Budgets: Seven levels ranging from 64 to 4,096 tokens ( $b \in \{64, 128, 256, 512, 1024, 2048, 4096\}$ ).
Experimental Setup: A total of 56,476 individual inferences were generated at temperature $T=0$ for determinism.
Evaluation Protocol:
- Truncation Handling: The study employs a three-tier analysis to disentangle genuine reasoning failures from truncation artifacts:
  1. All items: Standard scoring where truncated outputs are marked incorrect.
  2. Stop-only: Restricting analysis to items where a specific model completed generation.
  3. Common non-truncated: Restricting to items where all four models completed generation, enabling paired comparisons.
- Metrics: Binary correctness (exact match), McNemar's test for ranking significance, Jaccard similarity for model agreement, and Oracle gap analysis (difference between a per-item oracle ensemble and the best single model).

Key Contributions and Findings

1. Item-Level Behavioral Taxonomy

The authors classify model-item trajectories into four categories: always-correct, monotone-increasing, non-monotone, and always-wrong.

Non-Monotone Behavior: Contrary to the assumption that more tokens always help, 3.6–25.8% of items exhibit non-monotone behavior (accuracy decreases as budget increases), representing "overthinking."
Model Specificity: This phenomenon is largely model-specific. The cross-model overlap of non-monotone items is low (6–14%), meaning an item that triggers overthinking in one model rarely does so in another. This implies that "problematic" items cannot be simply removed from benchmarks to fix the issue.
Validation: Even after filtering out truncation artifacts (using stop-only trajectories), non-monotone rates remain substantial (e.g., 19.1% for LLaMA-3 8B on GPQA), confirming overthinking is a genuine reasoning failure.

2. Statistically Significant Ranking Reversals

Model rankings are not stable; the "best" model changes depending on the budget.

GSM8K: LLaMA-3.3 70B leads at $b=256$ (62.4%), but GPT-OSS 20B overtakes it at $b=512$ and maintains a significant lead at $b=4096$ (94.9% vs. 91.7%).
GPQA: The smallest model (LLaMA-3 8B) ranks first at low budgets ( $b=256, 512$ ), is overtaken by LLaMA-3.3 70B at $b=1024$ , and is eventually matched by GPT-OSS 20B at $b=4096$ .
Significance: These reversals are statistically significant ( $p < 0.01$ via McNemar's test) across all benchmarks, even when controlling for truncation on common non-truncated sets.

3. Oracle Gap Dynamics

An oracle ensemble (selecting the correct model per item) reveals significant potential for improvement via model selection.

Magnitude: The oracle gap (Oracle accuracy minus best single model) ranges from +3.5 pp to +27.8 pp.
Budget Dependence:
- On GSM8K, the gap is non-monotonic, peaking at $b=256$ (+16.9 pp) as models solve different items, then declining as they converge on easier items at higher budgets.
- On harder benchmarks (MATH-500, GPQA), the gap grows monotonically, reaching +12.8 pp and +27.8 pp respectively at $b=4096$ .
Complementarity: Even at high budgets, models solve meaningfully different subsets of items (Jaccard similarity $\approx 0.74$ at $b=4096$ ), indicating persistent complementarity.

4. Budget-Aware Routing Proof-of-Concept

The authors trained XGBoost classifiers to route items to the best model based on text features and token budget.

Cross-Domain Performance: When trained on GSM8K+MATH-500 and tested on GPQA, the router achieved +2.67 pp over the "Best-Per-Budget" baseline, capturing 14.1% of the oracle gap.
Feature Importance: SHAP analysis shows the token budget is the dominant feature (6.1 $\times$ more important than text features like LaTeX presence).
Domain Specificity: While budget features improve performance within a domain (+1.6 to +5.7 pp), they hurt cross-domain transfer (−1.2 pp). This indicates that the relationship between budget and accuracy is domain-specific and does not generalize directly from math to science.

Significance and Implications

The paper argues that the question "which model is best?" has no single answer; it depends on the token generation budget. The authors propose three actionable implications:

Evaluation Protocols: Benchmarks should adopt budget-conditioned evaluation, reporting rankings at multiple budget levels rather than a single static score. This is critical for constrained deployment scenarios (e.g., mobile, edge) where generous budgets are infeasible.
Routing Systems: Model selection and routing systems must treat the token budget as a first-class signal. However, because budget-accuracy mappings are domain-specific, routing systems should combine budget signals with domain-agnostic features or adaptation techniques.
Ensemble Opportunities: The substantial oracle gaps documented represent a concrete opportunity for ensemble and routing methods to exploit model complementarity, potentially yielding significant performance gains over single-model deployments.

The study concludes that the evaluation landscape of LLMs is fundamentally budget-dependent, with model rankings, item difficulty, and complementarity all varying as a function of the inference-time token limit.