💬 NLP

Thinking Hard, Not Smart: Reasoning Models Fail to Ration Test-Time Compute Across Questions

This paper introduces an exam-style evaluation framework revealing that current reasoning models fail to strategically allocate a shared test-time compute budget across multiple questions, instead behaving as greedy sequential solvers that prioritize presentation order over question difficulty or value.

Original authors: Chenrui Fan, Yize Cheng, Ming Li, Yongyuan Liang, Tianyi Zhou, Soheil Feizi

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenrui Fan, Yize Cheng, Ming Li, Yongyuan Liang, Tianyi Zhou, Soheil Feizi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are sitting in a giant, high-tech library where the books are written by super-smart computers called "reasoning models." These computers are amazing at solving puzzles, but they have a strange quirk: the harder the puzzle, the more they "think" before answering. This thinking process costs something called "compute," which is like a limited supply of energy or a strict time limit on a test. Scientists have long known that if you give these computers more time to think on a single hard problem, they get better at it. But a new question has popped up: What happens when you give the computer a whole stack of different puzzles, but only one shared time limit for the entire stack? Can the computer decide which puzzles are worth spending its energy on, or does it just plow through them in the order they appear, wasting its time on the easy ones and running out of steam before it even sees the hard ones? This is the big mystery researchers are trying to solve, because if these smart computers can't manage their own energy across a group of tasks, they might fail in real-world situations where they need to juggle many jobs at once.

This paper, titled "Thinking Hard, Not Smart," dives into that exact problem. The researchers set up a digital "exam" for several of the world's most advanced reasoning models. Instead of giving each question its own time limit, they gave the whole exam a single, shared budget of "thinking tokens" (a unit of computer effort). The exam included questions of varying difficulty, some worth more points than others, and the models had to figure out how to split their limited energy to get the highest total score. The results were a bit of a shock: the models failed to act like strategic exam-takers. Instead of skipping a hard, low-point question to save energy for an easy, high-point one, they behaved like a greedy student who just reads the questions from top to bottom. They spent a huge amount of effort on the first few questions, often running out of tokens before they even got to the end of the paper, regardless of how many points the later questions were worth.

The study found that these models are "position-blind" strategists. They don't seem to care if a question is labeled "Hard" or "Easy," or if it's worth 20 points or 5. They mostly just follow the order the questions are presented. If a question is first, they pour all their energy into it; if it's last, they often ignore it completely. The researchers tested this by shuffling the order of the questions and changing their point values, but the models didn't adapt. Even when the researchers gave the models a special prompt telling them to "plan ahead" and "allocate wisely," the models didn't change their habits. They still spent their energy in the same way, just spreading the same lack of strategy a bit more evenly. The paper suggests that while these models are getting better at "thinking hard" on a single problem, they haven't yet learned how to "think smart" about which problems are worth thinking about.

To make this clearer, imagine you have a backpack with a strict weight limit of 10 pounds, and you are at a flea market with 20 items. Some items are heavy but worth a lot of money (high value, high cost), some are light but worth very little (low value, low cost), and some are heavy but worthless. A smart shopper would look at the whole list, pick the items that give the best "value per pound," and fill the bag to get the most money. But these AI models acted like a shopper who just grabs the first item they see, then the second, then the third, stuffing them in until the backpack is too heavy to lift. By the time they reached the back of the line, they had no room left for the best deals. The researchers found that as the number of questions (or items) grew from 5 to 20, the models got worse at covering the whole list. For example, with 20 questions, the models only did serious work on about 4 to 8 of them, leaving the rest completely untouched.

The paper also looked at whether the models were reacting to the difficulty of the questions. It turns out they do spend more time on hard questions, but only after they have already started them. They don't look at a hard question and say, "This looks tough, I'll skip it to save energy for later." Instead, they dive in, realize it's hard, and keep going until they run out of energy. This is what the authors call "reactive" rather than "prospective" thinking. They are good at solving a problem once they start, but bad at deciding which problems to start in the first place.

Interestingly, the researchers tested this on both math problems and coding problems, and the results were the same. Whether the task was solving an equation or writing a computer program, the models behaved like a train on a fixed track: they moved forward, one question at a time, until the fuel ran out. They didn't jump tracks to the easier or more valuable stops. The study concludes that this ability to manage a shared budget across multiple tasks is a distinct skill that current models lack. They know how to think hard, but they haven't figured out how to ration their thinking. This isn't just a small glitch; it suggests that as we ask these models to do more complex, multi-step jobs, we might need to teach them a new kind of "metacognition"—the ability to think about their own thinking and manage their resources, not just solve the puzzle in front of them.

Technical Summary: Thinking Hard, Not Smart

Problem Statement

Current reasoning language models increasingly leverage test-time compute to improve performance on individual problems. However, existing evaluations typically assess these models one question at a time, assigning an independent budget to each. This setup fails to capture a critical constraint in real-world deployment: global budget allocation. When multiple problems compete for a single, finite inference budget (e.g., a token limit or latency constraint), a model must make strategic decisions about how to distribute its compute resources. It must determine which problems are worth attempting, when to skip or defer them, and how much effort to invest in each to maximize the total score.

The paper posits that while models may be capable of "thinking hard" on a single problem, they lack the metacognitive control to "think smart" across a set of competing problems. Specifically, it is unclear whether models can estimate the value of a question relative to its cost and dynamically adjust their reasoning strategy to optimize global utility, akin to solving a knapsack problem where the weights (costs) are not explicitly known.

Methodology

Evaluation Framework: The "Exam" Setting

The authors introduce an exam-style evaluation framework to probe shared-budget allocation.

Task: A model is presented with $N$ questions simultaneously, each with a visible point value ( $v_i$ ).
Constraint: The model receives a single, shared budget of $B$ reasoning tokens for the entire exam.
Objective: Maximize the total score (sum of points for correctly answered questions).
Mechanism: The model generates a single reasoning trace for the whole exam. The prompt does not enforce a specific order. After the reasoning phase, the model is asked to provide final answers based on the trace.

Dataset and Construction

Source: Primary experiments use Omni-MATH (Gao et al., 2024).
Variables: Exams are constructed with $N \in \{5, 10, 20\}$ questions.
Scoring Schemes: To test sensitivity to value, the authors vary point assignments:
- Fixed: All questions worth 10 points.
- Random: Scores 1–15, independent of difficulty.
- Aligned: Harder questions receive higher scores.
- Reversed: Easier questions receive higher scores.
Ordering: Questions are presented in random, ascending difficulty, or descending difficulty orders to decouple presentation position from intrinsic difficulty.

Models Evaluated

The study covers a range of open-weight and frontier models:

Open-Weight: DeepSeek-R1-Distill-Qwen (7B, 14B) and Qwen3 (8B, 14B, 32B).
API Models: DeepSeek-V4 Flash and Pro (preview versions).
Generalization: Findings are validated on a code reasoning domain using CRUXEval-O.

Metrics

To analyze allocation behavior, the authors define:

Token Effort ( $t_i$ ): The total number of reasoning tokens attributed to question $i$ .
Work Set ( $W$ ): Questions receiving substantive effort (defined as $\ge 200$ tokens or $\ge 2$ reasoning segments).
Solving Order: The sequence in which questions in $W$ are substantively addressed, determined by the token-weighted centroid of their reasoning segments.
Correlation Analysis: Spearman correlations are computed between allocation behaviors (effort, order) and signals (presentation position, difficulty, point value). Partial correlations are used to control for confounds between position and difficulty.

Key Findings

1. Sequential, Not Strategic, Allocation

Models behave largely as greedy sequential solvers rather than strategic allocators.

Position Dominance: The strongest predictor of both effort and solving order is the presentation position. Models prioritize questions based on the order they appear in the prompt.
Neglect of Later Questions: As the number of questions ( $N$ ) increases, reasoning becomes increasingly concentrated on the beginning of the exam. The correlation between presentation position and token effort is strongly negative ( $\rho \approx -0.48$ at $N=20$ ), while the correlation between position and solving order is strongly positive ( $\rho \approx +0.68$ ).
Insensitivity to Value: Models show negligible sensitivity to stated point values. The correlation between effort/order and point values remains near zero across all conditions.

2. Reactive, Not Prospective, Difficulty Handling

Models do not proactively allocate more compute to difficult questions before attempting them.
While harder questions do receive more tokens, this is a reactive effect: once a model enters a difficult question, it continues reasoning until the budget is exhausted or the problem is solved. This effect weakens as the budget tightens (i.e., as $N$ increases), suggesting models cannot pre-emptively identify high-cost problems to skip or defer.

3. Budget Pressure Amplifies Failure

Coverage Shrinkage: As the exam length grows, the fraction of questions receiving substantive effort (coverage) drops monotonically. For $N=20$ , coverage falls to roughly 40% for open models and as low as 23% for API models.
Zero-Token Rate: A significant fraction of questions (up to 69% for DSV4-F at $N=20$ ) receive no reasoning tokens at all, indicating they are effectively ignored.

4. Planning Prompts Do Not Solve the Issue

Explicit instructions to "plan the allocation" or "consider difficulty and cost" improve the coverage (models attempt more questions) but fail to induce value-aware prioritization.
Models still follow the presentation order and do not adapt their strategy when question values or difficulties are re-ordered or re-priced.

5. Generalization to Code Reasoning

The same behavioral patterns (position-driven solving, insensitivity to value) extend to code reasoning tasks on CRUXEval-O, indicating this is a fundamental limitation of current reasoning architectures rather than a domain-specific artifact.

Contributions

Evaluation Framework: The paper introduces a controlled "exam-style" framework to study shared-budget reasoning, distinguishing it from conventional per-question evaluations.
Trace-Based Analysis: It provides a granular, trace-based analysis of realized reasoning effort and solving order, revealing the gap between object-level reasoning capability and strategic control.
Empirical Evidence: It establishes that current reasoning models, despite their ability to solve complex individual problems, fail to exhibit the metacognitive control required to allocate compute strategically across competing tasks.

Significance and Claims

The paper concludes that global budget allocation is a distinct capability that is not captured by conventional evaluations. The authors argue that current models "know how to think hard about the question in front of them, but not how to decide which question is worth thinking about."

The findings suggest that simply scaling up reasoning models or improving per-question performance will not suffice for scenarios with finite global resources. The inability to ration compute strategically—specifically the failure to prioritize based on value and the tendency to front-load effort regardless of question worth—represents a significant bottleneck for deploying reasoning models in cost-constrained or multi-task environments. The paper does not propose a solution but highlights this as a critical, unaddressed challenge for the field.