💬 NLP

Do LLMs Overthink Basic Math Reasoning? Benchmarking the Accuracy-Efficiency Tradeoff in Language Models

This paper introduces LLMThinkBench, a comprehensive benchmark and empirical study of 53 LLMs that reveals a critical accuracy-efficiency tradeoff, demonstrating that models often "overthink" basic math problems by generating excessive tokens without improving accuracy and sometimes suffering significant performance drops when reasoning budgets are constrained.

Original authors: Gaurav Srivastava, Aafiya Hussain, Sriram Srinivasan, Xuan Wang

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaurav Srivastava, Aafiya Hussain, Sriram Srinivasan, Xuan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, over-enthusiastic assistant named "LLM" (Large Language Model). You ask them a simple question: "What is 234 plus 567?"

A normal human would say, "801," in about two seconds.

But your assistant, LLM, decides to write a 50-page thesis. They explain the history of addition, the concept of place value, the philosophy of carrying numbers, and then they write three different proofs to verify the answer. They even check their work using a method they invented on the spot. Finally, they give you the answer: 801.

But here's the twist: sometimes, in the middle of all that extra talking, they get confused, trip over their own words, and accidentally tell you the answer is 802.

This paper, "Do LLMs Overthink Basic Math Reasoning?", is basically a report card on this behavior. The researchers from Virginia Tech built a new testing ground called LLMTHINKBENCH to see if these AI models are actually getting smarter by thinking longer, or if they are just talking themselves into a corner.

Here is the breakdown of their findings using simple analogies:

1. The "Overthinking Score" (The Efficiency Meter)

The researchers realized that just checking if the answer is right isn't enough. You also need to know how much "effort" (or computer tokens) it took to get there.

They created a new metric called the Overthinking Score. Think of it like a fuel-efficiency rating for a car.

Car A gets you to the store (correct answer) but uses 100 gallons of gas (lots of words).
Car B gets you to the store using only 1 gallon.
Even if Car A is slightly faster, Car B is the better car because it's efficient.

The Overthinking Score punishes models that are accurate but wasteful. It forces them to be both smart and concise.

2. The Big Surprise: Bigger Isn't Always Better

You might think a bigger, more powerful AI (like a 70-billion-parameter model) would be better at math than a smaller one.

The Reality: Not necessarily. The study found that mid-sized models (around 14 billion parameters) often do the best job. They are like a well-trained athlete: strong enough to do the job, but not so bulky that they get in their own way.
The "Giant" Problem: The biggest models often get "bloated." They try to show off their knowledge by writing long, confusing explanations that actually make them more likely to make a mistake.

3. The "Reasoning" Trap

There is a new trend in AI where models are trained specifically to "think step-by-step" (Chain-of-Thought). The idea is that if they explain their work, they will be smarter.

The Paper's Finding: For simple math, this is a disaster. These "Reasoning Models" act like a student who is so afraid of making a mistake that they check their homework 10 times.
The Result: They use 18 times more computer power (tokens) than normal models, yet they often get the answer wrong more often. It's like using a sledgehammer to crack a nut, but the sledgehammer is so heavy it smashes the table.

4. The "Token Budget" Cliff

The researchers tested what happens if they tell the AI: "You only have 1,000 words to solve this."

Normal Models: They just get to the point faster. No big deal.
Reasoning Models: They collapse. Because they are trained to write long essays, when you cut their word count, they panic. They run out of space before they finish their "thought process," and their accuracy drops by nearly 36%.
Analogy: Imagine a marathon runner who is trained to run 26 miles. If you tell them, "You only have 1 mile to run," they don't just run faster; they trip and fall because their legs are used to a different rhythm.

5. The Four Ways They Waste Time

The researchers looked at thousands of AI answers and found that the "wasted" words usually fall into four bad habits:

Redundant Verification: Checking the answer they just got right, three or four times.
Self-Contradiction: Saying "The answer is 5," then "Wait, maybe it's 4," then "No, it's 5 again," confusing themselves.
Irrelevant Exploration: Going down rabbit holes that have nothing to do with the math problem.
Stalling: Just typing random letters or repeating words because they don't know when to stop.

6. The Takeaway: Less is More

The paper concludes that longer thinking does not equal better thinking for basic tasks.

The Best Models: The most practical models for everyday tasks are often the smaller, instruction-tuned ones that give short, direct answers.
The Future: We need to teach AI models to know when to stop. They need to learn that for a simple math problem, a short answer is better than a long, fancy one.

In a nutshell: The paper argues that current AI models are suffering from "analysis paralysis." They are so eager to show they are smart that they talk too much, use too much energy, and sometimes forget the actual answer. The solution isn't to make them think harder; it's to make them think smarter and stop talking when the job is done.

1. Problem Statement

While Large Language Models (LLMs) achieve high performance on complex mathematical benchmarks (e.g., GSM8K, MATH), they frequently fail at basic arithmetic reasoning. A critical, under-explored issue is "overthinking": the tendency of models, particularly those trained for reasoning, to generate excessively verbose chain-of-thought (CoT) traces for trivial problems. This verbosity often:

Does not improve accuracy: Models may generate hundreds of tokens explaining place values or verifying simple additions yet still arrive at incorrect answers.
Increases computational cost: Unnecessary token generation wastes inference resources.
Creates fragility: When token budgets are constrained, reasoning models often suffer "catastrophic collapse" in accuracy, whereas concise models remain stable.
Lacks principled metrics: Existing benchmarks focus solely on final accuracy, ignoring the efficiency-accuracy tradeoff, and static datasets are prone to data contamination.

2. Methodology: LLMTHINKBENCH

The authors introduce LLMTHINKBENCH, a comprehensive framework designed to evaluate the efficiency of reasoning in LLMs.

A. Task Space

The benchmark consists of 14 deterministic basic math tasks categorized into four groups:

Basic Arithmetic: Sorting, Comparison, Sum, Subtraction, Multiplication, Division, Absolute Difference.
Extremum Detection: Find Max, Find Min.
Statistics: Mean, Median, Mode.
Counting: Odd Count, Even Count.

Dynamic Generation: To prevent memorization, test instances are generated dynamically using cryptographic seeding. Parameters include list lengths (8–64) and values (Uniform[-1000, 1000]).
Scale: 53 models were evaluated, including base, instruction-tuned, and reasoning variants (e.g., GPT-4.1, Qwen3, Phi-4, Llama-3, Gemini).

B. The Overthinking Score (Metric)

The paper proposes a new metric to quantify the tradeoff between correctness and efficiency, moving beyond accuracy-only evaluation.

Token Efficiency ( $E_t$ ): Normalized based on the minimum and maximum token counts observed across all models.
Overthinking Score ( $O$ ): Calculated as the harmonic mean of Accuracy ( $A$ ) and Token Efficiency ( $E_t$ ):
$O = \frac{2 \cdot A \cdot E_t}{A + E_t}$
Rationale: The harmonic mean heavily penalizes imbalance. A model with 90% accuracy but 10% efficiency scores significantly lower than a balanced model, forcing the evaluation to favor models that are both correct and concise.

C. Evaluation Protocol

Parsing: A hierarchical extraction system (98.7% success rate) parses answers from diverse formats (LaTeX \boxed{}, explicit markers, code blocks).
Conditions: Evaluations were run under unconstrained budgets, constrained budgets (e.g., 1,024 tokens), and various quantization levels (FP16, 8-bit, 4-bit).

3. Key Contributions

Formalization of Overthinking: Defined overthinking as a state where a model generates more tokens than a concise alternative while achieving equal or lower accuracy.
LLMTHINKBENCH Framework: An open-source Python package (llmthinkbench) and public leaderboard providing a standardized protocol for dynamic, reproducible evaluation of reasoning efficiency.
The Overthinking Score: A novel metric that integrates accuracy and token efficiency, revealing performance rankings that differ significantly from accuracy-only leaderboards.
Large-Scale Empirical Study: Evaluation of 53 models across multiple families, quantization settings, and reasoning budgets.

4. Key Results & Findings

Finding 1: The Basic-Math Paradox

High performance on complex benchmarks (e.g., GSM8K) does not translate to basic math. Models scoring >95% on GSM8K often score <75% on basic arithmetic, suggesting reliance on pattern matching rather than compositional computation.

Finding 2: Non-Monotonic Scaling & Diminishing Returns

Scaling: Larger parameter counts do not guarantee better basic math performance. Some mid-sized models (e.g., Qwen3-14B) outperform larger siblings (Qwen3-32B).
Budgets: Extending reasoning budgets yields diminishing returns. Models like GPT-5 and O-series show zero accuracy gain when moving from "medium" to "high" effort levels, despite generating significantly more tokens.

Finding 3: Overthinking Hurts Correctness

Reasoning models generate ~18x more tokens than standard models but often achieve lower accuracy.

Example: Phi-4-reasoning uses ~6,066 tokens (vs. 379 for Phi-4) but scores 72.23% vs. 78.92%.
Catastrophic Collapse: When constrained to 1,024 tokens, reasoning models suffer massive accuracy drops (e.g., Phi-4-reasoning drops from 72% to 53%), indicating they rely on fixed, long exploration patterns rather than adaptive computation.

Finding 4: Pathological Reasoning Patterns

Analysis of ~5,000 traces revealed four wasteful patterns dominating long chains:

Redundant Verification: Repeatedly checking the same correct answer.
Self-Contradiction: Questioning correct intermediate steps and introducing errors.
Irrelevant Exploration: Generating mathematically valid but unnecessary steps.
Stopping Failures: Infinite character repetition or random token generation after the answer is found.

Finding 5: Quantization Robustness

Large models (≥32B) retain strong reasoning ability even under aggressive 4-bit quantization, while small models suffer significant degradation. This suggests overthinking is a behavioral artifact of training (CoT supervision) rather than a hardware/precision limitation.

Finding 6: Root Cause

CoT Supervision is the primary driver of overthinking. Matched pairs (e.g., Phi-4 vs. Phi-4-reasoning) show that training for long chains shifts the optimization target toward producing plausible text rather than verifiable computation. Prompt engineering and tool use reduce verbosity slightly but do not eliminate the gap.

5. Significance & Implications

Rethinking Evaluation: The field must move beyond accuracy-only metrics. The Overthinking Score provides a more holistic view of model utility, highlighting that smaller, instruction-tuned models (e.g., GPT-4.1-mini, Qwen3-14B) are often more practical for deployment than massive reasoning models.
Training Paradigms: Current CoT training encourages "performative verbosity." Future training should focus on adaptive stopping, step-level verification, and efficiency-aware rewards to prevent models from generating unnecessary text.
Deployment: For basic arithmetic and deterministic tasks, "thinking" models are often suboptimal due to high latency and cost. Concise, instruction-tuned models offer a better efficiency-accuracy balance.

Conclusion

The paper demonstrates that "more thinking" does not equal "better reasoning" in LLMs. Instead, current reasoning models often suffer from a fundamental inability to modulate their computational effort based on problem difficulty, leading to wasteful overthinking that degrades both efficiency and accuracy. The authors call for a shift in benchmarking and training to prioritize adaptive, concise reasoning.