🤖 machine learning

Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

This paper reveals that aggressive post-training quantization causes reasoning models to "overthink" by generating unnecessary intermediate steps and failing to output correct answers found earlier, a problem that can be effectively mitigated by applying a training-free logit penalty to specific overthinking markers to reduce reasoning length while preserving accuracy.

Original authors: Sanae Lotfi, Polina Kirichenko, Steven Li, Zechun Liu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sanae Lotfi, Polina Kirichenko, Steven Li, Zechun Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Overthinking" AI

Imagine you have a brilliant student (an AI) who is great at solving math problems. Usually, they solve a problem, write down the answer, and stop.

Now, imagine you put this student in a noisy, crowded room where they can't hear themselves think clearly (this represents Quantization, a technique used to make AI models smaller and faster).

The paper discovered something surprising: When the AI is in this "noisy room," it doesn't just get wrong answers. Instead, it gets confused. It finds the correct answer halfway through its thinking process, but then it starts doubting itself. It thinks, "Wait, maybe I'm wrong? What if I try a different way?" It opens new mental doors, questions its own logic, and spirals into a long, redundant loop of thinking.

Eventually, it runs out of time or memory before it can actually say the answer it found earlier. It "overthinks" itself into a failure.

The Discovery: It's Not a Lack of Brains, It's a Lack of Stopping Power

The researchers looked at thousands of failed attempts by these "noisy" AI models. They found that in up to 52% of the failures, the AI had actually figured out the right answer in the middle of its reasoning.

The Old Assumption: "The AI is too dumb to solve the problem because it's been shrunk down."
The New Reality: "The AI is smart enough to solve it, but the 'noise' makes it second-guess itself so much that it never stops talking to give the answer."

It's like a chef who cooks a perfect meal, tastes it, realizes it's delicious, but then starts worrying, "Is it too salty? Maybe I should add more salt? Or maybe I should cook it again?" The food gets ruined not because the chef can't cook, but because they can't stop cooking.

The Diagnosis: The "Wait" and "But" Triggers

To understand why this happens, the researchers compared the "noisy" AI to a "clear" AI (the original, full-size version). They looked at the specific words the AI generated.

They found that the "noise" specifically messes with hesitation words.

Stable Words: Words like numbers, math symbols, and formatting (e.g., "255", "sqrt", "answer") remain clear and stable.
Unstable Words: Words that signal doubt or branching paths (e.g., "Wait," "But," "Alternatively," "Maybe") become much more likely to appear when the AI is confused.

When the AI is at a point where it's already unsure (high uncertainty), the "noise" makes it much more likely to pick a word like "Wait." This triggers a new chain of thought, causing the AI to abandon the correct path it was already on.

The Solution: The "Silence Button"

The researchers came up with a clever, free fix that doesn't require retraining the AI.

They created a list of 50 "overthinking markers" (words like "Wait," "But," "However," "Reconsider"). During the AI's thinking process, they apply a tiny penalty to these specific words.

Think of it like a strict teacher tapping the student on the shoulder every time they start to say, "Wait, maybe I should..." The teacher doesn't stop them from thinking; they just gently discourage the doubtful phrasing.

The Results:

Shorter Thinking: The AI stops spiraling. It finishes its reasoning 12% to 23% faster.
Better Accuracy: Because it stops doubting itself, it actually gets the right answer more often. In some cases, accuracy improved significantly (e.g., jumping from 47% to 61% on a math test).
No Extra Cost: This fix requires no extra computing power or training time. It's just a simple rule applied while the AI is talking.

The Big Picture

The paper shows that when we shrink AI models to make them faster, we accidentally make them indecisive. They don't lose their ability to calculate; they lose their ability to trust their calculations.

By simply telling the AI to "stop hesitating" (penalizing words like "Wait" and "But"), we can make these smaller, faster models perform almost as well as the giant, expensive ones, without them wasting time talking in circles.

In short: Quantized models don't fail because they can't think; they fail because they can't stop thinking. A little bit of discipline (a penalty on hesitation words) fixes the problem.

Technical Summary: Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

Problem Statement

Post-training quantization (PTQ) is a standard technique for deploying Large Language Models (LLMs) efficiently by reducing precision without requiring retraining. However, its impact on reasoning models (models capable of Chain-of-Thought, or CoT, generation) remains poorly understood. While prior work suggests quantization degrades accuracy, this paper identifies a specific, counter-intuitive failure mode: aggressive quantization causes reasoning models to overthink.

Instead of failing to find the correct answer, quantized models frequently reach the correct solution in intermediate reasoning steps but fail to commit to it. They subsequently open redundant reasoning branches, express self-doubt, and spiral into excessive deliberation, ultimately producing incorrect final answers or failing to terminate. This behavior leads to significantly longer CoT lengths and reduced accuracy, particularly under aggressive quantization (e.g., 3-bit).

Methodology

1. Empirical Analysis of Quantization Effects

The authors evaluated five reasoning-specialized LLMs (ranging from 1.5B to 32B parameters, including DeepSeek-R1-Distill variants and QwQ-32B) across five benchmarks (MATH-500, GSM8K, AIME-120, LiveCodeBench, GPQA-Diamond). They applied three quantization methods:

Weight-only: GPTQ and AWQ at 3-bit and 4-bit.
End-to-end: FlatQuant (W4A4KV4, W8A8KV8).

They measured the correlation between accuracy degradation and CoT length increase, finding a strong negative correlation ( $\rho = -0.73$ ), suggesting that longer reasoning traces actively contribute to performance degradation.

2. Error Categorization and Diagnosis

The authors categorized model failures into four types:

Overthinking: The model finds the correct answer but abandons it for new, incorrect reasoning paths.
Logical Error: The reasoning trajectory is wrong from the start.
Arithmetic Error: The plan is correct, but calculation fails.
Formatting/Hallucination: Output format issues or hallucinated constraints.

They found that under aggressive 3-bit AWQ quantization, overthinking errors accounted for 52% of all failures (up from 26% in full-precision BF16), a 7.3× increase in absolute frequency.

3. Token-Level KL Divergence Analysis

To understand why overthinking increases, the authors computed the token-level Kullback-Leibler (KL) divergence between the output distributions of full-precision (BF16) and quantized (3-bit AWQ) models under identical generation prefixes.

High-KL Tokens: The tokens with the highest divergence were "hesitation" and "branching" markers (e.g., "Wait", "But", "Alternatively", "maybe", "verify").
Low-KL Tokens: The tokens with the lowest divergence were mathematical and formatting tokens (e.g., numbers, operators, <EOS>).
Entropy Correlation: There is a strong correlation ( $\rho = 0.92$ ) between position-level KL divergence and the next-token entropy of the full-precision model. Quantization noise disproportionately affects positions where the model is already uncertain (high entropy). At these high-entropy positions, overthinking markers appear 2–4× more frequently in the top-20 predictions compared to low-entropy positions.

4. Intervention: Training-Free Logit Penalty

Based on the diagnosis, the authors proposed a training-free intervention. They curated a set $S$ of 50 "overthinking markers" (tokens expressing hesitation or redirection). During decoding, a fixed logit penalty $\lambda$ is applied to these tokens:
$z'_t(v) = z_t(v) - \lambda \quad \text{if } v \in S$
This reduces the probability of sampling hesitation tokens without requiring additional forward passes or retraining.

Key Results

Efficiency and Accuracy Trade-offs

CoT Reduction: The logit penalty consistently reduced CoT length by 12–23% across all models, quantization methods, and benchmarks.
Accuracy Preservation/Improvement: In most cases, accuracy was preserved or improved. For example, on Qwen-1.5B with 3-bit AWQ, accuracy on MATH-500 improved from 47.0% to 61.2%, while CoT length dropped from 23.4k to 12.9k tokens.
Overthinking Error Reduction: The intervention reduced overthinking errors by up to 58% (e.g., from 139 to 58 errors on MATH-500 with 3-bit AWQ).
Pareto Frontier: The penalty shifted the accuracy-vs-reasoning-cost frontier favorably. Penalizing random tokens or low-KL tokens (mathematical tokens) either had no effect or catastrophically degraded performance (increasing CoT by up to 41% and dropping accuracy by 9.5%).

Model and Quantization Robustness

The results held across:

Models: 1.5B, 7B, 14B, 8B, and 32B parameter models.
Quantization: 3-bit and 4-bit weight-only, and 4-bit/8-bit end-to-end quantization.
Benchmarks: Math, coding, and science QA tasks.

Even full-precision (BF16) models showed a reduction in CoT length (5.6–10.0%) with the penalty, indicating that quantization exacerbates an existing tendency rather than creating a wholly new failure mode.

Significance and Claims

The paper claims three primary contributions:

Identification of Overthinking: It demonstrates that aggressive PTQ does not merely erode reasoning capability but specifically induces "overthinking," where models reach correct intermediate answers but fail to commit due to self-doubt and redundant branching.
Mechanistic Diagnosis: Through KL divergence analysis, it identifies that quantization noise disproportionately destabilizes "hesitation" and "branching" tokens at high-entropy positions, while leaving computational (mathematical) tokens stable.
Training-Free Mitigation: It validates that a simple, training-free logit penalty on curated overthinking markers can significantly improve the efficiency-accuracy trade-off. This approach yields a favorable Pareto frontier, reducing reasoning costs by ~20% while maintaining or improving accuracy, offering a practical solution for deploying quantized reasoning models.

The authors emphasize that this work provides a diagnostic framework for understanding how compression affects reasoning traces and offers a targeted, low-overhead intervention to restore the efficiency of quantized models.