🤖 AI

Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

This paper reveals that low-bit quantization of reasoning models, while preserving accuracy, often induces a hidden cost by inflating the length of chains of thought through increased intermediate steps and repetition, thereby offsetting expected inference speedups and necessitating the reporting of token usage alongside accuracy metrics.

Original authors: Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang

Published 2026-06-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyu Lian, Walid Krichene, Beichen Huang, Masahiro Tanaka, Olatunji Ruwase, Li Zhang, Minjia Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, fast-thinking assistant (a Large Language Model) who solves complex puzzles by writing out a long, step-by-step "thought process" before giving you the final answer. This is how modern AI "reasons."

To make this assistant cheaper and faster to run, engineers often use a technique called Quantization. Think of this like compressing a high-definition movie into a smaller file size. You lose a tiny bit of detail, but the file is much smaller and loads faster. Usually, this works great: the movie still looks good, and it plays quicker.

However, this paper discovers a weird glitch when they apply this "compression" to AI that is doing reasoning.

The Hidden Cost: "Thinking Too Much"

The researchers found that when they compressed the AI's brain (using low-bit quantization), the AI didn't just get slightly slower or less accurate. Instead, it started overthinking.

The Analogy:
Imagine a student taking a math test.

The Full-Precision Model (Original): The student reads the question, thinks clearly, writes down three steps, and gets the right answer. Total time: 5 minutes.
The Compressed Model (Quantized): The student is still smart enough to get the right answer, but because their brain is "compressed," they get confused. They write down the first step, then doubt it, write it again, check it, doubt it again, and write it a third time. They eventually get the right answer, but it took them 15 minutes.

Even though the student got the right grade (accuracy), they wasted three times as much time (compute cost).

The Paper's Key Findings

Accuracy is a Lie (for efficiency): If you only look at whether the AI got the answer right, the compressed model looks fine. But if you look at how much it thought to get there, it's actually much worse. The paper calls this "Token Inflation." The AI generates way more "thinking tokens" (words in its thought process) than it needs to.
The "Overthinking" is Repetitive: The paper analyzed why the AI was taking so long. It wasn't just thinking deeper; it was thinking in circles. The compressed AI would say things like, "Wait, let me check that," or "Let me verify," and then repeat the exact same thought it just had. It was stuck in a loop of self-doubt.
It Slows You Down in Real Life: Because the AI is generating so many extra "thinking" words, the user has to wait longer for the final answer. Even though the computer chip is technically processing each word faster, the sheer volume of extra words cancels out that speed boost. It's like driving a Ferrari, but you're stuck in traffic because you're driving in circles.
Bigger Models Aren't Immune: This happens to both small and large AI models, though the smaller ones get confused much faster.

Can We Fix It?

The researchers tried a few ways to stop the AI from overthinking:

Telling it to "Be Shorter" (Prompting): They tried telling the AI, "Don't think so much." This did shorten the answer, but it also made the AI dumber. It was a trade-off: shorter thoughts, but wrong answers.
Changing the Training Data: They tried teaching the compressed AI using examples of good math reasoning instead of generic text. This helped a bit, but not enough.
The Best Solution (Re-training): The most effective fix was to re-train the AI specifically while it was compressed. This is like teaching a student to take a test while wearing a blindfold, so they learn to navigate the confusion from the start. This method (called Quantization-Aware Training) managed to keep the AI fast, accurate, and concise all at once.

The Bottom Line

The paper concludes that we can't just measure AI performance by "Did it get the right answer?" anymore. For reasoning models, we also need to measure "How much did it think to get there?"

If we ignore this "thinking cost," we might think we've saved money by compressing our AI, only to find out we're actually paying more in time and computing power because the AI is stuck in a loop of overthinking.

Technical Summary: Quantization Inflates Reasoning

Problem Statement

While Post-Training Quantization (PTQ) is a standard technique for reducing the inference cost of Large Language Models (LLMs) by lowering memory footprint and accelerating per-token decoding, its impact on reasoning models is not fully captured by traditional evaluation metrics. Standard protocols focus on final-answer accuracy and per-token latency. However, reasoning models (e.g., those trained for step-by-step Chain-of-Thought or CoT) rely heavily on the volume of test-time computation.

This paper identifies a hidden cost: low-bit PTQ can cause Reasoning-Token Inflation. Quantized models often preserve final-answer accuracy but generate significantly longer chains of thought compared to their full-precision (BF16) counterparts. This increase in token count can offset, or even exceed, the per-token speedup gained from quantization, leading to worse end-to-end latency and serving costs.

Methodology

The authors conducted a comprehensive evaluation across multiple dimensions:

Models: Evaluated open-source reasoning models ranging from 4B to 30B parameters, including both dense (Qwen3-4B, Phi-4-14B) and Mixture-of-Experts (MoE) architectures (Qwen3-30B).
Benchmarks: Covered mathematical reasoning (AIME25, GSM8K), code generation (LiveCodeBench, MBPP), scientific reasoning (GPQA-Diamond), agentic tool-use (BFCLv4), and general language understanding (HellaSwag, MMLU-Pro, PIQA).
Quantization Methods: Focused on weight-only PTQ methods, including RTN (baseline), GPTQ, AWQ, HQQ, and ParoQuant, applied at INT4 and INT3 bit-widths. BF16 served as the full-precision reference.
Metrics:
- Accuracy: Final-answer correctness.
- CoT Token Inflation Ratio (CTIR): A new metric defined as the relative increase in average reasoning token count between a quantized model and its BF16 reference:
  $\text{CTIR}(M_q; D) = \frac{\bar{\ell}(M_q; D) - \bar{\ell}(M_{org}; D)}{\bar{\ell}(M_{org}; D)} \times 100\%$
- Behavioral Analysis: Measured intermediate step counts and semantic repetitiveness (using sentence embeddings to detect near-duplicate reasoning steps).
- Serving Efficiency: Measured Time-to-First-Token (TTFT) and throughput on vLLM with H100 GPUs.

Key Results

1. Accuracy Preservation Masks Efficiency Loss

INT4 Performance: For many models and benchmarks, INT4 quantization preserves final-answer accuracy (drops are often negligible or small). However, this accuracy stability does not translate to efficiency.
The Hidden Cost: Quantized models frequently generate longer CoT traces. For example, on the BBH benchmark with Qwen3-4B, INT4 quantization increased total generation time from 285s (BF16) to 292s, despite the kernel-level speedup of W4A16 Marlin kernels.
INT3 Sensitivity: As precision drops to INT3, accuracy degradation becomes more pronounced on hard reasoning tasks, but token inflation occurs even when accuracy is preserved.

2. Quantification of Token Inflation (CTIR)

Widespread Inflation: Low-bit PTQ consistently increases reasoning token usage.
- Qwen3-4B (INT3): CTIR reached extreme levels, e.g., +276.1% for AWQ and +292.1% for HQQ.
- Qwen3-30B (INT3): While more resilient, CTIR remained substantial (e.g., +60.1% for HQQ).
Method Variance: Reconstruction-based methods like GPTQ generally produce lower CTIR than activation-aware methods (AWQ) or calibration-free methods (HQQ), but they do not eliminate the effect entirely.

3. Behavioral Changes

Increased Steps: Quantized models take more intermediate steps to reach the same conclusion.
Semantic Repetition: Analysis of reasoning traces reveals increased self-doubt, back-tracking, and semantic repetition (e.g., "let me verify," "again"). These repetitions do not improve correctness but inflate token count.
Agentic Workloads: In multi-turn tool-use scenarios (BFCLv4), quantization increased CoT tokens by ~13–23% without significantly changing the number of tool calls, indicating the latency penalty stems from internal reasoning, not tool invocation frequency.

4. Impact on Serving

Latency: Even modest CTIR increases (e.g., 1.1×) significantly degrade user-perceived latency. A 1.1× CTIR increase raised p90 TTFT by 33% (from 1050s to 1400s in the specific test setup).
Throughput: Longer traces occupy decoding slots and KV-cache memory for longer periods, reducing effective serving capacity under continuous batching.

Mitigation Strategies Evaluated

The paper evaluates three approaches to reduce token inflation:

Inference-Time Controls (Decoding/Prompting):
- Repetition Penalty: Reduces token count but often at a significant cost to accuracy.
- Conciseness Prompting: Prompts like "do not think" or "no restate" reduce tokens but cause inconsistent and often severe accuracy drops (e.g., 5–11 point drops).
- Verdict: These are unstable heuristics that trade accuracy for length.
Calibration Data Optimization:
- Using reasoning-specific corpora (Math-CoT, Code-CoT) for PTQ calibration significantly improves both accuracy and reduces token length compared to generic text (Wikitext, C4).
- Verdict: Effective and low-cost, but still inferior to QAT.
Quantization-Aware Training (QAT):
- Using QAT (specifically DiscQuant) allows the model to adapt weights to low-bit constraints during training.
- Verdict: QAT achieves a Pareto improvement, simultaneously reducing token inflation and improving accuracy compared to all PTQ configurations. It aligns the quantized model's reasoning trajectory with the full-precision target.

Significance and Claims

The paper argues that reasoning-token usage must be reported alongside accuracy when evaluating quantized reasoning models.

Evaluation Gap: Standard quantization studies that report only accuracy and per-token latency are incomplete for reasoning models. They miss the "hidden test-time compute cost" where models reason longer to compensate for quantization noise.
Deployment Reality: For reasoning models, end-to-end latency is a function of both per-token speed and total token count. Quantization can inadvertently increase the latter, negating the former.
Recommendation: The authors suggest that while PTQ is practical, it requires careful monitoring of CoT length. For optimal deployment of reasoning models, Quantization-Aware Training (QAT) is presented as the most promising direction to mitigate both accuracy degradation and token inflation.