🤖 machine learning

Flat Score, Amplified Failures: How the Error Budget Masks Damage in Quantized LLM Agents

This paper reveals that while standard benchmarks suggest 4-bit quantization is lossless for multi-turn LLM agents, it actually amplifies existing failure modes by up to 2.5 $\times$ within a fixed error budget, a hidden degradation that only becomes visible through per-channel error analysis and stricter success criteria.

Original authors: Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

Published 2026-07-31

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiwon Jang, Kisu Yang, Heuiseok Lim, Hyunwoo Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a robot butler to help you with chores. You want it to be fast and not eat up all your computer's memory, so you decide to shrink its brain. You take its complex instructions and compress them down, like squashing a giant, fluffy cloud into a tiny, dense marble. In the world of artificial intelligence, this is called "quantization." Scientists have spent years testing this, and they've found that if you just ask the robot a simple question once—like "What's the weather?"—it seems to work perfectly fine even after being squashed. It's like saying the robot is "nearly lossless," meaning it lost almost nothing important.

But here's the twist: real life isn't just one simple question. It's a long conversation where the robot has to open doors, check the fridge, call a plumber, and fix mistakes along the way. This is called being an "agent." The big question scientists wanted to answer was: Does that "nearly lossless" claim still hold true when the robot is doing a complex, multi-step job? If the robot makes a mistake, can it fix it, or does the whole mission crash? This paper dives deep into that exact scenario, looking at whether squashing the robot's brain hides a ticking time bomb that only shows up when things get complicated.

The Flat Score and the Hidden Explosion

The researchers set up a massive experiment using a simulated world where AI agents act as customer service helpers. They tested these agents in two different "neighborhoods": a Retail store (where the agent just talks to a database) and a Telecom company (where the agent has to talk to a simulated user who is also fiddling with their own phone). They took several different AI models and ran them at full precision (the fluffy cloud) and then at 4-bit precision (the tiny marble).

The result? On the standard report card—the "Final Score"—everything looked perfect. The squashed models scored almost exactly the same as the full-size models. In fact, in most cases, the difference was so small it was statistically invisible. It looked like the compression was free! The authors call this a "flat score."

But then, they started looking under the hood, at the actual steps the robot took. That's when they found the explosion.

The Amplifier Effect

While the final grade stayed the same, the squashed models were actually making way more mistakes along the way. In the Telecom neighborhood, the 4-bit models started hallucinating (making things up) about 2.5 times more often than the full-size models.

Here is the most surprising part: The squashed models didn't invent new mistakes. They didn't start calling tools they never knew before. Instead, they just took the same mistakes the full-size model made occasionally and turned up the volume on them.

Think of it like a radio. If a full-size model occasionally picks up a bit of static (a mistake), the 4-bit model doesn't create new static; it just cranks the volume knob until that same static is deafening. In one specific case, the model went from making 649 "hallucinated tool calls" (calling tools that don't exist) to 1,646 calls. It was the exact same list of wrong tools, just shouted at much higher frequency.

The Safety Net That Hides the Danger

So, if the robots were making 2.5 times more mistakes, why did their final scores stay the same?

The answer lies in the rules of the game. The benchmark they used allows the robot to fail up to 10 times in a single episode before it counts as a total failure. It's like a video game where you have 10 lives. The full-size robot might miss a jump once or twice, but it recovers. The squashed robot, however, misses the jump 2.5 times as often. But because it still has 10 lives, it keeps recovering, keeps trying, and eventually finishes the level.

The "Error Budget" (those 10 lives) acts like a giant safety net that absorbs all the extra falls. The final score stays flat not because the robot is unharmed, but because the game is too forgiving. The damage is real, but it's hidden in the noise of the retries.

The "Tighten the Belt" Test

To prove this, the researchers played a clever trick. They took the safety net and shrank it. Instead of allowing 10 mistakes, they only allowed 2.

Suddenly, the mask fell off. When the budget was tight, the squashed models crashed hard. The gap between the full-size model and the 4-bit model exploded from a tiny 1.3 points to a massive 16.7 points. The squashed model failed because it didn't have enough "lives" left to recover from all its extra mistakes.

This test confirmed two things:

The damage was real and hidden by the lenient rules.
The damage only happened where the model was already prone to making mistakes. Some models (like the Qwen-3.6) were so good at their job that they barely made any mistakes to begin with, so squashing them didn't change anything. But for models that already had a "tendency" to hallucinate, squashing them made that tendency much worse.

The Fix: Stop the Leak, Don't Patch the Hole

The researchers also tried to fix the problem. Instead of trying to make the whole robot smarter, they gave it a simple rule: "If you try to call a tool that isn't on the list, stop and ask for a new one."

When they added this simple "reflexive repair," the squashed models in the damaged area saw their scores recover significantly, rising from a low of 65.4% back up to 71.3%. This was a major improvement over the damaged 4-bit version and even slightly higher than the original full-size model's score of 66.7% in that specific scenario. However, this fix only worked where the specific "leak" existed; in models that weren't prone to these specific mistakes, the repair didn't help and sometimes even slightly lowered the score. This proved that the problem wasn't that the robot's brain was broken; it was just that it kept reaching for the same few wrong tools over and over again.

The Takeaway

The big lesson here is that a good final score doesn't mean a safe robot. If you are building an AI agent that has to do complex tasks, you can't just look at the final grade. You have to look at the process.

If you compress an AI model to save space, you might not be creating new problems, but you could be turning up the volume on the ones that already exist. And if your system has a big safety net (like allowing many retries), you might not even notice the volume is turned up until the net gets too small to catch the falls. The paper suggests that before we start squashing AI brains for real-world use, we need to check if the model is already prone to making mistakes, because if it is, the compression will make those mistakes much louder.

Technical Summary: Flat Score, Amplified Failures

Problem Statement
Post-training quantization to 4-bit weights is widely accepted as "nearly lossless" for Large Language Models (LLMs), with systematic evaluations showing negligible degradation in standard single-turn benchmark accuracy. However, deployed LLMs increasingly function as multi-turn, tool-calling agents that interact with environments, observe feedback, and recover from mistakes. The paper investigates whether the "nearly lossless" verdict holds in this agentic setting. The authors hypothesize that standard end-of-episode task rewards may mask underlying reliability degradation because the agentic loop and benchmark error budgets allow for error recovery, potentially hiding the amplification of per-step failures.

Methodology
The study evaluates quantized agents on $\tau^2$ -bench, a tool-calling benchmark featuring simulated users and stateful environments across two domains: Retail (single-control) and Telecom (dual-control).

Models: The authors test two open-weight model families (Gemma-4 and Qwen-3.6/3.5) in both dense and Mixture-of-Experts (MoE) variants.
Precision Arms: Models are evaluated at three precision levels: BF16 (full precision), FP8, and INT4 (quantized via AWQ).
Experimental Scale: The study covers a full $2 \times 2 \times 2$ grid (families $\times$ architectures $\times$ domains) plus extensions, totaling eight primary cells with 456 simulated episodes each.
Metrics:
- Outcome Metric: Final task success rate (binary pass/fail).
- Process Metrics: Per-channel error rates (tool-name hallucinations in Telecom; entity/argument errors in Retail), trajectory length, and failure recovery rates.
- Counterfactual Analysis: The error budget ( $K$ , the maximum allowed failed tool calls per episode) is treated as a variable. The authors recompute success rates for tighter budgets ( $K=2, 5$ ) to test if the standard $K=10$ budget masks damage.
- Mechanism Analysis: Correlation of hallucinated tool names between BF16 and INT4, and logit-margin analysis to explain failure amplification.

Key Findings

The "Flat Score" Illusion:
On the standard benchmark metric (final task score with $K=10$ ), quantization appears lossless. Across all ten experimental cells, no score change survives multiple-comparison correction. In the cell with the largest observed damage, the score change is statistically equivalent to zero within a $\pm 7.5$ point margin. This suggests that standard evaluations provide a false "all-clear."
Amplification of Existing Failures:
While the final score remains flat, the process metrics reveal significant degradation. Quantization does not create new types of failures; instead, it amplifies the specific failure modes the model already exhibits at full precision.
- Volume Increase: In the most affected cell (Gemma-4 dense in Telecom), 4-bit quantization increases tool-name hallucinations by 2.5 $\times$ (from 19.5% to 38.3% of tool calls).
- Invariance of Failure Set: The set of hallucinated tool names is nearly identical between BF16 and INT4 (rank correlation $\ge 0.97$ ). Only 0.18% of INT4 hallucinations involve tools never invented by the BF16 model. Quantization makes the model reach for its "existing temptations" more frequently.
- Channel Specificity: The amplification is domain-dependent. Telecom (dual-control) amplifies tool-name hallucinations, while Retail (single-control) shows trends in entity/argument errors. Models with near-zero baseline error rates (e.g., Qwen-3.6) show no amplification.
The Error Budget as a Mask:
The final score remains flat because the benchmark's generous error budget ( $K=10$ ) absorbs the extra failures. The agent successfully recovers from individual errors at the same rate (55%) regardless of precision; the extra failures simply consume more of the budget.
- Revealing the Gap: When the error budget is tightened to $K=2$ , the hidden score gap re-emerges. In the damaged Gemma-4 Telecom cell, the gap widens from 1.3 points (at $K=10$ ) to 16.7 points (at $K=2$ ), a roughly 13 $\times$ increase.
- Falsifiability: This effect is localized. Cells without baseline failure amplification (e.g., Qwen-3.6) show no gap widening under tighter budgets, confirming the damage is specific to the error volume increase.
Mechanism: Logit Margins:
The paper proposes a logit-margin account to explain the phenomenon. Quantization introduces noise that flips "thin-margin" decisions (where the correct token and the strongest competitor are close in logit space). Since models only occasionally make these specific errors at full precision, quantization amplifies this specific subset of failures rather than generating new ones. Susceptibility is determined by the model's full-precision propensity to make thin-margin errors, not by model size or architecture (Dense vs. MoE).

Key Contributions

Demonstration of Masking: The authors show that end-of-episode task rewards are structurally blind to quantization damage in agentic settings because the environment's error budget and the agent's recovery loop hide per-step reliability degradation.
Characterization of Damage: The paper establishes that quantization acts as a multiplier on existing failure sets (precision-invariant) rather than a generator of new pathologies.
Diagnostic Framework: The authors propose two diagnostics derived from existing logs:
1. Per-channel error rates to detect amplification.
2. Success under shrinking error budgets to unmask the damage.
Targeted Mitigation: A reflexive error-repair prompt (diagnosing and re-issuing calls) successfully collapses the error channel in damaged cells, recovering the score, while having no effect (or mild negative effects) in cells where no damage exists.

Significance and Claims
The paper claims that the consensus of "nearly lossless" 4-bit quantization is an artifact of evaluation methodology, not a reflection of model robustness in closed-loop agentic workflows.

Risk Screening: Full-precision propensity (measuring per-channel error rates before quantization) serves as a cheap, effective risk screen. If a model rarely hallucinates at 16-bit, it is likely safe at 4-bit; if it has a standing failure tendency, that tendency will be magnified.
Deployment Implications: The masking effect relies on "forgiving" environments with high error tolerances. In real-world deployments with strict error limits or irreversible actions, the amplified per-step failures will translate directly into outcome failures.
Reporting Standards: The authors argue that benchmarks must report process-level metrics (per-channel error rates and success as a function of error budget) alongside task rewards to provide an accurate assessment of quantized agents.

The paper concludes that "lossless on the benchmark" does not equate to "lossless where deployed," and that the agentic loop serves as a sensitive probe for perturbations that single-turn accuracy metrics miss.