💬 NLP

Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs

Mix-Quant is a phase-aware quantization framework that accelerates agentic LLM inference by applying high-throughput NVFP4 quantization to the compute-intensive prefilling stage while maintaining BF16 precision for decoding, thereby achieving up to a 3x speedup without significant performance degradation.

Original authors: Haiquan Lu, Zigeng Chen, Gongfan Fang, Xinyin Ma, Xinchao Wang

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Haiquan Lu, Zigeng Chen, Gongfan Fang, Xinyin Ma, Xinchao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart assistant (an AI agent) that helps you solve complex problems. To do its job, this assistant doesn't just chat; it reads huge manuals, searches the web, remembers past conversations, and uses tools like calculators or code editors.

The paper "Mix-Quant" tackles a specific problem: This assistant is getting bogged down by the sheer amount of reading it has to do before it can start speaking.

Here is the breakdown using simple analogies:

1. The Problem: The "Reading vs. Writing" Imbalance

Think of the AI's work as a two-step process:

Step A (Prefilling): The "Reading Phase." The AI reads a massive stack of documents, instructions, and history all at once to understand the context. This is like a student reading a 500-page textbook before an exam. It requires a lot of brainpower (computation) but happens all at once.
Step B (Decoding): The "Writing Phase." The AI generates the answer, one word at a time. This is like the student writing the essay. It happens slowly, one word after another, and relies heavily on memory.

The Bottleneck: In "Agentic" tasks (where the AI uses tools and remembers things), the "Reading Phase" (Step A) is often hundreds of times longer than the "Writing Phase" (Step B). The AI spends most of its time just reading the prompt, making this the slow part of the process.

2. The Failed Solution: "The Speed-Reading Glasses"

Researchers tried to make the AI faster by putting "speed-reading glasses" on it (a technique called Quantization). This involves simplifying the numbers the AI uses to think, turning high-precision math into low-precision math.

The Uniform Approach: They tried putting these glasses on the AI for both reading and writing.
The Result: While the AI got faster at reading, it started making silly mistakes while writing. Because the AI is writing one word at a time, a tiny mistake in the first word can snowball into a completely wrong answer by the end. It was like a student who read the book quickly but forgot the details, leading to a bad essay.

3. The New Solution: "Mix-Quant" (The Hybrid Strategy)

The authors realized that the two phases have different needs. They proposed Mix-Quant, which treats the two phases differently:

For the "Reading Phase" (Prefilling): They use the Speed-Reading Glasses (NVFP4).
- Why? The AI is just processing a fixed block of text. Even if the math is slightly simplified, the errors don't "snowball" because the text isn't changing. The AI can read the massive context much faster without losing much accuracy.
For the "Writing Phase" (Decoding): They take the glasses off and keep the High-Precision Vision (BF16).
- Why? When the AI is generating words one by one, it needs to be precise. A tiny error here changes the next word, which changes the next, and so on. Keeping this phase precise ensures the final answer is correct and stable.

4. The Analogy: The Construction Crew

Imagine a construction crew building a house:

The "Reading" (Prefilling) is the team surveying the land and reading the blueprints. This is heavy lifting. Mix-Quant says, "Let's use a heavy-duty, fast-moving crane (NVFP4) to move all the blueprints and materials quickly. It doesn't matter if the crane is slightly less precise, as long as the materials get there fast."
The "Writing" (Decoding) is the team actually laying the bricks. Mix-Quant says, "Now, switch to the master mason with steady hands (BF16). We need perfect precision here. If a brick is slightly off, the whole wall could fall."

5. The Results

By mixing these two approaches, the paper claims:

Speed: The "Reading" phase became 2 to 3 times faster.
Accuracy: The AI still performed almost as well as the original, slow, high-precision version. It didn't lose its "smarts" or start making bad decisions.
Efficiency: It solved the bottleneck of long, complex tasks without breaking the AI's ability to think clearly.

In short: Mix-Quant is a smart way to speed up AI agents by letting them "speed-read" the massive context they need to understand, while forcing them to "slow down and be precise" when they actually generate the answer. This keeps them fast without making them dumb.

Technical Summary: Mix-Quant

1. Problem Statement

Large Language Model (LLM) agents have emerged as a paradigm for solving complex tasks through planning, tool use, and multi-step interactions. However, these agentic workflows introduce a unique efficiency bottleneck distinct from standard single-turn generation.

Input-Heavy Workloads: Agentic workflows involve repeated inference steps where the input context (comprising instructions, tool schemas, retrieved evidence, and execution traces) grows rapidly, often becoming tens to hundreds of times larger than the generated output.
The Prefilling Bottleneck: Unlike standard generation where decoding is the dominant cost, agentic inference is dominated by the prefilling stage (processing the long input context). This stage is compute-intensive due to large-scale matrix multiplications required to encode the context and build the Key-Value (KV) cache.
The Quantization Trade-off: Existing model-efficiency strategies, such as uniform low-bit quantization (e.g., weight-and-activation FP4), fail to address this specific bottleneck effectively.
- Weight-only quantization reduces memory footprint but offers limited acceleration for the compute-bound prefill phase because activations remain in high precision.
- Uniform aggressive quantization (applying low bits to both prefill and decoding) accelerates computation but causes significant performance degradation. In agentic workflows, quantization errors during autoregressive decoding can propagate and accumulate over long trajectories, leading to invalid tool calls, incorrect code edits, or divergent reasoning paths.

The core challenge is to accelerate the compute-intensive prefilling phase without introducing the error accumulation that destabilizes the subsequent decoding process.

2. Methodology: Mix-Quant

The authors propose Mix-Quant, a phase-aware quantization framework that decouples the optimization strategies for the prefilling and decoding stages.

Key Design Principles

Phase-Aware Quantization: The framework recognizes that prefilling and decoding have distinct computational profiles and error sensitivities.
- Prefilling: Processes a fixed input context in parallel. Quantization errors here are representation-level perturbations that do not recursively affect future inputs within the same pass. Furthermore, long contexts often contain substantial redundancy, with attention mass concentrated on a small subset of tokens, making this stage robust to aggressive quantization.
- Decoding: A sequential, autoregressive process where each token choice conditions future predictions. Errors here can trigger a "snowball effect," diverging the generation from the correct trajectory.
Hybrid Precision Strategy:
- Prefilling (NVFP4): The compute-intensive context encoding phase utilizes NVFP4 (NVIDIA Blackwell's microscaling FP4 format) for both weights and activations. NVFP4 employs fine-grained local scaling (groups of 16 elements) and a tensor-level scale to maintain numerical accuracy at ultra-low bit-widths. The implementation uses simple round-to-nearest (RTN) quantization, which is sufficient given the format's design.
- Decoding (BF16): The autoregressive token-by-token generation phase retains BF16 (Brain Floating Point 16) precision. This preserves the stability of the generation process and prevents the accumulation of quantization errors over long agentic trajectories.
System-Level Deployment:
- Mix-Quant leverages a prefill-decode disaggregation architecture.
- Prefill Workers: Process the input prompt using the NVFP4 W4A4 (Weight 4-bit, Activation 4-bit) path to generate the initial KV cache.
- Decode Workers: Receive the KV cache via a NIXL-based transfer mechanism and perform autoregressive generation using the original high-precision (BF16) path.
- This design avoids the overhead of low-level kernel switching and KV-cache misalignment that typically plagues mixed-precision pipelines.

3. Key Contributions

The paper outlines three primary contributions:

Identification of the Agentic Bottleneck: The authors reveal that LLM agentic workflows are inherently input-heavy, making the compute-intensive prefilling phase the primary efficiency bottleneck. They demonstrate that naive, uniform efficiency methods (like full low-bit quantization) compromise task performance, necessitating phase-aware optimization.
Proposal of Mix-Quant: A novel framework that applies high-throughput NVFP4 quantization exclusively to the prefilling phase while retaining BF16 precision for decoding. This approach improves inference efficiency without introducing severe error accumulation during generation.
Empirical Validation: Extensive experiments demonstrate that Mix-Quant largely preserves agentic task performance across diverse long-context and multi-turn benchmarks while achieving significant speedups (up to 3×) in the prefilling stage compared to BF16 baselines.

4. Experimental Results

The authors evaluated Mix-Quant on a comprehensive suite of benchmarks using state-of-the-art models (Qwen3, Qwen3.5, Gemma-4) on NVIDIA RTX 5090 and B200 GPUs.

Agentic Benchmarks (BFCL v4, LongMemEval, $\tau^2$ -bench):
- Uniform NVFP4 quantization consistently degraded performance (e.g., Qwen3-8B average score dropped from 42.85 to 38.64).
- Mix-Quant recovered a large portion of this loss, achieving scores nearly matching the BF16 baseline (e.g., 41.45 vs. 42.85 for Qwen3-8B).
- On LongMemEval, Mix-Quant significantly outperformed uniform quantization (e.g., improving Gemma-4-26B-A4B-it from 62.42 to 72.45).
Reasoning and Long-Context Benchmarks (Math500, AIME, LongBench-V2):
- Similar trends were observed where uniform quantization hurt reasoning capabilities, while Mix-Quant maintained high accuracy (e.g., Gemma-4-31B-it Mix-Quant achieved 81.39 vs. 82.36 for BF16, compared to 78.26 for uniform NVFP4).
Performance Speedup:
- Mix-Quant achieved a 2–3× speedup in prefilling latency across varying sequence lengths and batch sizes on NVIDIA RTX 5090 GPUs compared to the BF16 baseline.
- Ablation studies confirmed that quantizing only the prefill stage (Mix-Quant) is superior to quantizing only the decoding stage or quantizing both uniformly.

5. Significance and Claims

The paper claims that Mix-Quant offers a favorable efficiency-performance trade-off for input-heavy LLM agentic inference. By decoupling prefilling acceleration from decoding precision, the method addresses the specific computational bottleneck of long-context agents without sacrificing the stability required for complex, multi-step reasoning.

The authors emphasize that this work demonstrates the potential of phase-aware algorithm-hardware co-design. Rather than applying a single compression strategy uniformly, tailoring quantization to the specific characteristics of each inference phase (aggressive low-bit for compute-bound prefill, high-precision for error-sensitive decoding) is critical for building efficient and reliable LLM agents. The results suggest that such targeted optimization is essential for scaling agentic workflows in real-world, long-context scenarios.