Technical Summary: "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization

Problem Statement

The high computational cost of serving Large Language Models (LLMs) has driven the adoption of quantization to accelerate inference. However, the practical trade-offs between accuracy and performance across different quantization formats (FP8, INT8, INT4) remain unclear. This uncertainty has fueled skepticism regarding quantized models, such as the initial doubts surrounding the Llama-3.1-405B quantized release. Existing literature often relies on academic benchmarks that may not reflect real-world deployment scenarios, and some studies report significant accuracy degradation (e.g., >10% drops) for integer-based quantization, leading to a lack of systematic guidelines for practitioners.

Methodology

The authors conducted a comprehensive empirical study involving over 500,000 evaluations across the entire Llama-3.1 model family (8B, 70B, and 405B parameters). The study focused on three primary quantization formats supported by the vLLM framework:

1. W8A8-FP: 8-bit floating-point weights and activations.
2. W8A8-INT: 8-bit integer weights and activations.
3. W4A16-INT: 4-bit integer weights with 16-bit activations.

Evaluation Framework:

• Accuracy: Evaluated across academic benchmarks (Open LLM Leaderboard V1 and V2), real-world generative tasks (Arena-Hard-Auto-v0.1, HumanEval, HumanEval+), and long-context capabilities (RULER). Additionally, text similarity analysis (ROUGE, BERTScore, STS) was performed to assess generative consistency between quantized and full-precision (BF16) models.
• Algorithms: The study compared specific quantization algorithms, including GPTQ, AWQ, SmoothQuant, and dynamic per-token activation quantization. Notably, the authors optimized hyperparameters (e.g., using MSE-optimal clipping for GPTQ and high-quality calibration data) to ensure fair comparisons.
• Performance: Inference performance was benchmarked using vLLM (v0.6.4.post1) across three GPU architectures (A6000, A100, H100) in seven deployment scenarios. The study analyzed both synchronous (latency-sensitive) and asynchronous (throughput-driven) deployments, calculating latency, throughput (QPS), and cost-efficiency (queries per USD).

Key Contributions and Findings

1. Accuracy Performance

• FP8 (W8A8-FP) is Effectively Lossless: Using dynamic per-token activation quantization combined with symmetric weight quantization, FP8 quantization preserves the uncompressed model's accuracy across all scales and benchmarks, often falling within the margin of error of the evaluation.
• INT8 (W8A8-INT) is Highly Competitive: Contrary to prior claims of >10% accuracy drops, well-tuned INT8 quantization (using dynamic activation quantization or SmoothQuant paired with GPTQ) achieves only a modest 1–3% accuracy degradation on average.
• INT4 (W4A16-INT) Rivals 8-bit: Weight-only INT4 quantization performs on par with W8A8-INT. The study demonstrates that a simple variant of GPTQ with MSE-optimal clipping outperforms the more recent AWQ method on real-world tasks, challenging previous assumptions that AWQ is superior for low-bit strategies.
• Text Similarity: Larger quantized models (70B, 405B) closely adhere to the word choices and sentence structures of their full-precision counterparts. Smaller models (8B) show moderate structural variability but preserve semantic meaning.

2. Deployment and Performance Trade-offs

• Synchronous Deployments (Latency-Sensitive): W4A16-INT is the most cost-efficient choice. It reduces cost per query by 2–3× for smaller models and 5–7× for the 405B model compared to BF16, while significantly improving latency. It enables the 405B model to run on fewer GPUs (e.g., 4× A100/H100 vs. 16×).
• Asynchronous Deployments (Throughput-Driven): W8A8 formats (both FP and INT) generally maximize throughput in continuous batching scenarios. While W4A16-INT remains competitive, W8A8 dominates in high-throughput settings where latency constraints are relaxed.
• Hardware Dependence: The optimal choice depends on the specific hardware (A6000, A100, H100), model size, and the ratio of prefill to decode tokens.

Significance

This work provides the first in-depth, data-driven study of the accuracy, performance, and cost trade-offs for quantized LLMs across diverse formats, algorithms, use cases, and hardware types. By systematically debunking myths regarding severe accuracy loss in integer quantization and providing clear deployment guidelines, the paper aims to serve as a practical guide for deploying quantized LLMs at scale. The findings suggest that with judicious algorithm selection and parameter tuning, practitioners can achieve a balance of speed, efficiency, and accuracy that was previously thought unattainable, moving beyond the binary choice of "BF16 or Death."

Limitations

The authors acknowledge that their study primarily focuses on weight and activation quantization, leaving the impact of compressing other components (such as KV-cache, input embeddings, and the language modeling head) open for future investigation. Additionally, the analysis does not fully explore specialized use cases like multi-lingual tasks, where accuracy degradation patterns may vary based on language distribution.