Technical Summary: Reliability Scaling Laws for Quantized Large Language Models

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable performance on standard benchmarks, their practical deployment faces significant challenges regarding resource constraints and reliability. Model compression techniques, particularly quantization, are widely used to reduce inference latency and memory requirements. However, existing evaluations of quantized LLMs predominantly focus on "benign" task performance (e.g., accuracy and perplexity on unperturbed inputs), often overlooking critical reliability dimensions such as uncertainty quantification, calibration, and robustness to real-world input variations.

Furthermore, prior work on scaling laws (e.g., Dettmers & Zettlemoyer, 2023) has established that accuracy generally scales monotonically with the total number of model bits. It remains unclear whether this monotonicity holds for reliability metrics, or if there exists an optimal precision level that maximizes the trade-off between reliability and efficiency. This gap is critical because models deployed in practice must handle semantically preserving input perturbations (e.g., typos, slang, emojis) and provide trustworthy uncertainty estimates.

Methodology

The authors propose a comprehensive reliability evaluation framework for quantized LLMs, analyzing how reliability scales with the total number of model bits (defined as the product of the number of parameters and weight precision in bits).

1. Reliability Evaluation Framework

The study evaluates quantized models across three key dimensions:

• Uncertainty: Measured using token-level entropy (Shannon entropy) and log-likelihood of ground-truth tokens to assess the model's confidence and trustworthiness.
• Calibration: Assessed using the Brier Score to determine how well the predicted probability distributions align with actual outcomes (i.e., whether low uncertainty correlates with high accuracy).
• Robustness: Evaluated by introducing 15 natural, semantically-preserving perturbations to input prompts. These include:
  • Character-level: Emoji insertion, character replacement (adjacent keyboard keys), leetspeak, noise insertion, deletion, swapping, repetition, and case changes.
  • Word-level: Slang insertion, word deletion, word swapping, word repetition, word insertion, and translation to other languages without back-translation.
  • Perturbations are applied at varying intensity levels (4 and 16) to simulate realistic digital communication noise.

2. Experimental Setup

• Models: The study covers four base model families (LLaMA-3.2-1B/3B, LLaMA-3-8B/70B, OPT, and Qwen3) ranging from 1 billion to 70 billion parameters.
• Quantization: Six state-of-the-art Post-Training Quantization (PTQ) methods are evaluated: BitsandBytes (BNB), AWQ, GPTQ, HQQ, Quanto, and AQLM (including the AQLM-PV variant for 2-bit).
• Benchmarks: Evaluation is conducted on TriviaQA, CoQA, CommonsenseQA, and additional benchmarks for language modeling (WikiText-2, PTB, C4) and reasoning (MMLU, HellaSwag, etc.).
• Scaling Analysis: The authors fit log-quadratic functions to characterize the relationship between total model bits and performance/reliability metrics, moving beyond individual data points to identify underlying trends.

Key Contributions

1. Reliability Evaluation Framework: Introduction of a framework that integrates uncertainty, calibration, and robustness to semantically-preserving perturbations for both full-precision and quantized LLMs.
2. Bit-Level Scaling Laws for Reliability: The first comprehensive characterization of how reliability scales with total model bits across different architectures, datasets, and quantization methods.
3. Identification of the 4-Bit "Sweet Spot": Empirical evidence that 4-bit quantization offers the optimal trade-off between reliability and efficiency, outperforming both higher precision (8/16-bit) and extreme low-bit (2/3-bit) regimes in terms of reliability metrics.
4. Robustness Enhancement: Demonstration that quantization can enhance model robustness to natural input perturbations compared to full-precision baselines.
5. Comparative Analysis: Extension of scaling analysis to pruning strategies (SparseGPT, Wanda) and Quantization-Aware Training (EfficientQAT), providing a broader context for compression techniques.

Key Results

1. Non-Monotonic Reliability Scaling

Contrary to the monotonic scaling of accuracy, reiability follows a non-monotonic trend:

• Accuracy: Scales monotonically with the total number of model bits.
• Reliability: Peaks at 4-bit quantization.
  • Models quantized to 4 bits achieve the best reliability-efficiency trade-off.
  • Extreme quantization (2-bit) leads to significant behavioral shifts (measured via KL-Divergence) and poor reliability, despite some performance recovery in larger models.
  • Higher precision models (8-bit, 16-bit) often exhibit worse uncertainty estimates and calibration than their 4-bit counterparts when matched for total bit budget, likely due to lower parameter counts.

2. Robustness to Perturbations

• Quantized models, particularly at 4-bit precision, demonstrate enhanced robustness to natural input perturbations (e.g., typos, slang, emojis) compared to their full-precision counterparts.
• The AUCROC (Entropy) scores, a measure of reliability under perturbation, are often higher for 4-bit models than for 16-bit models with equivalent total bit budgets.

3. Behavioral Shift (KL-Divergence)

• The study measures the Kullback-Leibler (KL) Divergence between the token distributions of base models and compressed models.
• 2-bit models exhibit a substantial behavioral shift, contributing to poor performance.
• 3-bit models show a noticeable shift that diminishes as model size increases.
• 4-bit models maintain a small KL divergence across model scales, suggesting they retain generalization capabilities while achieving significant compression.

4. Consistency Across Architectures and Methods

• The "4-bit sweet spot" is consistent across different model families (LLaMA, OPT, Qwen) and quantization methods (HQQ, GPTQ, AWQ, etc.).
• While specific methods perform better at specific bitwidths (e.g., HQQ for 3-bit, AWQ/GPTQ for 4-bit), the overarching trend of 4-bit optimality holds.
• Pruning vs. Quantization: Pruning (SparseGPT, Wanda) does not yield the same reliability gains as quantization; reliability degrades as sparsity increases, and quantization-induced error is generally lower than sparsification-induced error.

Significance and Claims

The paper claims that reliability does not necessarily scale monotonically with model size or precision. The authors argue that relying solely on accuracy metrics is insufficient for safe deployment. Instead, they posit that:

• 4-bit quantization represents an optimal regime where models are sufficiently close to their full-precision counterparts (low behavioral shift) while benefiting from increased parameter counts within a fixed bit budget.
• Quantization is not merely a tool for efficiency but can enhance model reliability and robustness against real-world input noise.
• This finding challenges the assumption that higher precision or larger models are always superior for trustworthy AI, suggesting that moderate quantization is a viable strategy for deploying reliable LLMs in resource-constrained environments.

The study concludes that future work should explore how calibration, multi-shot prompting, and Quantization-Aware Training (QAT) further influence these scaling trends, particularly for extreme low-bit regimes.