💬 NLP

NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs

The paper proposes NeUQI, a method that improves low-bit uniform quantization for large language models by deriving near-optimal initialization parameters through a simplified scale-only optimization, thereby outperforming existing techniques and even resource-intensive distillation approaches.

Original authors: Li Lin, Xinyu Hu, Xiaojun Wan

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Li Lin, Xinyu Hu, Xiaojun Wan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model, or LLM). This library is so big that it requires a giant, expensive warehouse (a high-end server) to store it and a team of heavy-lifters (powerful GPUs) to read the books. Most people, however, just want to carry a few favorite books in their backpack (a laptop or phone) to read on the go.

The problem is that the books are written in a very heavy, complex language (high-precision numbers like bfloat16). To fit them in a backpack, you need to translate them into a simpler, lighter language (low-bit integers like 2-bit or 3-bit). This process is called Quantization.

The Old Way: The "Ruler and Ruler" Method

For a long time, the standard way to translate these books was to use a method called Min-Max.

Think of Min-Max like using a rigid ruler to measure a room. You look at the shortest point in the room and the longest point, and you say, "Okay, our ruler must stretch exactly from the shortest wall to the longest wall."

The Flaw: This approach is too rigid. It forces the ruler to be determined only by the extreme outliers (the very shortest and very longest points). If the room has a weird bump or a tiny dent at the very edge, your ruler gets distorted to accommodate it, leaving the rest of the room poorly measured.
The Constraint: Furthermore, this old method insisted that the "zero point" (where your ruler starts counting) had to be a whole number, like 1, 2, or 3. It couldn't be 2.5. This was like saying you can only measure in whole inches, never half-inches, which limits your accuracy.

The New Way: NeUQI

The authors of this paper, Li Lin and colleagues, realized that this rigid "Min-Max" ruler was holding us back, especially when trying to shrink the library down to a tiny 2-bit or 3-bit size. They proposed a new method called NeUQI (Near-Optimal Uniform Quantization Parameter Initialization).

Here is how NeUQI works, using a creative analogy:

1. The "Flexible Tape Measure" vs. The Rigid Ruler
Instead of letting the two extreme walls dictate the entire ruler, NeUQI looks at the shape of the whole room. It asks: "If I move my starting point (the zero-point) just a tiny bit, does the overall measurement get better?"

The Breakthrough: NeUQI realized that for every specific ruler length (scale), there is a mathematically perfect "zero point" that minimizes errors. It calculates this perfect zero point efficiently, even if that zero point is a decimal number (like 2.53) rather than a whole number.
The Result: By allowing the zero point to be a precise decimal, the "translation" of the library is much more accurate. The books fit better in the backpack without losing their meaning.

2. The "Coarse-to-Fine" Search
Finding this perfect decimal zero point for every single book could take forever. To solve this, NeUQI uses a smart search strategy:

Step 1 (Coarse): It quickly scans the room with a wide net to find the general neighborhood where the best ruler starts.
Step 2 (Fine): It then zooms in on that specific neighborhood to find the exact decimal spot.
This is like finding a lost key in a field. First, you walk the whole field to find the right patch of grass. Then, you get down on your knees and search that specific patch carefully. This saves a massive amount of time.

What Did They Find?

The authors tested NeUQI on famous AI models like LLaMA and Qwen.

Better Performance: In their experiments, models compressed using NeUQI were significantly smarter than those using the old Min-Max method. In some cases, a 2-bit model using NeUQI performed almost as well as a full-size, uncompressed model.
Beating the Heavyweights: They even combined NeUQI with a "lightweight distillation" strategy (a way of teaching the small model by having it mimic the big one). Surprisingly, this simple combination beat a much more complex, resource-heavy method called PV-tuning.
The "Magic" Result: In some scenarios, the NeUQI-compressed model (taking up less memory) actually answered questions better than the original, uncompressed model that took up more memory.

The Bottom Line

The paper argues that the way we start the process of shrinking AI models is just as important as the shrinking technique itself. By fixing the "ruler" (the initialization parameters) and allowing for more flexible, precise starting points, NeUQI lets us pack the world's largest AI brains into much smaller spaces without them losing their intelligence. It's a simple change in how we measure the beginning, but it leads to a massive improvement in the final result.

Technical Summary: NeUQI: Near-Optimal Uniform Quantization Parameter Initialization for Low-Bit LLMs

Problem Statement

Large Language Models (LLMs) face significant deployment challenges on consumer-grade hardware due to high memory consumption and inference latency. Post-training quantization (PTQ), particularly using uniform quantization schemes (e.g., converting weights to int4/3/2), offers a practical solution. However, existing low-bit uniform quantization methods largely rely on the conventional Min-Max formula for initializing quantization parameters (scale $s$ and zero-point $z$ ).

The authors identify two critical limitations inherent in the Min-Max formula that hinder performance in low-bit settings (2-bit and 3-bit):

Extreme Value Dependency: Both $s$ and $z$ are determined solely by the minimum and maximum values of the weight distribution. This constrains the optimization algorithm, forcing search-based methods (like LeanQuant) to enumerate scaling factors associated with these extremes, resulting in a computationally expensive $T^2$ search space rather than a more efficient $T$ search.
Integer Zero-Point Constraint: The Min-Max formulation restricts the zero-point $z$ to be a $k$ -bit unsigned integer. This artificially limits the parameter space, preventing the discovery of optimal non-integer values that could minimize quantization error, particularly in low-bit-width scenarios.

Methodology: NeUQI

The paper proposes NeUQI (Near-Optimal Uniform Quantization Parameter Initialization), a method designed to bypass the constraints of the Min-Max formula by explicitly exploiting the structure of the quantization loss.

1. Loss Formulation and Approximation

The method minimizes the quantization loss for a linear layer, defined as the expected squared error between the original and quantized weights weighted by the proxy Hessian ( $H$ ). To make optimization tractable, NeUQI adopts a diagonal approximation of the Hessian, reducing the loss to a sum of independent terms for each weight element:
$L(s, z) = \sum_i H_{i,i} (Q_{s,z}(w_i) - w_i)^2$

2. Zero-Point Optimization (Fixed Scale)

Instead of jointly optimizing $s$ and $z$ , NeUQI fixes the scale $s$ and derives the optimal zero-point $z$ analytically.

Piecewise Quadratic Structure: The loss function $L(z)$ is identified as a piecewise quadratic function.
Efficient Search: A naive approach to finding the global minimum would be $O(n \cdot 2^k)$ . NeUQI introduces an algorithm (Algorithm 1) that updates the quadratic function incrementally as it traverses transition points, reducing complexity to $O(n \log n)$ after sorting.
Coarse-to-Fine Strategy: To further reduce computational burden, NeUQI approximates the loss function to identify a candidate region for the optimal $z$ in $O(n \log n)$ time. It then performs a precise search within a restricted neighborhood of this candidate, maintaining the $O(n \log n)$ complexity.

3. Scale Optimization

With the ability to compute the optimal $z$ for any given $s$ , the problem reduces to a single-variable optimization over the scale $s$ .

Search Space: The search space is bounded by the scale derived from the Min-Max formula.
Coarse-to-Fine Search: NeUQI employs a two-stage search strategy. First, a coarse search over $T_c$ candidates ( $T_c = O(\sqrt{T})$ ) identifies a promising region. Second, a fine-grained search over a neighborhood of the coarse optimum refines the scale estimate. This reduces the number of zero-point computations from $O(T)$ to $O(\sqrt{T})$ .

4. Relaxing Constraints

Crucially, NeUQI removes the integer constraint on the zero-point, allowing $z$ to take floating-point values. This expands the parameter space, enabling the model to find a "near-optimal" initialization that minimizes quantization error more effectively than integer-constrained methods.

Key Contributions

Identification of Min-Max Limitations: The authors formally identify and analyze the two constraints (extreme-value dependency and integer zero-point) of the standard Min-Max initialization, demonstrating how they limit optimization flexibility and search efficiency.
NeUQI Algorithm: The proposal of a novel initialization method that decouples scale and zero-point optimization. By deriving the optimal zero-point for a fixed scale, the method transforms a complex joint optimization into a tractable single-variable search.
Empirical Superiority: Extensive experiments across LLaMA 2/3 and Qwen 2.5 families show that NeUQI consistently outperforms existing PTQ methods (GPTQ, GPTAQ, MagR, LeanQuant) in low-bit settings (2-bit and 3-bit).
Synergy with Fine-Tuning: The paper demonstrates that NeUQI serves as a powerful initialization for downstream tasks. When combined with lightweight distillation, it outperforms resource-intensive methods like PV-tuning. When used to initialize EfficientQAT, it yields further performance gains, suggesting that a high-quality initialization is critical even for strong fine-tuning pipelines.

Experimental Results

Performance: In 2-bit channel-wise quantization, NeUQI significantly reduces perplexity on C4 and WikiText2 and improves average accuracy across five benchmarks (ARC, HellaSwag, etc.) compared to baselines. For example, on LLaMA 2 7B (2-bit), NeUQI achieves a C4 perplexity of 17.50 compared to 2592 for GPTQ and 246 for GPTAQ.
Comparison with Non-Quantized Models: Under comparable memory usage, NeUQI-initialized 3-bit models often outperform their non-quantized (BF16) counterparts. For instance, a 3-bit Qwen 2.5 72B model using NeUQI achieves lower perplexity and higher accuracy than the 16-bit non-quantized 14B model.
Efficiency: The proposed optimizations (transition-point reduction and coarse-to-fine search) reduce the runtime overhead of initialization. While NeUQI takes longer than GPTQ (e.g., ~30 mins vs ~9 mins for LLaMA 2 7B on 2-bit), it remains practical and significantly faster than methods requiring extensive fine-tuning or heavy search.
Hardware Compatibility: The paper notes that floating-point zero-points are supported by existing GPU inference libraries (e.g., BitBLAS integrated with vLLM), ensuring that the theoretical gains translate to practical deployment without specialized hardware.

Significance and Claims

The paper claims that the initialization of quantization parameters is a critical, often overlooked factor in low-bit quantization. The authors argue that poor initial conditions are difficult to rectify even with extensive fine-tuning, whereas a well-initialized model can achieve superior performance with significantly less effort.

By moving beyond the Min-Max formula, NeUQI provides a "near-optimal" starting point that:

Enables low-bit models (2-bit/3-bit) to approach or exceed the performance of higher-bit or non-quantized models with similar memory footprints.
Offers a lightweight alternative to heavy fine-tuning methods (like PV-tuning) when combined with distillation.
Establishes that relaxing the integer zero-point constraint is a necessary step for maximizing the potential of uniform quantization in LLMs.

The work suggests that future research should focus on fine-tuning strategies that operate beyond the first constraint (the Min-Max formula) to further enhance post-initialization performance.