🤖 machine learning

BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models

The paper proposes Bit-Plane Decomposition Quantization (BPDQ), a novel method that utilizes variable quantization grids and second-order optimization to significantly improve the accuracy of large language models at ultra-low bit-widths (2-3 bits), enabling the efficient deployment of a 72B model on a single consumer GPU.

Original authors: Junyu Chen, Jungang Li, Jing Xiong, Wenjie Wang, Qingyao Yang, He Xiao, Zhen Li, Taiqiang Wu, Mengzhao Chen, Zhen Peng, Chaofan Tao, Long Shi, Hongxia Yang, Ngai Wong

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junyu Chen, Jungang Li, Jing Xiong, Wenjie Wang, Qingyao Yang, He Xiao, Zhen Li, Taiqiang Wu, Mengzhao Chen, Zhen Peng, Chaofan Tao, Long Shi, Hongxia Yang, Ngai Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model) that you want to carry in your pocket. The problem is, the library is so heavy and bulky that it won't fit in your backpack, and your phone can't read the books fast enough.

To solve this, scientists use quantization. Think of this as translating the library's complex, high-definition books into a simplified, low-resolution version that takes up less space. Usually, they try to shrink the books down to 4-bit (like turning a 4K movie into a standard DVD). This works well. But when they try to shrink them even further to 2-bit (like turning that movie into a tiny, grainy GIF), the story falls apart. The meaning is lost, and the library becomes useless.

The Problem: The "Cookie Cutter" Trap

The paper explains that existing methods for shrinking these models to 2-bit use a fixed grid.

Imagine you are trying to pack a bunch of oddly shaped rocks (the model's data) into a box.

Old Method (Fixed Grid): You have a rigid cookie cutter. No matter what shape the rock is, you force it to fit into one of four pre-cut slots: 0, 1, 2, or 3. If a rock needs to be "2.5" to fit perfectly, the old method forces it to be "2" or "3," creating a gap or a crack. Because the "cookie cutter" shape is the same for every group of rocks, the model loses too much detail when the bits get this small.

The Solution: BPDQ (The "Custom Mold" Approach)

The authors propose a new method called Bit-Plane Decomposition Quantization (BPDQ).

Instead of using a single, rigid cookie cutter for everyone, BPDQ builds a custom mold for every single group of rocks.

How it works: It breaks the data down into "bit-planes" (like layers of a cake) and uses flexible coefficients (adjustable knobs) to shape the mold.
The Result: Instead of being forced into the rigid slots of 0, 1, 2, or 3, the data can now fit into a flexible set of values like 0, 1.2, 3.5, or 4.1, depending on what that specific group of rocks needs.

The paper calls this a "variable grid." It breaks the rule that all groups must look like a scaled-up copy of the same template. This gives the model much more freedom to find the perfect fit, minimizing the "cracks" (errors) in the data.

The "Second-Order" Magic

To make sure these custom molds are perfect, the method uses something called Hessian-induced geometry.

The Analogy: Imagine you are trying to balance a stack of plates. A simple method just looks at the plates and guesses where to put them. BPDQ, however, uses a "smart balance" that understands the weight and wobble of the entire stack. It doesn't just fix one plate; it adjusts the whole stack to ensure that fixing one part doesn't make another part fall over. This "second-order" information allows the model to correct its own mistakes as it shrinks the data.

The Results: Fitting the Giant on a Small Phone

The authors tested this on a massive model called Qwen2.5-72B (which usually requires a supercomputer to run).

The Feat: Using BPDQ, they were able to shrink this giant model down to 2-bit and run it on a single consumer graphics card (an RTX 3090, which you might find in a high-end gaming PC).
The Performance: Even at this extreme compression, the model kept 83.85% of its original intelligence on math problems (GSM8K).
Comparison: Other methods at 2-bit failed miserably, dropping to near 0% accuracy on those same math problems. It was like trying to read a book written in a language you don't know, whereas BPDQ kept the language readable.

Why It Matters

The paper claims that the main reason previous methods failed at 2-bit wasn't that the math was wrong, but that the "cookie cutter" (the fixed grid) was too rigid. By switching to a variable grid that adapts to the data, BPDQ unlocks the ability to run huge, smart AI models on much smaller, cheaper hardware without losing the "brainpower" needed for complex tasks like math or reasoning.

In short: BPDQ stops forcing square pegs into round holes by making the holes flexible, allowing giant AI models to fit into tiny spaces.

Technical Summary: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models

Problem Statement

Large language model (LLM) inference is frequently constrained by memory footprint and bandwidth, particularly in resource-limited environments. While Post-Training Quantization (PTQ) effectively maintains high fidelity at 4-bit precision, performance deteriorates significantly when pushing to 2–3 bits. Existing optimization-based PTQ methods, such as GPTQ, rely on minimizing output discrepancy (e.g., $\|WX - \hat{W}X\|_F$ ) under a Hessian-induced geometry. However, these methods enforce a shape-invariant quantization grid (e.g., fixed uniform intervals like UINT2 or fixed non-uniform templates) for each weight group.

The authors identify that this shape invariance is the fundamental bottleneck in low-bit regimes. A fixed grid restricts the feasible set of quantization levels to a scaled version of a single template (e.g., $s \cdot \{0, 1, 2, 3\}$ ). This rigidity prevents the optimizer from finding the nearest-point projection in the Hessian-induced geometry when the weight distribution does not align with the fixed template's relative spacing, leading to severe representational loss and model collapse at 2-bit precision.

Methodology: Bit-Plane Decomposition Quantization (BPDQ)

To address the rigidity of fixed grids, the paper proposes Bit-Plane Decomposition Quantization (BPDQ). BPDQ constructs a variable quantization grid for each group, theoretically expanding the feasible solution set for error minimization.

1. Variable Grid Formulation

Instead of a fixed template, BPDQ represents quantized weights $\hat{W}$ using bit-planes and scalar coefficients:
$\hat{W} = \text{REP}(C_0) + \sum_{i=1}^{k} \text{REP}(C_i) \odot B_i$
Where:

$B_i \in \{0, 1\}^{d_{out} \times d_{in}}$ are the $k$ most significant bit-planes derived from the original weights.
$C_i$ are group-wise scalar coefficients (including a bias $C_0$ ).
$\text{REP}(\cdot)$ expands coefficients across the input dimension.
$\odot$ denotes element-wise multiplication.

This formulation allows the relative spacing of quantization levels to vary across groups based on the learned coefficients $C_i$ , breaking the shape invariance constraint.

2. Optimization Framework

BPDQ operates within the Hessian-induced geometry to ensure alignment with the output reconstruction objective. The process involves:

Initialization: Bit-planes are selected via bit-plane decomposition (BPD) of the weights (keeping the $k$ most significant planes). Scalar coefficients are fitted using a closed-form weighted least-squares solution under the local Hessian metric.
Iterative Refinement: The algorithm alternates between:
1. Bit-Plane Update: For fixed coefficients, bit-planes are updated column-by-column via exact enumeration to minimize local reconstruction error.
2. Coefficient Refitting: With updated bit-planes, scalar coefficients are re-optimized via closed-form least squares.
Delta Correction: To maintain consistency with the error-propagation state (crucial for sequential PTQ), a delta correction is applied after coefficient refitting. This adjusts the accumulated propagation errors to account for the change in the quantized weight block, ensuring all iterations adhere to the same optimization objective.

Key Contributions

Insight: The paper identifies shape invariance in fixed quantization grids as the primary constraint limiting optimization-based PTQ in low-bit regimes. It demonstrates that the failure at 2-bit is not due to the optimization objective itself, but the rigidity of the feasible set.
Methodology: BPDQ extends optimization-based PTQ to variable grids. By decomposing weights into bit-planes and fitting independent scalar coefficients per group, it expands the feasible set from a 1D manifold (scaled template) to a higher-dimensional geometry. Theoretical analysis proves that BPDQ strictly contains the feasible set of standard uniform grids and can achieve strictly lower quantization error for certain weight vectors.
Performance: Extensive experiments show BPDQ achieves high fidelity in 2-bit and 3-bit regimes where state-of-the-art methods (GPTQ, AWQ) suffer catastrophic degradation.

Experimental Results

The authors evaluated BPDQ on the Qwen-3/2.5 family (0.6B–72B) and Ministral-3 (3B, 8B) across language modeling, reasoning, and long-context benchmarks.

2-bit Regime: On Qwen2.5-72B, 2-bit GPTQ and AWQ suffer severe performance drops (e.g., GSM8K accuracy falls below 41% or 0%). In contrast, BPDQ preserves high fidelity, achieving 83.85% accuracy on GSM8K (compared to 90.83% at 16-bit) using a single RTX 3090 (22.69 GB VRAM).
Reasoning Tasks: BPDQ significantly outperforms GPTQ and AWQ on reasoning benchmarks (GSM8K, MATH500) at 2-bit and 3-bit. For instance, on Qwen2.5-72B (W2-G64), BPDQ achieves 87.72% on GSM8K, while GPTQ achieves only 63.46%.
System Efficiency: While BPDQ requires approximately 3× the quantization time of GPTQ due to iterative refinement, it is significantly faster than Vector Quantization (VQ) methods like VPTQ (which incur ~40× overhead). For inference, BPDQ utilizes a bit-plane Look-Up Table (LUT) kernel, enabling low-latency decoding comparable to or better than GPTQ in 2/3-bit regimes.
Outlier Preservation: Activation analysis confirms that BPDQ inherently preserves critical outliers (features with high activation magnitude) that are often suppressed by fixed-grid methods, which correlates with the observed performance gains.

Significance and Claims

The paper claims that BPDQ offers a promising direction for extreme model compression by relaxing the rigidity of quantization grids while maintaining a hardware-friendly format. By theoretically expanding the feasible solution set, BPDQ enables the deployment of massive models (e.g., 72B parameters) on consumer-grade hardware with minimal fidelity loss. The authors position BPDQ as a rigorous solution that aligns the quantization process with the Hessian-induced optimization objective, overcoming the limitations of previous shape-invariant approaches without incurring the prohibitive costs of vector quantization or the training overhead of Quantization-Aware Training (QAT).

The work suggests that future hardware implementations (FPGA/ASIC) could benefit from the binary nature of bit-planes, and the unified basis of BPDQ naturally supports mixed-precision serving without requiring diverse hardware data types.