💬 NLP

pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Training

The paper proposes pQuant, a novel quantization-aware training method that decouples linear layers into a dominant 1-bit branch and a sparsely-activated high-precision expert branch to address parameter homogenization and achieve state-of-the-art performance in extremely low-bit language models.

Original authors: Wenzheng Zhang, Bingzheng Liu, Yang Hu, Xiaoying Bai, Wentao Zhang, Bin Cui

Published 2026-02-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenzheng Zhang, Bingzheng Liu, Yang Hu, Xiaoying Bai, Wentao Zhang, Bin Cui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a super-smart robot assistant (a Large Language Model) that can live on a tiny, battery-powered device like a smartwatch or a phone. The problem is, these robots are usually huge, requiring massive servers and lots of electricity.

To make them small enough to fit on a watch, scientists try to "shrink" the robot's brain. They do this by quantization, which is like taking a high-definition photo and turning it into a low-resolution pixel art image to save space.

The paper introduces a new method called pQuant to solve a specific problem with shrinking these robots too much.

The Problem: The "Democratization" Disaster

Imagine a classroom of 100 students. In a normal, high-precision robot, some students are geniuses (sensitive parameters) who solve the hardest problems, while others are helpers who do the routine work.

When scientists tried to shrink the robot's brain to 1-bit (the smallest possible size, like a light switch that is only "On" or "Off"), something went wrong. They accidentally treated every single student as if they were equally important.

The paper calls this "Parameter Democratization."

What happened: The system forced the geniuses and the helpers to wear the same "On/Off" uniform.
The result: The geniuses couldn't do their special work anymore because they were forced to be as simple as the helpers. The whole class became mediocre, and the robot got stupid.

The Solution: pQuant (The "Special Forces" Strategy)

The authors of pQuant realized, "Hey, we need to stop treating everyone the same!" They designed a new classroom layout with two specialized teams:

The 1-Bit Team (The Efficient Masses):
- Most of the students (95%+) are put in the 1-bit team.
- They are super fast, use very little energy, and handle the boring, everyday tasks.
- Think of them as the assembly line workers who keep the factory running efficiently.
The High-Precision Team (The Special Forces):
- A tiny group of students (the "sensitive" ones) are kept in a high-precision team (8-bit).
- They are allowed to keep their complex, detailed knowledge.
- Think of them as the elite surgeons or master engineers who handle the critical, difficult problems that the assembly line can't solve.

How It Works: The "Feature Scaling" Traffic Cop

The big question is: How does the robot know which student belongs to which team?

In the old methods, they just guessed or picked random spots. pQuant uses a smart Traffic Cop called Feature Scaling.

As the robot learns, the Traffic Cop watches the students.
If a student is struggling with a complex idea, the Cop says, "You! Go to the High-Precision Team!"
If a student is doing a simple task, the Cop says, "You! Stay with the 1-bit Team!"
This ensures the "geniuses" are always in the right place to save the day, while the "masses" keep things fast.

The Secret Weapon: The "Mixture of Experts"

To make this even better, pQuant adds a twist. Instead of just one High-Precision team, they create multiple small teams (like 4 or 8 different squads of surgeons).

For every single question the robot answers, a Router (a smart manager) looks at the question and picks only one of these expert squads to help.
Why is this cool? It's like having a library with 8 different sections of rare books. You don't need to open all 8 sections to find the answer; you just open the one you need. This makes the robot incredibly smart without slowing it down or using too much memory.

The Results: Small but Mighty

The paper shows that pQuant is a game-changer:

Smarter: It performs much better than previous 1-bit models (beating the "democratization" problem).
Faster: Because most of the work is done by the super-fast 1-bit team, it runs incredibly quickly on hardware.
Scalable: As you make the robot bigger, it doesn't get "dumb" like other 1-bit models do. It keeps getting smarter, almost as good as the giant, heavy 16-bit models, but with a fraction of the weight.

The Bottom Line

pQuant is like building a lean, mean, efficient machine by realizing that not all parts of a brain are created equal. Instead of forcing the whole brain to be simple, it gives the important parts a VIP pass to stay smart, while the rest of the brain stays super efficient. This allows us to put powerful AI into tiny devices without losing its intelligence.

1. Problem Statement

The paper addresses the critical challenge of deploying Large Language Models (LLMs) on resource-constrained edge devices. While Quantization-Aware Training from scratch (QAT-Scratch) has emerged as a promising approach for creating extremely low-bit (sub-2-bit) models, existing methods suffer from two major limitations:

Accuracy Degradation: Even state-of-the-art 1-bit models exhibit a non-negligible accuracy gap compared to full-precision (FP16) models.
Poor Scalability: As model size increases, performance gains in 1-bit models grow sublinearly, failing to match the scaling laws of FP16 models.

The authors identify the root cause of these issues as "Parameter Democratization." In existing 1-bit QAT-Scratch models, the sensitivity of all parameters becomes homogenized (uniform). In contrast, high-precision models naturally possess a small subset of "sensitive" parameters that disproportionately influence output. The extreme quantization in current 1-bit models destroys this structure, forcing all parameters to be treated with equal importance, which severely limits the model's expressivity and capacity to learn complex representations.

2. Methodology: pQuant

To counteract parameter democratization, the authors propose pQuant, a novel QAT-Scratch method based on decoupled linear quantization and guided parameter sensitivity.

Core Architecture: Decoupled Linear Layers

Instead of applying uniform 1-bit quantization to all weights, pQuant structurally splits linear layers (specifically in Feed-Forward Networks, and 1-bit only in Attention) into two specialized branches:

Dominant 1-bit Branch: Handles the majority of parameters using 1-bit weights (INT1) for maximum computational efficiency.
Compact High-Precision Branch: A small subset of parameters is retained in 8-bit precision (INT8). This branch is dedicated to preserving the most sensitive parameters.

This design is viewed through the lens of a Mixture of Experts (MoE):

The 1-bit branch acts as a shared expert for all inputs.
The 8-bit branch acts as a specialized, routable expert.

Key Mechanisms

Feature Scaling for Importance Allocation:
Since parameters are initialized randomly in QAT-Scratch, their sensitivity is not predetermined. pQuant introduces learnable scalar weights ( $\alpha$ and $\beta$ ) to modulate the outputs of the two branches.
- The 8-bit branch is initialized with a significantly higher scaling factor ( $\alpha \gg \beta$ , e.g., $\alpha=2.0, \beta=0.2$ ).
- This explicitly guides the optimizer to allocate sensitive, high-influence representations to the high-precision path during training, effectively "recovering" the differentiated sensitivity structure lost in standard 1-bit models.
Sparse Activation (Scaling):
To further scale capacity without linearly increasing inference cost, the high-precision branch is expanded into $N$ parallel experts. A lightweight router (a simple linear layer with Top-1 gating) dynamically selects the most suitable 8-bit expert for each token.
- This allows the model to increase total parameter count (capacity) while maintaining the same number of active parameters per forward pass as a dense layer.
Training Strategy:
- QAT-Scratch: Trained from random initialization without compressing pre-trained weights.
- Two-Phase Schedule: Uses a specific learning rate and weight decay schedule to stabilize 1-bit optimization and prevent gradient explosion.
- Straight-Through Estimator (STE): Used to approximate gradients for non-differentiable quantization operations.

3. Key Contributions

Identification of Parameter Democratization: The paper formally defines and demonstrates that the homogenization of parameter sensitivity is a primary bottleneck in 1-bit LLMs.
Decoupled Architecture: Introduces a structural solution that separates efficient 1-bit computation from high-precision sensitivity preservation, avoiding the "dynamic range collapse" of per-tensor quantization.
Guided Sensitivity Learning: Proposes feature scaling to dynamically guide the model in allocating sensitive parameters to the high-precision branch, rather than relying on fixed spatial patterns.
Efficient Scaling: Extends the high-precision branch into a sparse MoE-like structure, enabling capacity scaling with near-constant inference latency.

4. Experimental Results

The authors evaluated pQuant on models ranging from 300M to 2.6B parameters, comparing against BitNet (1-bit), BitNet1.58 (2-bit), and FP16 LLaMA-2 baselines.

Performance Gains:
- pQuant reduces perplexity by 32.0% compared to state-of-the-art 1-bit baselines.
- At the 1.3B scale, pQuant (1.35-bit) narrows the accuracy gap to the 2-bit BitNet1.58 to just 0.4 average accuracy points, despite using significantly fewer bits.
- The 2.6B pQuant model outperforms the 1.3B FP16 LLaMA-2 and significantly exceeds the 7B PTQ1.61 model.
Scalability:
- Unlike standard 1-bit models where performance plateaus, pQuant follows a scaling law much closer to FP16 models. Increasing the number of 8-bit experts ( $N$ ) consistently improves performance.
- With $N=8$ , pQuant nearly matches the loss of FP16 LLaMA-2 at the 1.3B scale.
Efficiency:
- Memory: pQuant reduces memory footprint by 92% compared to FP16 LLaMA-2 and 31% compared to BitNet1.58.
- Throughput: pQuant achieves 18.2% higher inference throughput than 2-bit models and is 82% faster than FP16 LLaMA-2 on CPU (Apple M2).
- Stability: pQuant demonstrates superior training stability compared to BitNet, which frequently suffers from gradient explosions.

5. Significance

The paper makes a significant contribution to the field of efficient LLMs by demonstrating that extremely low-bit models can achieve near-lossless performance if the architecture respects the inherent sensitivity structure of neural networks.

Paradigm Shift: It moves away from the idea that "all bits are equal" in low-bit models, advocating instead for a heterogeneous precision approach where critical parameters are protected.
Edge Deployment: By achieving FP16-level accuracy with sub-2-bit weights and significantly higher throughput, pQuant provides a viable path for deploying powerful LLMs on edge devices with strict memory and compute constraints.
Scalability: The method solves the scaling bottleneck of 1-bit models, suggesting that future low-bit LLMs can scale effectively to massive sizes without sacrificing accuracy.