💬 NLP

Channel-Wise Mixed-Precision Quantization for Large Language Models

This paper introduces Channel-Wise Mixed-Precision Quantization (CMPQ), a novel method that dynamically allocates arbitrary precision levels to weight channels based on activation distributions and employs non-uniform quantization with outlier extraction to significantly reduce memory usage while maintaining high performance for Large Language Models on edge devices.

Original authors: Zihan Chen, Bike Xie, Jundong Li, Cong Shen

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zihan Chen, Bike Xie, Jundong Li, Cong Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model, or LLM). This library is so huge that it doesn't fit on a normal bookshelf; it requires a warehouse the size of a football field just to hold the books. You want to take this library and shrink it down so it fits on a small, portable bookshelf (like a phone or a laptop), but you can't throw away the important stories, or the library won't make sense anymore.

This paper introduces a new way to shrink these digital libraries called CMPQ (Channel-Wise Mixed-Precision Quantization). Here is how it works, explained simply:

The Problem: The "One-Size-Fits-All" Suit

Currently, most methods try to shrink the library by putting every single book into a box of the exact same size.

The Old Way: If you decide to use "3-bit" boxes (a specific small size), every book gets forced into a 3-bit box.
The Issue: Some books are thick and complex (they need a big box), while others are thin and simple (they fit in a tiny box). If you force a thick book into a tiny box, the pages get crumpled (the AI makes mistakes). If you put a thin book in a huge box, you are wasting space.
The Limitation: Existing methods are rigid. They can only use whole-number sizes (like 2-bit, 3-bit, or 4-bit). If your device has just a little bit more space than a 2-bit box allows, but not enough for a full 3-bit box, current methods can't use that extra space effectively.

The Solution: CMPQ's "Smart Packing"

CMPQ is like a super-smart packing service that looks at every single book individually and decides the perfect box size for it, based on how important that book is.

1. The "Channel" Concept (The Bookshelves)
Think of the AI's brain as having thousands of different "channels" or bookshelves. The paper discovered that not all shelves are equally important. Some shelves hold the most critical stories (high activation), while others hold filler content.

CMPQ's Move: Instead of treating the whole library the same, it looks at each shelf. If a shelf has very important stories, it gives those books a larger, higher-quality box (more precision, like 4-bit). If a shelf has less important stories, it gives them a smaller, tighter box (less precision, like 2-bit).

2. The "Fractional" Magic (The Custom Fit)
This is the paper's biggest trick. Because CMPQ mixes big boxes and small boxes together, it can create an average size that isn't a whole number.

The Analogy: Imagine you have a budget that fits exactly 2.5 boxes.
- Old methods say: "We can only use 2-boxes or 3-boxes. We can't do 2.5."
- CMPQ says: "No problem! We'll make 50% of the books fit in 2-boxes and 50% in 3-boxes. The average is 2.5, and we use every inch of your space perfectly."
The Result: You can squeeze the library into a slightly larger space than before, and the AI gets significantly smarter because it didn't have to crush the important books.

3. Protecting the "Outliers" (The Rare Gems)
Sometimes, a book has a few pages that are incredibly weird or unique (called "outliers"). If you try to shrink these pages, the whole story breaks.

CMPQ's Strategy: It has a special "Outlier Protection" system. It identifies these rare, weird pages before shrinking the rest of the library. It keeps these specific pages in their original, full-size format (like keeping them in a glass case) while shrinking everything else. This ensures the weird, critical details aren't lost.

What the Paper Found

The authors tested this on nine different large AI models (like OPT and LLaMA). Here is what they found:

Better Performance: CMPQ made the AI smarter than previous methods, even when using very small box sizes (like 3-bit).
Fractional Wins: When they allowed the AI to use "in-between" sizes (like 2.2 bits or 3.4 bits), the AI's performance jumped up significantly compared to methods that were stuck on whole numbers.
Efficiency: It didn't require retraining the AI from scratch (which is expensive and slow). It just rearranged the existing books.
Speed: It runs just as fast as the older methods, so you don't lose speed to gain accuracy.

In a Nutshell

CMPQ is a smarter way to compress AI models. Instead of forcing every piece of the AI into the same small container, it treats different parts of the AI differently. It gives the important parts more room and the less important parts less room. This allows the AI to fit into smaller devices without losing its "brainpower," and it can even use tiny bits of extra space that other methods ignore.

Technical Summary: Adapting to Any Bit-Width: Channel-Wise Mixed-Precision Quantization for LLMs

1. Problem Statement

Large Language Models (LLMs) face significant deployment challenges on edge devices due to their massive parameter sizes, which impose substantial memory requirements. While weight-only Post-Training Quantization (PTQ) offers a solution by reducing precision (e.g., from FP16 to 3-bit), existing approaches suffer from two primary limitations:

Integer-Bit Rigidity: Most state-of-the-art PTQ methods (e.g., GPTQ, AWQ) focus on uniform integer-bit quantization across all layers. This restricts their adaptability to devices that could support fractional average bit-widths (e.g., 2.2 or 3.8 bits), leading to underutilization of available storage.
Ineffective Mixed-Precision Strategies: Existing mixed-precision methods, such as LLM-MQ, often rely on layer-wise sensitivity estimation using first-order information (gradients). However, in converged LLMs, gradients are approximately zero, making it difficult to differentiate layer sensitivities effectively. Furthermore, layer-wise allocation can concentrate information loss in specific layers, degrading overall performance. Additionally, methods that passively retain outliers in FP16 often incur increased inference latency without proportional performance gains.

2. Methodology: Channel-Wise Mixed-Precision Quantization (CMPQ)

The authors propose CMPQ, a PTQ framework designed to adapt LLMs to any average bit-width constraint, including non-integer values, without sacrificing performance. The method operates through three core components:

A. Channel-Wise Non-Uniform Quantization

Unlike uniform quantization which divides weight ranges into evenly spaced bins, CMPQ employs non-uniform quantization (via K-means clustering) to better handle the non-uniform weight distributions typical of LLMs.

Precision Allocation: Instead of assigning a fixed bit-width to an entire layer, CMPQ allocates precision at the channel level.
Activation-Based Heuristic: The method computes the L2-norm of activations for each channel during a forward pass on a calibration set.
- If the target average bit-width $b > 3$ , channels with large activation norms (salient channels) are assigned higher precision (e.g., 4-bit) to preserve critical information.
- If $b < 3$ , less critical channels are assigned lower precision (e.g., 2-bit) to minimize storage.
- For $b = 3$ , a small fraction (~1%) of salient channels are protected with 4-bit precision, while the rest use 3-bit.
Algorithm: The allocation is determined by Algorithm 1, which uses quantiles of the activation norm vector to assign bit-widths $c_i \in \{2, 3, 4\}$ to each channel $i$ .

B. Dual Outlier Protection

To mitigate the degradation caused by extreme values (outliers), CMPQ integrates two distinct outlier extraction techniques that preserve specific weights in FP16:

Activation-Based Outliers ( $O_{act}$ ): Channels corresponding to the top 0.45% of activation L2-norms are identified and preserved in FP16. This addresses the observation that outliers often exhibit a channel-wise pattern.
Quantization-Aware Outliers ( $O_q$ ): After removing $O_{act}$ , the remaining weights undergo channel-wise non-uniform quantization. The top 0.05% of weights with the largest absolute quantization errors ( $|W' - W'_q|$ ) are identified and preserved in FP16. This ensures that the clustering centroids used for quantization represent the majority of the weight distribution rather than being skewed by extreme values.

Total Overhead: The combined outlier ratio is maintained at 0.5%, consistent with baseline methods, ensuring fair storage comparisons.

C. Efficiency and Deployment

No Backpropagation: CMPQ requires only a single forward pass for calibration, avoiding the memory-intensive backward passes required by methods like SqueezeLLM.
Inference: The framework handles mixed-precision matrix multiplication by decomposing weights into sub-matrices based on channel allocation. Outliers are processed via standard CSR sparse kernels, while the bulk of weights are processed via low-bit kernels.

3. Key Contributions

Adaptive Framework: Introduction of CMPQ, a post-training quantization framework that supports arbitrary average bit-widths (including fractional values) via channel-wise precision allocation.
Novel Mechanism: Integration of non-uniform quantization with a dual outlier handling strategy (activation-based and quantization-aware) that provides fine-grained control with low overhead.
Performance Gains: Demonstration that CMPQ outperforms prior methods in both integer-bit and fractional-bit settings, achieving significant performance improvements with modest increases in memory usage.

4. Experimental Results

Experiments were conducted on nine LLMs (OPT, LLaMA2, LLaMA3) across various bit-widths (2, 3, 4 bits) and datasets (WikiText-2, C4, PIQA, HellaSwag, MMLU).

Integer-Bit Performance: CMPQ achieves state-of-the-art or competitive perplexity scores compared to baselines like GPTQ, AWQ, QuIP, and LLM-MQ. For example, on OPT-2.7B at 3-bit, CMPQ achieves a WikiText-2 perplexity of 13.38, outperforming LLM-MQ (18.09) and GPTQ (17.09).
Fractional-Bit Performance: CMPQ demonstrates superior adaptability to fractional bit-widths (e.g., 2.2, 3.4 bits). On LLaMA2-7B, moving from 2-bit to 2.2-bit average precision yields a 30% improvement in perplexity (from 15.97 to 11.11), a gain unachievable with integer-only methods.
Robustness: CMPQ shows greater robustness to calibration set variations compared to Hessian-based methods like QuIP, which can diverge when the evaluation dataset differs from the calibration set.
Efficiency:
- Memory: The overhead for storing bit-allocation metadata and cluster centers is minimal (<0.03 bits per weight).
- Latency: Inference latency and peak memory usage are comparable to integer-only quantization methods like SqueezeLLM, with only a modest increase (approx. 11-12%) for larger models (13B) due to the auxiliary high-precision path.
- Calibration: CMPQ is highly data-efficient, achieving desired performance with as few as 1-2 calibration samples, whereas other methods often require 50+.

5. Significance and Claims

The paper claims that CMPQ represents a significant step forward in LLM quantization by bridging the gap between theoretical storage efficiency and practical deployment constraints.

Adaptability: It enables the "structured use of fractional-bit precision," allowing developers to fully utilize available device memory budgets rather than being restricted to integer-bit schemes.
Trade-off Optimization: The method achieves a superior memory-accuracy trade-off by proactively allocating precision to low-bit channels while keeping the FP16 footprint fixed, avoiding the latency penalties associated with passive outlier retention.
Practicality: By eliminating the need for backpropagation and requiring minimal calibration data, CMPQ offers a scalable, efficient solution suitable for real-world deployment scenarios where computational and memory resources are constrained.

The authors acknowledge limitations, including the focus on weight-only quantization, the reliance on a discrete set of precision levels ({2, 3, 4}), and the assumption that the calibration set is representative of the target usage distribution. However, they assert that CMPQ provides a highly adaptable and effective approach to LLM quantization across diverse device capabilities.