🤖 machine learning

MosaicQuant: Inlier-Outlier Disaggregation for Unified 4-Bit LLM Quantization

MosaicQuant introduces a unified 4-bit LLM quantization framework that disaggregates weights into a dense base and a sparse residual to preserve accuracy, while its ZipperEngine component fuses their execution into a single low-bit pipeline to achieve near-FP16 performance with up to 1.24× speedup over baselines.

Original authors: Yangjia Hu, Haodong Wang, Zicong Hong, Qianli Liu, Quanxin Shou, Jian Lin, Song Guo, Xiaowei Shen, Xiangjun Huang, Dian Wang, Jian Yang

Published 2026-06-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yangjia Hu, Haodong Wang, Zicong Hong, Qianli Liu, Quanxin Shou, Jian Lin, Song Guo, Xiaowei Shen, Xiangjun Huang, Dian Wang, Jian Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of books (Large Language Models, or LLMs) that you want to carry in your backpack. The problem is, the books are so heavy and thick that your backpack can't hold them, and reading them takes forever.

To solve this, you decide to shrink the books. You want to compress them down to tiny, 4-bit "pocket editions" (quantization). This makes them light and fast to read. However, there's a catch: when you shrink a book too much, you lose the important details.

Most books have a lot of common, boring words (like "the," "is," "and") that are easy to shrink. But every now and then, there's a rare, complex word or a dramatic plot twist (an "outlier") that is crucial for the story. If you try to shrink these rare words down to the same tiny size as the common ones, the story falls apart, and the AI starts making mistakes.

The Old Way: The "VIP Section" Problem

Previous methods tried to fix this by saying, "Okay, let's keep the rare, important words in their original, heavy format (high precision) and only shrink the boring ones."

While this kept the story accurate, it created a new problem. Imagine a bus where most passengers are sitting on tiny folding stools (the small, compressed data), but a few VIPs are sitting in giant, heavy armchairs (the high-precision data).

The Issue: The bus driver (the computer chip) has to constantly switch gears to handle the different seats. It has to stop, move the heavy chairs, and convert them back and forth. This switching slows the bus down, canceling out the speed you gained by shrinking the other passengers.

The New Way: MosaicQuant

The authors of this paper propose a new system called MosaicQuant. Instead of keeping the VIPs in big chairs, they keep everyone on the same tiny folding stools (unified 4-bit). But, they add a clever twist to handle the rare, difficult words.

Here is how it works, using a "Mosaic" analogy:

1. The Base Layer (The Dense 4-bit)
They take the whole book and shrink it down to a uniform 4-bit size. This captures all the common words perfectly. However, the rare, complex words get a little blurry or distorted because they were forced into the tiny format.

2. The "Stitch" Layer (The Sparse Residual)
Instead of giving the rare words a whole new big chair, they create a tiny, invisible "patch" or "stitch" just for the parts of the story that got blurry.

They don't patch the whole book. They only patch the specific pages where the distortion is worst.
This patch is also 4-bit, so it fits on the same tiny stool.
Because they only patch a few specific spots (about 10% of the book), it's very light and doesn't weigh down the backpack.

3. The "Zipper" Engine (ZipperEngine)
This is the secret sauce that makes it fast. In the old "VIP" method, the computer had to stop and switch between the heavy chairs and the tiny stools.

MosaicQuant's ZipperEngine acts like a master tailor. It takes the main book (the dense base) and the tiny patches (the sparse residuals) and sews them together while the bus is moving.
It doesn't stop to switch gears. It processes the main text and the patches at the exact same time in a single, smooth motion. This means the computer chip never has to stop to "convert" data, keeping the speed high.

Why This Matters

The paper tested this on powerful AI models (like LLaMA and Qwen) and found two main results:

Accuracy: The stories (AI outputs) remained almost as perfect as the original heavy books. The "patches" fixed the blurry parts so well that the AI didn't lose its intelligence.
Speed: Because they didn't have to switch between different formats, the AI ran up to 1.24 times faster than the standard 16-bit version, and much faster than other methods that tried to mix high and low precision.

In short: MosaicQuant is like shrinking a whole library into a pocket size, but instead of leaving the important parts out or making them heavy, it adds tiny, lightweight "patches" to fix the errors, all while keeping the reading process smooth and fast.

Technical Summary: MosaicQuant

Problem Statement

While 4-bit quantization offers significant potential for reducing the memory footprint and accelerating the inference of Large Language Models (LLMs), it faces a fundamental limitation: the restricted bit-width cannot faithfully represent both dense common values (inliers) and rare, large-magnitude values (outliers). This discrepancy leads to substantial accuracy degradation.

Existing solutions typically employ mixed-precision quantization, retaining outliers in high precision (e.g., FP16 or INT8) while quantizing the rest to 4-bit. However, this approach breaks the uniformity of low-bit execution. The necessity for high-precision fallback operations introduces extra data movement, format conversion, and precision conversion overhead, which ultimately undermines the practical hardware speedup promised by low-bit inference.

Methodology

The authors propose MosaicQuant, a unified 4-bit quantization paradigm based on the principle of inlier–outlier disaggregation. Instead of preserving outliers in high precision, MosaicQuant decomposes the weight matrix entirely into 4-bit components: a dense base and a sparse residual.

1. Inlier–Outlier Disaggregation

The method begins by migrating activation outliers to the weight matrix using equivalent transformations (e.g., SmoothQuant or QuaRot). The migrated weight matrix $\hat{W}$ is then decomposed as:
$\hat{W} \approx Q(\hat{W}) + R$
Where:

Dense 4-bit Base ( $Q(\hat{W})$ ): Captures the common inliers faithfully.
Sparse 4-bit Residual ( $R$ ): Compensates for the quantization errors introduced by the outliers.

Crucially, both components operate in 4-bit, preserving the uniformity of the low-bit execution pipeline.

2. Block-Wise Residual Allocation

The paper identifies that quantization errors are not uniformly distributed across channels but exhibit fine-grained block-level concentration. Therefore, MosaicQuant allocates the sparse residual budget at the block level rather than the channel level.

To select the most critical blocks without expensive full-output evaluations, the authors introduce a Hessian-guided $\Delta$ -score. This metric approximates the marginal reduction in output distortion for each block by weighting the residual error with the activation sensitivity (diagonal Hessian estimate) of the input channels. The algorithm selects the top- $K$ blocks with the highest $\Delta$ -scores to receive the sparse 4-bit residual compensation. Experiments show that a sparse budget of only 10% of the blocks is sufficient to recover over 50% of the output distortion.

3. ZipperEngine: Unified Execution

A unified representation alone is insufficient if the sparse residual is executed via a separate kernel, which would reload activations and merge partial outputs, adding significant latency. To address this, the authors introduce ZipperEngine, a specialized execution engine that:

Fuses Kernels: Integrates the sparse residual computation directly into the dense 4-bit GEMM kernel.
Overlapped Pipeline: Utilizes a tiled, overlapped pipeline where quantization, dense computation, sparse compensation, and reduction-add operations are scheduled concurrently across Tensor Cores, CUDA Cores, and HBM.
Result: This eliminates the overhead of separate kernel dispatch and redundant memory traffic, realizing the hardware efficiency of a fully unified 4-bit inference pipeline.

Key Contributions

MosaicQuant Paradigm: A 4-bit quantization approach that performs block-wise inlier–outlier disaggregation, mitigating accuracy loss while maintaining a fully low-bit execution pattern.
Hessian-Guided Selection: An efficient, activation-aware block selection strategy using a $\Delta$ -Hessian score to identify critical residual blocks, replacing expensive exact output-distortion evaluation.
ZipperEngine: A dense–sparse 4-bit execution engine that fuses sparse residual computation into the dense GEMM kernel via an overlapped pipeline, enabling practical speedups even with additional residual blocks.

Experimental Results

The authors evaluated MosaicQuant on LLaMA3 (3B, 8B) and Qwen3 (4B, 8B, 14B, 32B) models.

Accuracy: MosaicQuant preserves near-FP16 accuracy. Under W4A4 quantization, it significantly outperforms other low-bit baselines (RTN, SmoothQuant, QuaRot, SpinQuant, ResQ, Atom). For instance, on Qwen3-32B, MosaicQuant achieves a perplexity of 8.4, compared to 9.2 for ResQ and 9.3 for Atom. It also maintains robust performance where other methods (like RTN and SmoothQuant) often collapse on Qwen3 models.
Throughput: On an NVIDIA RTX 4090, MosaicQuant achieves up to 1.24× speedup over the W16A16 baseline. It consistently delivers the highest throughput across batch sizes 4–32 compared to TensorRT-LLM and other quantization baselines.
Overhead: The sparse 4-bit branches incur minimal overhead, with memory usage increasing by only 8.5–9.5% and latency overhead remaining below 6.2% due to the fused pipeline.

Significance

The paper claims that MosaicQuant resolves the trade-off between accuracy and hardware efficiency in 4-bit LLM quantization. By eliminating the need for high-precision fallback operations and unifying both representation and execution into a single coherent low-bit pipeline, MosaicQuant enables efficient LLM serving without sacrificing the regularity of uniform 4-bit execution. This approach demonstrates that sparse, low-precision compensation can effectively recover outlier-induced errors while maintaining the throughput advantages of dense low-bit inference.