💬 NLP

Multi-Bitwidth Quantization for LLMs Using Additive Codebooks

The paper introduces Drop-by-Drop, a novel post-training quantization framework that leverages additive codebooks and Matryoshka-style supervision to enable a single LLM checkpoint to support adaptive, multi-bitwidth inference with minimal storage overhead while maintaining competitive accuracy across various model architectures.

Original authors: Liza Babaoglu, Shuangyi Chen, Ashish Khisti

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Liza Babaoglu, Shuangyi Chen, Ashish Khisti

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: One Size Doesn't Fit All

Imagine you have a massive, incredibly detailed library of knowledge (a Large Language Model or LLM). This library is so huge that it requires a giant warehouse to store it.

The Cloud: If you run this library on a giant supercomputer in the cloud, you have plenty of space. You can keep every single book open and read the finest details.
The Phone: If you try to run this same library on a smartphone or a small edge device, the warehouse is too big. The phone's memory (RAM) and battery can't handle the full size.

The Current Solution (The "Static" Approach):
Right now, if you want to use this library on different devices, you have to make separate copies of the library:

A "Deluxe Edition" for the supercomputer (high quality, huge size).
A "Pocket Edition" for the phone (compressed, lower quality).
A "Travel Edition" for a tablet (medium size).

This is inefficient. You have to train, store, and maintain three different versions of the same library. If you want to switch from the phone to the tablet, you have to swap the whole file.

The New Solution: "Drop-by-Drop"

The authors propose a new method called Drop-by-Drop. Instead of making three different libraries, they create one single library that can magically shrink or grow depending on the device it's on, without needing to be retrained or swapped out.

Think of the library not as a stack of books, but as a Russian Nesting Doll (Matryoshka doll).

The Outer Shell: The smallest, most basic version. It fits in your pocket but only knows the most important facts.
The Middle Layer: If you open the shell, you find a slightly bigger doll with more details.
The Inner Core: The full-sized doll with every single detail.

With Drop-by-Drop, you don't need to carry three different dolls. You carry one.

On a phone, you just use the outer shell.
On a tablet, you open the shell to use the middle layer.
On a supercomputer, you open it all the way to use the full core.

How It Works: The "Additive Codebook" Recipe

To make this nesting doll work, the authors use a technique called Additive Quantization.

Imagine you are trying to describe a complex painting to someone over a phone call.

Standard Method: You send a low-resolution photo (blurry) for the phone, and a high-resolution photo for the computer. These are two different files.
Drop-by-Drop Method: You send a base sketch first.
- If the listener has a tiny screen, they stop there. They have a rough idea of the painting.
- If they have a bigger screen, you send Layer 2: a few more lines to define the shapes.
- If they have a huge screen, you send Layer 3: the final colors and details.

The magic is that Layer 2 and Layer 3 are useless without Layer 1, but Layer 1 is perfect on its own. The layers are "additive"—they stack on top of each other to build a clearer picture.

The Secret Sauce: "Matryoshka" Training

Usually, when you train an AI to compress data, it learns to make the best possible final picture. It doesn't care if the first few layers look bad. If you stop early, the picture is a mess.

The authors introduced a special training rule called Matryoshka Supervision.

The Analogy: Imagine a teacher grading a student's essay.
- Old Way: The teacher only grades the final, 10-page essay. If the student stops at page 1, the teacher says, "That's garbage, I can't grade it."
- Drop-by-Drop Way: The teacher grades the student at every stage. "Okay, page 1 is a good summary. Page 3 adds good details. Page 10 is perfect."
The Result: Because the AI was trained to make sure that every partial version (1 layer, 2 layers, 3 layers) was accurate, you can safely "drop" the extra layers when you are on a small device, and the remaining layers still work perfectly.

Why This Matters (According to the Paper)

One Model, Many Devices: You only need to store and download one file. The device decides how much of the file to "unpack" based on its memory.
No Retraining: You don't need to train a new model for every phone or tablet. The single model adapts automatically.
Smooth Transition: As you move from a low-power device to a high-power one, the quality improves smoothly, rather than jumping between different, incompatible models.
Proven Theory: The paper uses math (Information Theory) to prove that this "nesting" approach is theoretically possible for the types of data LLMs use, ensuring that you don't lose efficiency by adding these layers.

Summary

Drop-by-Drop turns a rigid, one-size-fits-all AI model into a flexible, "smart" model that can shrink down to fit in your pocket or expand to fill a supercomputer, all from a single file. It does this by training the AI to be good at every level of detail, not just the final, most detailed version.

Technical Summary: Drop-by-Drop for Multi-Bitwidth Quantization of LLMs

Problem Statement

Large Language Models (LLMs) face significant deployment challenges due to the vast memory and computational resources required by their billions of parameters. While quantization compresses these models into finite-precision representations, conventional methods typically produce models at a fixed compression level (e.g., a static 4-bit or 8-bit model). This rigidity is suboptimal for heterogeneous hardware environments ranging from cloud servers to resource-constrained edge devices. Maintaining multiple separately quantized model variants for different bitwidths incurs prohibitive storage and maintenance overhead. Furthermore, existing adaptive methods often rely on retraining or fail to provide a smooth trade-off between precision and efficiency without catastrophic performance degradation when components are removed.

Methodology: Drop-by-Drop (DbyD)

The authors propose Drop-by-Drop, a novel post-training quantization (PTQ) framework that enables a single trained model to operate at multiple precision levels without retraining. The method is grounded in information theory and builds upon Additive Quantization (AQLM).

Theoretical Foundation

The approach is motivated by the concept of successive source refinement. The paper establishes a theoretical theorem proving that Gaussian sources (which LLM weights empirically approximate) are successively refinable under a Weighted Mean Squared Error (WMSE) distortion measure. This distortion measure is derived from LLM loss functions, where the error is weighted by the input data matrix $X$ . The theorem demonstrates that a source can be encoded such that progressively adding bits yields progressively accurate reconstructions with no rate penalty compared to encoding directly at the target distortion level.

Algorithmic Implementation

Drop-by-Drop adapts the AQLM framework, which represents weights as sums of vectors from multiple learned codebooks.

Additive Codebooks: Instead of a single quantization stage, weights are reconstructed as a sum of vectors selected from $M$ distinct codebooks.
Matryoshka-style Supervision: To ensure that subsets of these codebooks yield accurate partial reconstructions, the authors introduce a specialized loss function during training. This loss aggregates the WMSE distortions of partial reconstructions using the first $k$ $k$ codebooks (for $k = 1, \dots, M$ $k = 1, \dots, M$ ).
- $L_{DbyD} = \sum_{k=1}^{K} \lambda_k D^{(k)}$ , where $D^{(k)}$ is the distortion using the first $k$ codebooks.
- This "Matryoshka" supervision explicitly encourages the first few codebooks to capture coarse, fundamental features, while subsequent codebooks provide finer refinements.
Inference-Time Flexibility: At inference, the model can dynamically adjust its precision by dropping codebooks. A device with strict constraints can use only the first few codebooks (e.g., $k=3$ ), while a resource-rich device can utilize all $M$ codebooks. This allows a single checkpoint to serve multiple bitwidths.

Key Contributions

Theoretical Proof: The paper proves that Gaussian sources are successively refinable under the WMSE distortion measure relevant to LLMs, providing a theoretical basis for progressive quantization without rate loss.
Drop-by-Drop Framework: A practical PTQ framework that integrates Matryoshka supervision into additive codebook training, enabling a single model to support multiple precision levels via codebook dropping.
Empirical Validation: Extensive experiments across diverse architectures (Qwen, LLaMA, Gemma, Mistral) demonstrate that Drop-by-Drop maintains competitive perplexity and task accuracy across the full range of supported bitwidths, exhibiting graceful degradation compared to standard AQLM when codebooks are dropped.

Experimental Results

The authors evaluated Drop-by-Drop on models ranging from 0.5B to 32B parameters using WikiText-2, C4, and five zero-shot reasoning benchmarks (ARC, HellaSwag, Winogrande, PIQA).

Perplexity: Drop-by-Drop variants consistently outperformed the baseline "AQLM (Drop)" method, which trains a standard AQLM model and simply drops codebooks at inference. For instance, on Mistral-7B at 3-bit precision, standard AQLM (Drop) suffered a massive perplexity increase (112.75), whereas Drop-by-Drop (Weighted) maintained a perplexity of 6.19, closely tracking the performance of individually trained models.
Accuracy: In zero-shot tasks, Drop-by-Drop (Weighted) achieved accuracy scores comparable to individually trained models (AQLM Ind.) across all bitwidths.
Efficiency: The method eliminates the need to train and store separate models for different bitwidths. Figure 8 illustrates that while conventional methods require cumulative training time and disk space for three separate models (3-bit, 4-bit, 5-bit), Drop-by-Drop requires only a single training run and storage instance while maintaining similar performance.

Significance and Claims

The paper claims that Drop-by-Drop bridges the gap between information-theoretic successive refinement theory and practical LLM quantization. Its primary significance lies in enabling adaptive precision control at inference time without retraining or recalibration. By allowing a single model to adapt its effective resource usage to match the capabilities of the deployment device, the method addresses the practical challenges of deploying LLMs on heterogeneous hardware. The authors position this as a novel connection between multi-bitwidth PTQ and successive refinement, offering a solution for run-time hardware-aware dynamic loading and mixed-precision quantization in real-world scenarios.