🤖 machine learning

Float8@2bits: Entropy Coding Enables Data-Free Model Compression

The paper introduces EntQuant, a data-free post-training compression framework that leverages entropy coding to achieve state-of-the-art extreme bit-rate compression (below 4 bits) for large models like 70B parameters in under 10 minutes, successfully unifying the speed and universality of data-free methods with the high fidelity typically requiring calibration data.

Original authors: Patrick Putzky, Martin Genzel, Mattes Mollenhauer, Sebastian Schulze, Thomas Wollmann, Stefan Dietzel

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Patrick Putzky, Martin Genzel, Mattes Mollenhauer, Sebastian Schulze, Thomas Wollmann, Stefan Dietzel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly detailed library of books (a Large Language Model). To read these books on a small device like a smartphone, you need to shrink them down. Usually, there are two ways to do this:

The "Fast & Dirty" Way: You quickly photocopy the books onto tiny index cards. It's fast and needs no extra research, but if you shrink them too much (below 4 bits), the words become gibberish, and the story collapses.
The "Slow & Perfect" Way: You hire a team of editors to carefully rewrite every sentence, using a specific set of reference books to make sure the meaning stays perfect. This works great, but it takes days, requires a lot of expensive computers, and you need access to the original reference books (which often don't exist or are private).

Enter EntQuant: The paper introduces a new method called EntQuant that acts like a "magic shrink-ray" that gets the best of both worlds. It's fast, needs no reference books, and keeps the story intact even when shrunk to an extreme size.

Here is how it works, using simple analogies:

1. The Problem: The "Fixed Box" Trap

Current methods force the library's words into fixed-size boxes.

If you want to save space, you must use tiny boxes (low bits).
But tiny boxes can only hold a few specific words. If the library needs to express a complex idea, it gets stuck because the box is too small. This is why current methods fail when trying to shrink models below 4 bits; they run out of "expressive power."

2. The Solution: The "Smart Filing System"

EntQuant changes the rules. Instead of forcing words into tiny boxes, it keeps the words in standard, high-quality boxes (like Float8 or Int8) but organizes them so cleverly that they take up almost no space.

Think of it like a packing service for a move:

Old Way: You have to fit your furniture into pre-sized crates. If you have a giant sofa, you have to cut it up to fit a small crate.
EntQuant Way: You keep the furniture whole (high precision), but you arrange it so tightly that the truck is nearly empty. You use a "smart algorithm" to pack the items so efficiently that the truck is 90% full of air, but the items are perfectly preserved.

3. The Secret Sauce: "Entropy Coding"

The paper uses a technique called Entropy Coding (specifically something called ANS).

Imagine you are writing a secret code. If the letter "E" appears 50% of the time, you give it a very short code (like "1"). If "Z" appears rarely, you give it a longer code (like "11010").
EntQuant first "trains" the model to use certain numbers more often than others (making the data "low entropy").
Then, it uses this smart coding to compress the data. Because the numbers are predictable, the computer can store them using less than 2 bits per parameter on average, even though the numbers themselves are still stored in high-precision formats.

4. Why This is a Big Deal

No "Calibration" Needed: Most high-quality compression methods need a "test drive" using a dataset to tune the model. EntQuant is data-free. You can take a model, compress it in under 10 minutes, and it just works. This is crucial for private or specialized models where you don't have the training data.
Extreme Compression: It successfully compresses massive models (like a 70-billion-parameter model) down to a size that fits on a consumer graphics card, while keeping the model smart enough to follow complex instructions and solve math problems.
Speed: While it has to "unpack" the data slightly while reading (decoding), the paper shows this happens so fast on modern computers that the slowdown is negligible (about 1.5 to 2 times slower than the original, which is still very fast).

The Bottom Line

The authors, Patrick Putzky and Martin Genzel (and their team at Merantix Momentum), have created a tool that lets you shrink giant AI models to the size of a small suitcase without losing their intelligence, and you can do it instantly without needing any extra data. It's like being able to fit a 70-foot yacht into a backpack without it breaking.

Key Takeaway: They broke the rule that said "to save space, you must lose quality." By using a smart filing system (entropy coding) instead of just chopping up the data, they kept the quality high while achieving extreme size reduction.

Technical Summary: EntQuant – Entropy Coding Enables Data-Free Model Compression

1. Problem Statement

Post-training quantization (PTQ) for Large Language Models (LLMs) currently faces a dichotomy between accessibility and fidelity.

Data-Free Methods (Level 1): Techniques like NF4 and Half-Quadratic Quantization (HQQ) are fast, model-agnostic, and require no calibration data. However, they suffer from "functional collapse" when compressed below 4 bits, failing to preserve model performance.
Data-Dependent Methods (Levels 2–4): Approaches utilizing calibration data or recovery training (e.g., QuIP#, EfficientQAT) achieve superior fidelity at extreme bit-rates but impose high computational costs and require access to training corpora. This is often impossible for specialized instruction-tuned or reasoning models due to data sovereignty, legal restrictions (e.g., GDPR), or the unavailability of training data.

Existing methods strictly couple compression rate with representational bit-width. To achieve 8× compression, standard frameworks force weights into a 2-bit representation (4 distinct values), which lacks the expressivity to model complex weight distributions, leading to performance degradation.

2. Methodology: EntQuant

The authors introduce EntQuant, a framework that decouples storage cost from representational precision by integrating entropy coding directly into the quantization pipeline.

Core Concept

Instead of forcing weights into a rigid low-bit format (e.g., Int2), EntQuant maintains a higher-precision representation (Float8 or Int8) but optimizes the weight values to minimize their entropy. These low-entropy weights are then losslessly compressed using an Asymmetric Numeral Systems (ANS) coder. This allows the model to achieve arbitrary effective bit-rates (down to ~2 bits per parameter) while retaining the full dynamic range of the base format during inference.

Technical Components

Entropy-Constrained Optimization:
- The goal is to minimize the empirical entropy of quantized weights $W_q$ subject to a reconstruction error constraint.
- Since direct entropy minimization is non-differentiable, the authors formulate a relaxed Lagrangian objective:
  $\min_{W_q} d(W, \hat{W}) + \lambda R(W_q)$
  where $d$ is the relative $\ell_1$ -loss (robust to outliers) and $R(W_q) = \|W_q\|_1$ serves as a differentiable proxy for entropy.
- The optimization is performed per layer using L-BFGS, tuning only the channel-wise scaling parameters $S$ initialized via the AbsMax algorithm.
Weight Encoding:
- Optimized weights are flattened into a symbol stream and compressed using a GPU-optimized ANS coder (via NVIDIA's nvCOMP).
- The final storage consists of the compressed bitstream, the scaling parameters, and metadata. The overhead of scales is negligible ( $<1\%$ ).
Inference-Time Decoding:
- Unlike traditional offline compression, EntQuant integrates decoding into the inference pipeline.
- Weights are stored as a compact bitstream in VRAM. Before the forward pass of a transformer block, a parallelized ANS decoder decompresses the weights into a buffer on the fly.
- This block-wise decompression strategy, inspired by DFloat11, allows for dense alternation between decompression and forward kernels, keeping GPU utilization high.
Handling Super Weights:
- For Int8 base formats, the method incorporates a mechanism to detect and exclude "super weights" (extreme outliers in early down-projection layers) from the entropy optimization, preventing performance collapse similar to methods requiring explicit outlier handling.

3. Key Contributions

Calibration-Free Extreme Compression: EntQuant enables post-training quantization down to effective 2-bit rates without recovery training or calibration data. It is a Level-1 method suitable for models where data access is restricted.
Decoupling Compression Rate from Bit-Width: The method breaks the strict coupling between storage size and quantization precision. It achieves low-bit storage costs (e.g., 2.1 bits/parameter) while maintaining the expressivity of Float8 or Int8, avoiding the functional collapse seen in fixed 2-bit quantization.
Simplified Outlier Handling: By using channel-wise scaling and entropy optimization, EntQuant naturally concentrates precision where needed without complex grouping schemes or explicit outlier detection (though it can optionally exclude super weights for Int8).
High-Speed Optimization: The compression stage is highly efficient, taking less than 10 minutes for a 70B parameter model, comparable to fast data-free methods like HQQ.

4. Experimental Results

The authors evaluated EntQuant on over 480 runs across 16 open-weight LLMs (including LLaMA-1/2/3, Qwen3, OLMo, and Mistral Large).

Data-Free Performance: EntQuant significantly outperforms existing data-free methods (HQQ, NF4) in the 2–3 bit regime. While HQQ and NF4 exhibit functional collapse (e.g., perplexity exploding to $>1000$ ) at 2 bits, EntQuant maintains usable performance (e.g., 2.1 bits on LLaMA-2 70B yields a perplexity of 6.47 vs. 32.24 for HQQ).
Comparison with Calibration Methods: EntQuant achieves results comparable to state-of-the-art calibration and fine-tuning methods (e.g., QuIP#, EfficientQAT) at 3 bits, with only a slight gap at 2 bits. Crucially, it achieves this without requiring access to training data.
Instruction-Tuned Models: EntQuant retains strong performance on instruction-following (IFEval), mathematical reasoning (GSM8K CoT), and scientific reasoning (GPQA) benchmarks, whereas other data-free methods fail catastrophically on these specialized tasks.
Inference Overhead: EntQuant is approximately 1.5–2× slower than the BFloat16 baseline but matches the inference speed of NF4. The decompression overhead is amortized over longer sequence lengths.
Memory Efficiency: A 70B parameter model compressed to 2.1 effective bits fits within ~18.8 GiB of compressed weights, allowing deployment on consumer-grade GPUs (e.g., 32 GiB RTX 5090) at 3 bits or lower.

5. Significance and Claims

The paper claims that EntQuant represents a paradigm shift by applying classical information theory principles (separating discretization from efficient representation) to modern Large Foundation Models.

Breaking the Barrier: It is presented as the first method to achieve functional extreme compression (below 3 bits) in a purely data-free manner, overcoming the performance barriers that fixed bit-width quantization approaches.
Practical Utility: By enabling the deployment of massive models (e.g., 70B+) on consumer hardware without the need for calibration data or retraining, EntQuant addresses the critical bottleneck of memory constraints in self-hosting scenarios.
Accessibility: The method democratizes access to frontier models, allowing immediate adoption of newly released models (weekly cadence) without the multi-hour calibration pipelines required by other high-fidelity methods.

The authors acknowledge that while EntQuant currently incurs a modest inference slowdown compared to uncompressed baselines, this is a pre-kernel-optimization phase similar to early GPTQ implementations, suggesting significant future speedups are possible with dedicated fused kernels.