EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices

EntroLLM is a post-training compression framework that combines mixed quantization with entropy coding to significantly reduce storage requirements and accelerate inference for large language models on edge devices without retraining.

The Gist

Imagine you have a massive library of books (a Large Language Model) that you want to carry in your backpack to read while hiking (on an edge device like a smartphone or a small robot). The problem is that the library is too heavy and too big to fit in your backpack, and even if you could, your arms would get tired just trying to pull the books out one by one to read them.

The paper introduces a new method called EntroLLM to solve this. Think of it as a three-step magic trick to make the library smaller and easier to carry without losing any of the stories inside.

1. The "Spiky" Sorting (Mixed Quantization)

Usually, when people try to shrink these libraries, they just round off the numbers in the books to make them simpler (like rounding 3.14159 to 3.14). This is called quantization. However, standard methods often make the numbers look too "flat" and random, which is hard to compress further.

The authors' trick is to look at each chapter (or "layer") of the book individually. Depending on how the numbers in that specific chapter are distributed, they choose a special way to round them off:

• Unsigned Quantization: Like counting only positive steps.
• Asymmetric Quantization: Like shifting the zero point to fit the numbers better.

By doing this, the numbers in the library become "spiky." Imagine a mountain range where most peaks are clustered tightly in the middle, with very few extreme outliers. This "spiky" shape is much easier to compress than a flat, random landscape.

2. The "Abbreviation" Dictionary (Huffman Coding)

Once the numbers are sorted into this "spiky" pattern, the authors use a technique called Huffman coding.

Think of this like writing a secret code for the library. In English, the letter "E" appears very often, so you might decide to represent "E" with a single dot (•), while a rare letter like "Z" gets a long code (•••••).

• Because the "spiky" sorting made certain number values appear very frequently, the code gives those common numbers very short, tiny labels.
• The rare numbers get longer labels.

This shrinks the total size of the library significantly. The paper claims this step makes the compression 7 to 11 times better than current top methods. It's like turning a 100-page book into a 10-page pamphlet without changing the story.

3. The "Team Reading" Strategy (Parallel Decoding)

Here is the tricky part: Usually, to read a secret code, you have to read it one letter at a time from start to finish. If you have a huge library, this takes forever, and your backpack (the device) gets stuck waiting.

The authors realized that even though the code is short, the books are still organized in big chunks (tensors). So, they cut the library into many separate, independent sections.

• Instead of one person reading the whole code sequentially, they hire a team of readers (parallel threads).
• Each reader grabs a different chunk of the library and decodes their section simultaneously.
• Because the chunks are independent, they don't have to wait for each other.

This means that even though the library is tiny and compressed, the device can "unpack" the books almost instantly when needed, making the reading speed very fast.

The Results: A Lighter, Faster Backpack

The authors tested this on three different "libraries" (AI models) of varying sizes on a small device (an NVIDIA JETSON, which is like a powerful but tiny computer).

• Storage: They saved up to 30% more space compared to standard 8-bit models and 65% more compared to 4-bit models.
• Speed: Because less data had to be moved around, the device could think (infer) 30% to 146% faster.
• Accuracy: The "stories" (the AI's answers) remained just as accurate as the original, unshrunk library.

In short: EntroLLM is a way to pack a giant AI brain into a tiny backpack by organizing the data into a "spiky" shape, writing it in a super-efficient shorthand, and having a team of workers unpack it all at once. This makes it possible to run smart AI on small, battery-powered devices without needing a supercomputer.

Technical Summary

Technical Summary of EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices

Problem Statement
Large Language Models (LLMs) face significant deployment barriers on resource-constrained edge devices due to their massive parameter counts. Even smaller models, such as mistral-7B-Instruct, require over 14 GB of memory in 16-bit floating-point (fp16) format, exceeding the capacity of most edge hardware. Consequently, the primary bottleneck for LLM inference, particularly in generative tasks, is memory bandwidth rather than computational power. While quantization has reduced model sizes, further compression without compromising accuracy remains challenging, especially for smaller LLMs (≤10B parameters) designed for edge applications. Existing methods often require retraining or fail to achieve sufficient compression ratios for practical edge deployment.

Methodology
The authors propose EntroLLM, a compression framework that integrates mixed quantization with entropy coding (specifically Huffman coding) to reduce storage and memory bandwidth demands without retraining. The methodology consists of three core components:

1. Mixed Quantization Scheme: Unlike standard block-based quantization which flattens integer distributions and increases entropy, EntroLLM employs a tensor-level mixed quantization strategy. Based on the specific weight distribution of each layer, the framework applies either unsigned quantization or asymmetric quantization. This approach preserves a "spikier" distribution of integer codes, significantly lowering entropy and increasing compressibility compared to state-of-the-art (SOTA) block-based methods.
2. Huffman Weight Encoding: Leveraging the low-entropy distributions generated by the mixed quantization, the framework applies Huffman coding to the quantized weights. This lossless compression technique assigns shorter variable-length bit codes to frequent weight values and longer codes to rare ones. This step further reduces the effective bit-width of the model parameters, minimizing storage overhead and memory bandwidth requirements.
3. Parallel Decoding Strategy: To address the latency issues inherent in sequential entropy decoding, the authors implement a parallel decoding mechanism. By segmenting the encoded parameter space while preserving the original weight-tensor structure, the system allows different encoded tensors (chunks) to be decoded simultaneously across multiple CPU threads. This design eliminates the serial bottleneck, enabling real-time processing on edge hardware.

Key Contributions

• Layer-Specific Mixed Quantization: A strategy that balances storage efficiency and model performance by selecting between unsigned and asymmetric quantization per layer, improving downstream entropy compressibility by up to 8.1× (8-bit) and 13.1× (4-bit) over SOTA techniques.
• Entropy-Based Huffman Encoding: The application of Huffman coding to quantized weights to achieve further lossless compression while maintaining accuracy.
• Parallel Decoding Implementation: A novel partitioning scheme that enables parallelized decoding of Huffman-encoded weights, mitigating latency on edge devices.
• Comprehensive Evaluation: Validation across multiple edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) demonstrating significant storage savings and reduced inference latency.

Experimental Results
Experiments were conducted on edge-scale LLMs (smolLM-1.7B, phi3-mini-4k, mistral-7B) deployed on the NVIDIA JETSON P3450.

• Storage Efficiency: EntroLLM achieves up to 30% storage savings over uint8 models and 65% savings over uint4 models. For the phi3-mini-4k model, the effective bit-width was reduced to 5.58 bits (from 8-bit) and 1.39 bits (from 4-bit) after Huffman encoding.
• Inference Latency: On memory-limited devices, the framework demonstrated 31.9% to 146.6% faster inference compared to standard quantized models. Specifically, for uint4 weights on the phi3-mini-4k model, token generation latency improved by 146.6%, and pre-fill time improved by 13.3%.
• Accuracy: The method requires no retraining and maintains performance comparable to full-precision and standard quantized models across benchmarks including WIKITEXT2 (perplexity), HELLASWAG (accuracy), and GSM8K COT (accuracy).
• Decoding Overhead: Parallel decoding ensures that the decompression overhead is negligible. For instance, decoding 4-bit weights took only 1.66 seconds, a small fraction of the total inference time, and the cost is amortized over many generated tokens.

Significance and Claims
The paper claims that EntroLLM provides a practical solution for deploying LLMs on edge devices by addressing the memory bandwidth bottleneck through efficient weight compression. The framework is significant because it is compatible with existing post-training quantization pipelines and requires no retraining, making it immediately applicable to current models. By combining tensor-level mixed quantization with parallel Huffman decoding, EntroLLM enables real-time, low-power inference suitable for time-critical applications such as robotics and IoT, while preserving the accuracy of the original models. The authors note that future work will explore adaptive entropy coding and hardware-aware optimizations.