🤖 machine learning

SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression

SLIM is a one-shot compression framework for large language models that holistically integrates hardware-friendly uniform quantization, semi-structured sparsity, and a novel saliency-based low-rank approximation method to significantly improve accuracy and inference speed while eliminating the need for computationally expensive retraining.

Original authors: Mohammad Mozaffari, Amir Yazdanbakhsh, Maryam Mehri Dehnavi

Published 2025-08-14✓ Author reviewed ⓘ

📖 6 min read🧠 Deep dive

Original authors: Mohammad Mozaffari, Amir Yazdanbakhsh, Maryam Mehri Dehnavi

✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Large language models are the engines behind a new generation of artificial intelligence, capable of writing stories, solving problems, and holding conversations with remarkable fluency. These systems are built from billions of mathematical connections, or parameters, that work together to process information. While this massive scale allows them to understand nuance and context, it also creates a heavy burden: the models require enormous amounts of computer memory and processing power to run. This makes them difficult to use on everyday devices like smartphones or laptops, often restricting them to expensive, specialized data centers. For years, researchers have tried to shrink these models to make them more accessible, but the process has been a trade-off. Techniques that compress the models often degrade their intelligence, forcing a choice between a smaller, faster program and a smart, accurate one.

A team of researchers has developed a new method called SLIM that aims to break this trade-off. Their work focuses on a specific challenge: how to compress a large language model in a single step, without the need for expensive retraining, while keeping its performance nearly identical to the original. The researchers combined three distinct compression techniques—reducing the precision of the numbers the model uses, removing unnecessary connections, and adding small, flexible adjustments—to create a unified process. The result is a system that can shrink a model significantly while maintaining, and in some cases even improving, its ability to answer questions and generate text.

The core difficulty the researchers faced was that compressing a model usually introduces errors. Imagine trying to fit a large, detailed painting into a small frame; if you simply cut off the edges or squish the image, you lose important details. In the digital world, this happens when researchers reduce the number of bits used to store numbers or when they delete parts of the model's structure. Previous methods tried to fix these errors by retraining the model on vast amounts of data, a process that is slow and computationally expensive. Other methods attempted to fix the errors in a single step but often failed when trying to combine different compression techniques, leading to a significant drop in quality.

The SLIM framework approaches this problem by treating the compression process as a coordinated sequence of three steps. First, the researchers simplified the numbers inside the model. Instead of using complex, variable scales for different parts of the data, they developed a new way to apply a single, uniform scale to entire groups of numbers. This makes the process much faster and easier for standard computer hardware to handle. However, they knew this uniform approach would still leave some inaccuracies, so they immediately moved to the second step: removing the least important connections. They used a proven method to identify which parts of the model were doing the least work and pruned them away, creating a sparse structure that is faster to process.

The third step is where the innovation truly shines. After simplifying the numbers and removing connections, the model inevitably loses some of its original precision. To recover this lost accuracy without retraining the whole system, the researchers added small, low-rank adapters. Think of these adapters as a set of fine-tuning tools that can be attached to the compressed model to correct its mistakes. The researchers devised a unique mathematical way to calculate exactly what these tools should be, based on how much each part of the model contributed to the errors. By using a specific function that measures the importance of different parts of the data, they could mathematically derive the perfect settings for these adapters in a single pass. This allowed them to compensate for the errors introduced by the first two steps, effectively restoring the model's intelligence.

The results of this approach were tested on several large language models, including versions with 7 billion and 13 billion parameters. When the researchers applied their method, they found that the compressed models performed significantly better than previous attempts. For a 7-billion-parameter model, the new method improved accuracy by up to 5.66 percent compared to other one-shot compression techniques. In some cases, the compressed models were even more accurate than the original, uncompressed models when measured by the same amount of memory they occupied. This suggests that the method does not just preserve quality; it can actually optimize the model's efficiency to a degree that the original, bulky version could not achieve within the same resource limits.

Beyond accuracy, the method delivers substantial speed improvements. Because the models are both smaller and structured in a way that modern graphics cards can process efficiently, they run much faster. On a standard consumer graphics card, the compressed models ran up to 4.3 times faster than the original dense models. On more powerful professional hardware, the speedup was still significant, reaching 3.8 times faster. Additionally, the memory required to run these models dropped dramatically, with some configurations using less than a quarter of the memory needed for the original dense versions. This reduction in memory footprint means that models which previously required massive servers could potentially run on more accessible hardware.

The researchers also explored whether a small amount of additional training could help even further. They found that by allowing the small adapters to adjust slightly on a tiny amount of data, the model's accuracy could be boosted by another 1.66 percent. This optional step offers a way to squeeze out even more performance if the application allows for a brief period of adjustment, but the core method remains effective without it. The team compared their work against other leading methods and found that their approach consistently outperformed them, particularly when combining different types of compression. While other methods struggled to handle the combination of removing connections and reducing number precision simultaneously, SLIM managed to integrate them seamlessly.

This work represents a significant step forward in making large language models practical for a wider range of devices. By solving the problem of how to compress a model in a single step without sacrificing its intelligence, the researchers have provided a pathway for deploying advanced artificial intelligence on hardware that is more common and affordable. The method does not rely on complex, custom hardware or massive retraining efforts, making it a viable solution for real-world applications. As the demand for efficient and accessible artificial intelligence grows, techniques like SLIM offer a way to balance the need for speed and memory with the demand for high-quality, intelligent performance.

Technical Summary: SLIM - One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression

1. Problem Statement

Large Language Models (LLMs) face significant challenges regarding high memory consumption and slow inference costs due to their extensive parameter counts. While conventional compression techniques like pruning and quantization can mitigate these issues, they typically require computationally expensive retraining on large-scale datasets to restore accuracy, rendering them impractical for many real-world applications.

Existing one-shot compression methods eliminate retraining costs but struggle to achieve accuracy comparable to dense models, particularly when combining multiple compression techniques. Specifically:

Joint Application Issues: Combining pruning and quantization often compounds accuracy degradation.
Hardware Constraints: State-of-the-art one-shot methods (e.g., SparseGPT, Wanda) perform well with unstructured sparsity but struggle with semi-structured patterns (e.g., NVIDIA's 2:4 sparsity) crucial for hardware-accelerated inference.
Low-Rank Adapter Limitations: While low-rank adapters (LoRA) can recover accuracy, existing approaches typically require expensive retraining or fail to seamlessly integrate with both sparsity and quantization simultaneously.

2. Methodology

The paper proposes SLIM, a one-shot compression framework that holistically integrates hardware-friendly quantization, semi-structured sparsity, and low-rank approximation into a unified process without retraining. The pipeline consists of three sequential steps:

2.1. SLIM-Quant: Probabilistic Uniform Quantization

To address the non-convex nature of optimizing uniform quantization scaling factors, SLIM introduces SLIM-Quant.

Formulation: Instead of relying on grid search or complex group quantization, the authors formulate the quantization process using a probabilistic approach. They reformulate the mean squared error (MSE) minimization problem into a convex optimization problem by integrating the weight magnitude probability density function (PDF).
Optimization: Since weight distributions do not strictly follow standard PDFs, the method employs numerical integration on the weight histogram using a multi-grid strategy to efficiently find the optimal scaling factor ( $\alpha$ ).
Activation-Aware Variant: An optional extension (SLIM-QuantO) scales weights and activations based on a novel saliency metric (product of normalized input and weight magnitudes) to minimize output error for critical channels, similar to AWQ but with a distinct saliency definition.

2.2. Saliency-Based Pruning

Following quantization, the framework applies a one-shot pruning method (specifically Wanda) to the quantized weights. This step introduces semi-structured sparsity (e.g., 2:4 patterns) or unstructured sparsity. The combined effect of quantization and pruning is modeled as additive noise ( $E_Q + E_S$ ).

2.3. SLIM-LoRA: Saliency-Based Low-Rank Approximation

To compensate for the aggregated errors from quantization and sparsity, SLIM introduces SLIM-LoRA, a one-shot low-rank adapter method.

Novel Saliency Function: Unlike "Naive-LoRA" which minimizes total error norms, SLIM-LoRA utilizes a saliency function $F$ $F$ that is both invertible and additive.
- Additivity: $F(A + B) = F(A) + F(B)$ , allowing the isolation of error saliency from original weight saliency.
- Invertibility: Enables the mathematical derivation of adapter values without iterative optimization.
Mathematical Derivation: The method defines $F(W) \triangleq \text{diag}(x)W$ , where $x$ is the average absolute value of inputs from a calibration set. The optimal low-rank adapters ( $L, R$ ) are computed by performing Singular Value Decomposition (SVD) on the saliency-weighted error matrix:
$\text{diag}(x)L, R = -\text{SVD}(\text{diag}(x)(E_Q + E_S))$
The left adapter $L$ is then recovered via an inverse saliency transformation.
Adapter Quantization: To prevent the adapters from reintroducing memory overhead, the low-rank adapters are quantized using a 4-bit AbsMax group quantization scheme.

2.4. Optional Post-Compression Fine-Tuning

The framework includes an optional Parameter-Efficient Fine-Tuning (PEFT) phase. During this phase, the sparse and quantized weights are frozen, and only the low-rank adapters are fine-tuned using a straight-through estimator (STE) for quantization-aware training. This requires minimal computational resources (thousands of tokens) compared to full fine-tuning.

3. Key Contributions

Unified One-Shot Framework: SLIM is the first framework to seamlessly integrate uniform quantization, semi-structured sparsity, and low-rank approximation in a single one-shot process without retraining.
SLIM-Quant: A probabilistic formulation that enables efficient uniform quantization, achieving accuracy comparable to complex group quantization methods while retaining the speed benefits of uniform quantization.
SLIM-LoRA: A novel low-rank adaptation method that uses an invertible and additive saliency function to mathematically compute adapter values, eliminating the need for expensive retraining or iterative optimization.
Hardware Efficiency: The method is designed specifically for hardware acceleration, supporting semi-structured sparsity (2:4) and utilizing optimized kernels (e.g., Sparse Marlin) for inference.

4. Experimental Results

The authors evaluated SLIM on LLaMA-2 (7B, 13B) and OPT (125M to 13B) models using zero-shot downstream tasks (MMLU, Piqa, Arc, etc.) and perplexity on WikiText2.

Accuracy:
- Under 2:4 sparsity and 4-bit weight quantization, SLIM improved model accuracy by up to 5.66% on LLaMA-2-7B compared to prior methods.
- SLIM outperformed existing compression techniques and, in some configurations, even surpassed dense models of equivalent parameter size (up to 0.6% higher accuracy).
- The optional PEFT recipe provided an additional accuracy boost of up to 1.66% on LLaMA-2-13B.
Speedup:
- On NVIDIA RTX 3060 GPUs, SLIM achieved up to 4.3× layer-wise speedup.
- On NVIDIA A100 GPUs, it achieved up to 3.8× layer-wise speedup.
Memory Reduction:
- Compressed models achieved up to 0.23× end-to-end memory reduction compared to their dense counterparts (i.e., reducing memory usage to roughly 23% of the original).
Comparison: SLIM consistently outperformed state-of-the-art methods including SparseGPT, Wanda, Group OPTQ, L2QER, and MaskLLM, particularly in low-bitwidth (4-bit) and semi-structured sparsity scenarios where other methods struggled or failed (e.g., OOM errors).

5. Significance and Claims

The paper claims that SLIM shifts the Pareto frontier for model compression, delivering higher model accuracy at the same total bit budget compared to existing techniques. By eliminating the need for retraining while effectively combining quantization, sparsity, and low-rank approximation, SLIM offers a practical solution for the efficient deployment of large-scale LLMs on resource-constrained hardware (e.g., consumer GPUs, edge devices).

The authors emphasize that their approach addresses the persistent accuracy gap in low-bitwidth quantization and semi-structured sparsity, making it an appealing solution for sustainable and accessible AI deployment. They also note the ethical considerations of increased accessibility, highlighting the need for safeguards in critical applications.