💬 NLP

SPQ: An Ensemble Technique for Large Language Model Compression

This paper introduces SPQ, an ensemble compression technique that synergistically combines SVD, activation-based pruning, and post-training quantization to achieve significant memory reduction and improved inference speed for large language models like LLaMA-2-7B while maintaining or enhancing their perplexity and downstream task accuracy compared to existing baselines.

Original authors: Jiamin Yao, Eren Gultepe

Published 2026-02-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiamin Yao, Eren Gultepe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a Large Language Model or LLM) that can write stories, solve math problems, and answer questions. The problem is, this library is so huge that it takes up an entire warehouse to store, and it takes a giant crane just to move a single book. You want to shrink this library down so it fits in a backpack and can be read quickly on a regular laptop, but you don't want to lose any of the smart stories inside.

This paper introduces a new method called SPQ (SVD-Pruning-Quantization) to do exactly that. Instead of just trying to shrink the library in one way, SPQ uses a "three-pronged attack," applying a different shrinking strategy to different parts of the library where it works best.

Here is how the three parts of SPQ work, using simple analogies:

1. The "Focus Filter" (SVD on Attention Layers)

The Problem: The library has a massive "Attention" section. This is where the model looks at a sentence and figures out which words are important to each other. In a huge model, this section is like a giant, messy filing cabinet with thousands of drawers, many of which are empty or contain duplicate files.
The SPQ Solution: SPQ uses a technique called SVD (Singular Value Decomposition). Think of this as a super-smart librarian who looks at the filing cabinet and says, "We don't need all these drawers. We can combine the contents of 100 drawers into just 10 compact boxes without losing any important information."

The Result: It keeps the most "vibrant" or important information (the variance) and throws away the redundant, low-energy noise. It shrinks the attention section significantly without making the librarian forget anything important.

2. The "Weeding the Garden" (Pruning on MLP Layers)

The Problem: The library also has a "Reasoning" section (called MLP layers), where the model does the heavy lifting of thinking. This section is like a garden with thousands of plants (neurons). Some plants are huge and blooming, but many are tiny, weak, or dead weeds that take up space but do nothing useful.
The SPQ Solution: SPQ uses Pruning. Instead of cutting the whole garden down, it looks at how much "sunlight" (activation) each plant gets. If a plant is rarely used, it gets cut out entirely.

The Result: It removes the dead weight. By only keeping the plants that actually contribute to the garden's beauty, the model becomes much smaller and faster, but the garden still looks and smells the same.

3. The "Paperback Conversion" (Quantization)

The Problem: Even after we've combined drawers and cut weeds, the books themselves are still written in high-definition, heavy, gold-leaf ink (32-bit floating point numbers). They are precise, but they take up a lot of space.
The SPQ Solution: SPQ uses Quantization. Imagine taking those heavy, gold-leaf books and reprinting them as standard, lightweight paperbacks (8-bit integers).

The Result: You lose a tiny bit of the "gold-leaf" shine, but for most stories, the text is still perfectly readable. This step alone cuts the size of the books by about 75%.

Why Combine All Three? (The Ensemble Magic)

If you only did one of these things, you'd run into trouble:

If you only pruned (cut weeds), you might accidentally cut a rare flower that was just sleeping, and the model gets confused.
If you only compressed the files (SVD), you might lose too much detail in the complex reasoning.
If you only reprinted as paperbacks (Quantization), the books get so small that the text becomes blurry and hard to read.

SPQ is the magic trick of doing all three at once.
It's like taking a heavy, overstuffed suitcase.

You first rearrange the clothes (SVD) to fit them tighter.
You throw away the socks you never wear (Pruning).
You roll everything up to save space (Quantization).

The result? The suitcase is now 75% smaller, but you can still find your favorite shirt instantly.

The Real-World Win

The paper tested this on a famous model called LLaMA-2-7B.

Before: It weighed about 27 GB (too big for most phones or small laptops).
After SPQ: It weighs only 6.86 GB.
Speed: It doesn't just fit in a backpack; it runs 1.9 times faster than other popular shrinking methods (like GPTQ).

The Bottom Line:
SPQ proves that you don't have to choose between a small model and a smart model. By being smart about where to apply each shrinking technique, you can fit a giant brain into a backpack and make it run faster than ever before. This makes powerful AI accessible to regular people on regular computers, not just massive data centers.

1. Problem Statement

Large Language Models (LLMs) have achieved remarkable performance but suffer from massive computational and memory costs, hindering their deployment on resource-constrained or real-time systems. Existing compression techniques—such as Singular Value Decomposition (SVD), structured pruning, and quantization—often operate in isolation. When applied aggressively, individual methods tend to degrade model performance (measured by perplexity and downstream accuracy). Furthermore, many combined approaches fail to account for the structural heterogeneity of LLMs (e.g., the distinct roles of Attention vs. MLP layers), leading to suboptimal compression ratios or performance loss.

2. Methodology: The SPQ Framework

The authors propose SPQ (SVD-Pruning-Quantization), a layer-aware ensemble technique that combines three complementary compression methods. The core philosophy is to apply each technique to the specific layer type where it is most effective, rather than applying a uniform method across the entire model.

The framework consists of three sequential, modular components:

A. Variance-Retained Low-Rank SVD (Attention Layers)

Target: Attention projection layers.
Mechanism: The authors apply SVD to weight matrices ( $W = U\Sigma V^\top$ ) but retain only the top- $k$ singular values.
Selection Criteria: Unlike methods that rely on reconstruction loss estimation, SPQ uses a variance retention threshold ( $\epsilon$ ). It keeps singular values such that the cumulative variance of the retained components meets a specific fraction (e.g., 84%–96%) of the total variance.
Benefit: This provides an interpretable, per-layer rank ratio without requiring complex loss estimation or group-level allocation.

B. Activation-Based Structured Pruning (MLP Layers)

Target: Feedforward Multilayer Perceptron (MLP) layers.
Mechanism: Entire neurons (rows of the weight matrix and corresponding biases) are removed based on their activation magnitude.
Selection Criteria: Neuron magnitude is calculated using $L1$ (mean absolute) or $L2$ (mean squared) norms over a calibration dataset.
Normalization: To determine pruning ratios per layer, the authors use log-scale normalization (or linear inverse/sigmoid decay) to map layer-wise activation statistics to pruning ratios ( $r_{min}$ to $r_{max}$ ). This ensures highly active layers are pruned less, while redundant neurons are removed.

C. Post-Training Linear Quantization (All Linear Layers)

Target: All linear layers (including those modified by SVD and pruning).
Mechanism: 8-bit symmetric linear quantization is applied post-training.
Scaling Strategies: The framework supports hybrid quantization modes to balance accuracy and efficiency:
- Per-tensor: One scale per layer.
- Per-channel: One scale per output channel (higher accuracy, higher compute cost).
- Hybrid (LNH, PBH, MSH): Dynamically selects per-channel or per-tensor based on layer sensitivity (e.g., applying per-channel only to the most sensitive layers).
Benefit: This adds a uniform compression layer without requiring retraining or calibration data, preserving the structural changes made by SVD and pruning.

D. LoRA Fine-Tuning (Recovery Step)

After compression, a lightweight Low-Rank Adaptation (LoRA) step is applied. Trainable low-rank matrices are injected into selected layers while original weights remain frozen. This recovers accuracy lost during aggressive compression with minimal parameter overhead.

3. Key Contributions

Layer-Aware Ensemble Design: SPQ is the first method to explicitly combine SVD on Attention layers, activation-based pruning on MLP layers, and linear quantization across all layers in a single pipeline. This addresses the "one-size-fits-all" limitation of prior work.
Variance-Based SVD Truncation: Introduces a variance-threshold approach for SVD that avoids the need for loss approximation, making it more interpretable and stable than previous SVD-LLM variants.
Lightweight Pruning Strategy: Utilizes activation statistics and log-scale normalization to derive pruning ratios, avoiding complex importance estimation or multi-structure pruning frameworks.
Hybrid Quantization: Proposes sensitivity-based hybrid quantization modes (PBH, LNH, MSH) that optimize the trade-off between perplexity and throughput without the heavy calibration costs of methods like GPTQ or AWQ.
Efficiency: The entire pipeline is designed to be computationally efficient, with a compression time faster than GPTQ despite the additional pruning and SVD steps.

4. Experimental Results

The framework was evaluated primarily on LLaMA-2-7B and other models (OPT, Vicuna, Mistral) ranging from 1B to 7B parameters.

Compression Ratio & Memory: SPQ achieves up to 75% memory reduction (reducing LLaMA-2-7B from ~26.95 GB to 6.86 GB). This is lower (better) than GPTQ (7.16 GB) at similar compression levels.
Perplexity & Accuracy:
- On WikiText-2, SPQ improved perplexity from 5.47 (original) to 4.91, outperforming the original model and all baselines.
- On downstream benchmarks (C4, TruthfulQA, GSM8K), SPQ maintained or improved accuracy compared to the uncompressed model and outperformed strong baselines like GPTQ and SparseGPT.
- Comparison: At 75% compression, SPQ significantly outperformed SparseGPT (50% compression) and ASVD (21% compression) in both memory savings and accuracy.
Inference Throughput:
- SPQ demonstrated a 1.9× speedup in tokens/second compared to GPTQ (4-bit) and a 1.3× speedup over GPTQ (8-bit).
- This is attributed to the removal of redundant neurons (pruning) and the efficiency of the hybrid quantization strategy.
Generalization: The method generalized well across different model families (LLaMA, OPT, Mistral), showing that larger models benefit most from the aggressive compression while maintaining or improving perplexity.

5. Significance and Conclusion

The paper demonstrates that heterogeneous, layer-aware compression is superior to applying a single technique uniformly. By recognizing that Attention layers benefit from low-rank approximation (SVD) while MLP layers benefit from neuron removal (Pruning), SPQ achieves a synergistic effect that preserves model quality even under extreme compression.

Key Takeaways:

Practicality: SPQ enables the deployment of 7B parameter models on consumer-grade hardware (e.g., single GPU with <7GB VRAM) without sacrificing performance.
Speed: Unlike many compression methods that trade speed for size, SPQ actually accelerates inference due to reduced parameter counts and efficient quantization.
Accessibility: The method does not require massive calibration datasets or complex iterative solvers, making it easier to implement than GPTQ or AWQ.

The authors conclude that SPQ provides a robust, practical solution for memory-constrained LLM deployment, offering a new standard for balancing compression ratios with model fidelity. Code is available at the provided GitHub repository.