💬 NLP

SigmaScale: LLM Compression with SVD-based Low-Rank Decomposition and Learned Scaling Matrices

SigmaScale is a novel LLM compression method that optimizes learned diagonal scaling matrices to reduce the effective intrinsic rank of weight matrices via activation-aware SVD, achieving competitive performance with state-of-the-art techniques on models like Llama 3.1 and Qwen3.

Original authors: Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

Published 2026-06-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ernests Lavrinovics, Marco Letizia, Roy Janco, Shai Segal, Johannes Bjerva, Maurizio Pierini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Giant Models are Heavy

Imagine Large Language Models (LLMs) like Llama 3.1 or Qwen3 as massive, incredibly detailed libraries. These libraries contain billions of books (parameters) that allow the AI to be smart and answer questions. However, because they are so huge, they are heavy to carry, expensive to run, and require powerful, energy-hungry computers just to read a single sentence.

Scientists want to shrink these libraries down to fit on smaller devices without losing the "smart" parts. This is called compression.

The Old Way: The "Blindfolded Editor"

One popular way to shrink these models is called SVD (Singular Value Decomposition). Think of the model's brain as a giant spreadsheet of numbers.

The Analogy: Imagine you have a 100-page document, but you only have space to keep 10 pages. The old SVD method acts like a blindfolded editor. It looks at the document and says, "Okay, I'll keep the first 10 pages that have the most ink on them and throw the rest away."
The Flaw: Sometimes, the most important information isn't on the pages with the most ink. It might be hidden in the margins or written in a tiny font. By blindly cutting off the "tail end" of the data, the model loses its ability to understand nuance, and it starts making mistakes.

The New Solution: SigmaScale (The "Smart Shrinker")

The authors of this paper propose a new method called SigmaScale. Instead of just blindly cutting off the bottom 90% of the data, SigmaScale acts like a smart editor with a magnifying glass and a highlighter.

Here is how it works, step-by-step:

1. The "Stretch and Shrink" Trick

Before the editor cuts the pages, SigmaScale applies a special "stretch and shrink" filter to the document.

The Analogy: Imagine the document is made of rubber. SigmaScale stretches the important parts (making them bigger and easier to see) and shrinks the unimportant parts (making them smaller).
How it does this: It learns two sets of numbers (vectors) that act like row and column rulers. It doesn't just guess where to cut; it learns exactly how to stretch the data so that the most critical information moves to the top of the list.

2. The "Activation-Aware" Loss

The paper mentions that this method is "activation-aware."

The Analogy: A normal editor just looks at the paper. SigmaScale looks at how the paper is actually used. It asks, "When a human reads this, which words do they actually need to understand the story?" It optimizes the stretching based on how the model actually processes information, not just how the numbers look on the page.

3. The Result: A Smaller, Smarter Library

After stretching the important parts, the editor (SVD) cuts off the bottom. Because SigmaScale stretched the important bits first, the "cut" happens in a place where less valuable information is stored.

The Outcome: The resulting "shrunken" model is much smaller, but it keeps the "intrinsic rank" (the core intelligence) much better than the old methods.

What the Experiments Showed

The researchers tested this on Llama 3.1 (8B) and Qwen3-8B. They compared SigmaScale to other top-tier methods (SVD-LLM and ASVD+).

Mild Compression (Keeping 90% of the size): SigmaScale was a clear winner. It kept the model's "perplexity" (a measure of how confused the model is) much lower than the others. It was like keeping 90% of the library but somehow keeping 95% of the intelligence.
Moderate Compression (Keeping 75% of the size): SigmaScale still performed very well, often beating the competition on specific logic and reasoning tests.
Extreme Compression (Keeping 50% of the size): Here, SigmaScale struggled, just like the other methods. The paper admits that if you cut the library down to half its size, even a smart editor can't save the model from losing its mind. It's not a magic solution for extreme shrinking.

The "Fine-Tuning" Step

After shrinking the model, it's a bit wobbly. The researchers also tested two ways to steady it back up:

Supervised Fine-Tuning: Teaching the shrunken model with new examples.
Knowledge Distillation (KD): Having a giant, un-shrunken "Teacher" model guide the small "Student" model.

The Finding: Surprisingly, for this specific method, the "Teacher" (KD) didn't help much more than just teaching the student directly. Both methods worked about the same.

The Bottom Line

SigmaScale is a new technique that makes shrinking AI models smarter. Instead of just chopping off the bottom of the data, it first learns how to rearrange the data so the most important parts are saved.

Best for: Situations where you need to save space but can't afford to lose too much intelligence (mild to moderate compression).
Not for: Trying to shrink a model down to a tiny size (extreme compression), where the method breaks down.
Key Takeaway: It's a flexible, learned approach that adapts to the specific shape of the model's brain, rather than using a one-size-fits-all formula.

Technical Summary: SigmaScale

Problem Statement

Large Language Models (LLMs) have achieved remarkable performance through scaling parameters to tens or hundreds of billions, but this growth creates significant barriers regarding technological accessibility and environmental impact due to high inference power consumption. While model compression techniques such as quantization, pruning, and knowledge distillation exist, many require specialized hardware or complex retraining pipelines. Low-rank decomposition, specifically Truncated Singular Value Decomposition (SVD), offers a hardware-agnostic alternative by approximating weight matrices $W$ as the product of lower-rank matrices.

However, standard SVD faces two primary limitations in the context of LLMs:

Analytical Suboptimality: The Eckart–Young–Mirsky theorem provides the optimal analytical solution for minimizing the Frobenius norm $||W - W'||_F$ . Yet, this structural minimization often performs poorly on neural network weights because it ignores activation outliers and the functional behavior of the network.
Optimization Challenges: Naïve SVD is computationally expensive ( $O(n^3)$ ) and possesses unstable derivatives, making it difficult to integrate into gradient-based optimization routines. Furthermore, previous methods attempting to improve SVD via scaling matrices (e.g., to absorb activation outliers) rely on analytical derivations (such as Cholesky decomposition of activation covariance) rather than learning these transformations directly.

Methodology: SigmaScale

The authors propose SigmaScale, a method that learns auxiliary scaling matrices $S$ to aid truncated SVD-based compression. Instead of deriving scaling matrices analytically, SigmaScale optimizes them via gradient descent under an activation-aware loss.

The pipeline consists of three main stages:

Sensitivity Probing: Before compression, the method determines optimal truncation ranks for each layer and module (MLP and attention) to achieve a global target compression ratio while minimizing perplexity increase. This utilizes a binary search algorithm similar to ASVD.
Learned Scaling Matrices:
- For a weight matrix $W \in \mathbb{R}^{m \times n}$ , two vectors $d_r \in \mathbb{R}^m$ and $d_c \in \mathbb{R}^n$ are initialized from a scaled Gaussian distribution.
- These vectors define positive diagonal scaling matrices via exponentiation: $S_r = \text{diag}(\exp(d_r))$ and $S_c = \text{diag}(\exp(d_c))$ .
- The weight matrix is scaled ( $S_r W S_c$ ), truncated SVD is performed to obtain rank- $k$ factors, and inverse scaling is applied to reconstruct the compressed matrix $W'$ .
- The parameters $d_r$ and $d_c$ are optimized to minimize an activation-aware loss function:
  $L_F = \frac{1}{mn} ||WX - W'X||_F^2$
  where $X$ represents activations from a calibration set. This ensures the compression preserves the model's functionality rather than just its structure.
- Note: To handle the unstable derivative of SVD, the training loop skips update steps where singular value differences approach zero to prevent NaN values.
Post-Compression Fine-Tuning: After compression, the model undergoes fine-tuning to realign weights. The authors compare Supervised Fine-Tuning (SFT) against Knowledge Distillation (KD) using an uncompressed teacher model to recover performance.

Key Contributions

Learning Scaling Parameters: To the best of the authors' knowledge, this is the first work to explore learning the parameters of row- and column-wise scaling matrices $S$ specifically to improve SVD compression performance, moving away from analytical derivations.
Performance Recovery Analysis: The paper provides a comparative analysis of KD versus SFT for performance recovery in the context of SVD-based compression, utilizing varied post-compression datasets.
Custom Dataset: The authors introduce a custom variant of the Alpaca dataset, generated using Llama 3.1-8B Instruction outputs, to introduce variance for weight realignment experiments.

Results and Analysis

Experiments were conducted on Llama 3.1 8B Instruct and Qwen3-8B models across various compression ratios (0.90x, 0.75x, 0.50x retention).

Competitiveness: SigmaScale is competitive with state-of-the-art SVD-based methods (SVD-LLM and ASVD+).
- At 0.90x retention, SigmaScale substantially improves perplexity over SVD-LLM for both models while recovering most zero-shot benchmark performance.
- At 0.75x retention, SigmaScale generally improves several zero-shot benchmarks, though perplexity gains are marginal.
- At 0.50x retention, performance degrades sharply, particularly for Llama 3.1. The authors note that the method is most effective when the retained subspace is large enough for learned scaling to compensate for discarded singular directions; it is not a complete solution for extreme low-rank compression.
KD vs. SFT: Contrary to some prior literature suggesting KD is superior for recovery, the authors' results do not show major improvements of KD over SFT for SigmaScale in their specific setup.
Intrinsic Rank Reduction: The authors analyze the "effective rank entropy" of the singular values ( $\Sigma$ ) during optimization. They observe a strong correlation (approx. 0.81–0.92) between the reduction in compression loss and the reduction in effective rank entropy. This suggests that learned scaling lowers the effective intrinsic rank of weight matrices, aligning them better with the compression constraints.

Significance and Limitations

The paper positions SigmaScale as a valid option for applications requiring reduced LLM inference costs, particularly in mild-to-moderate compression regimes. By adapting to the structure of individual model weights through learned, activation-aware transformations, it offers a more flexible route to low-rank compression than analytical methods.

The authors explicitly acknowledge several limitations:

Computational Cost: The method still requires computing SVD at every update step during scaling matrix training, retaining the $O(n^3)$ complexity, and does not explore faster approximate SVD alternatives.
Extreme Compression: The method degrades significantly under aggressive compression (e.g., 0.50x), indicating it is a mechanism for improving truncation quality in the retained-rank regime rather than a solution for extreme compression.
Evaluation Scope: Evaluation is limited to perplexity and specific zero-shot benchmarks; effects on long-form generation or coding tasks were not explored.
Calibration Data: The robustness to different calibration distributions is not formally verified, and the use of Wikitext is noted as a subpar choice made primarily for consistency with prior baselines.