📊 statistics

Low-Rank Compression of Pretrained Models via Randomized Subspace Iteration

This paper proposes Randomized Subspace Iteration (RSI) as a superior alternative to Randomized SVD for compressing large pretrained models, demonstrating that its multi-power iteration approach effectively handles slowly decaying singular value spectra to achieve near-optimal approximation quality and higher predictive accuracy under aggressive compression.

Original authors: Farhad Pourkamali-Anaraki

Published 2026-04-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Farhad Pourkamali-Anaraki

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly detailed library of knowledge. This library represents a "pretrained model"—a massive AI brain (like the ones used in your phone or on the internet) that has already learned how to recognize cats, write poems, or diagnose diseases. It's brilliant, but it's also huge. It takes up so much space and requires so much electricity to run that it's impossible to carry around in your pocket or run on a small device.

To make this library portable, we need to compress it. We want to shrink the library down without throwing away the most important books.

The Problem: The "Blurry Photocopy"

The standard way to shrink these models is like taking a high-resolution photo and making a low-resolution photocopy. In math terms, this is called Low-Rank Decomposition. It tries to keep only the "most important" patterns in the data and throw away the rest.

However, there's a catch. Modern AI models are so complex that their "importance" isn't neatly organized. It's like trying to sort a pile of sand where the big grains and the tiny grains are mixed together perfectly.

The Old Method (RSVD): The traditional tool used for this (Randomized SVD) is like a sieve with a fixed hole size. If the sand grains are all roughly the same size (a "slow decay" of importance), the sieve lets too much important stuff fall through and keeps too much junk. The result is a blurry, inaccurate copy of the library. The AI starts making silly mistakes.

The Solution: The "Smart Refiner" (RSI)

The author of this paper, Farhad Pourkamali-Anaraki, proposes a new tool called Randomized Subspace Iteration (RSI).

Think of RSI not as a simple sieve, but as a smart, multi-pass filter.

First Pass: It looks at the data and picks out the obvious big patterns (like the sieve).
The Magic Step (Power Iterations): Instead of stopping there, RSI takes the result and runs it through the filter again and again (this is the "iteration" part).
The Result: With each pass, the "big, important" patterns get louder and clearer, while the "small, unimportant" noise gets quieter and quieter. It's like turning up the volume on your favorite song while turning down the static on the radio.

By doing this extra work (which is surprisingly fast on modern computers), RSI creates a crystal-clear, high-quality compressed version of the AI, even when the data is messy.

Why Does This Matter? (The "Prediction" Guarantee)

The paper doesn't just say "it looks better"; it proves mathematically why it works.
Imagine the AI is a weather forecaster.

If you compress the model poorly (using the old sieve), the forecaster might say "70% chance of rain" when it's actually sunny. That's a big error.
The paper shows that if you use the new "Smart Refiner" (RSI), the difference between the original forecast and the compressed forecast is guaranteed to be tiny. The math proves that the "noise" you introduce by shrinking the model won't confuse the AI's final decision.

The Real-World Test

The author tested this on two famous types of AI brains:

VGG19: An older, classic "convolutional" network (like a traditional camera lens).
ViT (Vision Transformer): A modern, state-of-the-art network (like a futuristic digital eye).

The Results:

Speed: The new method is incredibly fast. Compressing a massive layer of the AI took less than a second.
Accuracy: When they squeezed the models down to be very small (aggressive compression), the old method made the AI forget how to recognize things (accuracy dropped to near zero). The new method (RSI) kept the AI smart, maintaining high accuracy even when the model was tiny.

The Big Picture

This paper is like inventing a better compression algorithm for your digital life.

Before: You had to choose between a huge, slow AI that fits on a server, or a tiny, dumb AI that fits on your phone.
Now: With RSI, you can shrink the giant AI down to fit in your pocket without losing its brainpower.

It allows us to take the massive, powerful models trained in data centers and run them efficiently on our everyday devices, making advanced AI accessible to everyone, everywhere.

1. Problem Statement

The rapid proliferation of large-scale pretrained models (e.g., Transformers, VGG) has created a bottleneck for deployment in resource-constrained environments (mobile, edge devices) due to massive parameter counts and memory requirements. While Low-Rank Decomposition via Singular Value Decomposition (SVD) is a principled method for model compression, it faces two critical challenges:

Computational Cost: Exact SVD of large weight matrices ( $O(DC^2)$ complexity) is prohibitively expensive for modern models with high-dimensional layers.
Approximation Quality in Slow-Decay Regimes: Standard Randomized SVD (RSVD) is efficient but suffers when the singular value spectrum of the weight matrix decays slowly. In such cases (common in pretrained models), RSVD fails to accurately capture the dominant subspace, leading to significant approximation errors and degraded predictive performance.

2. Methodology

The paper proposes a two-pronged approach: a theoretical analysis linking approximation error to classification performance, and a new algorithmic method for compression.

A. Theoretical Analysis: Softmax Perturbation

The authors establish a direct mathematical link between the spectral error of the compressed weights and the stability of the model's output probabilities.

Key Insight: They analyze the perturbation of the softmax function under low-rank approximation.
Result: They prove that the deviation in predicted class probabilities ( $\|\hat{p}(x) - p(x)\|_\infty$ ) is bounded by the spectral norm of the weight approximation error ( $\|W - \hat{W}\|_2$ ).
Implication: Minimizing the spectral approximation error is theoretically sufficient to guarantee that the compressed model maintains reliable classification probabilities. This justifies the focus on improving spectral approximation quality over simple parameter reduction.

B. Proposed Algorithm: Randomized Subspace Iteration (RSI)

To overcome the limitations of RSVD in slow-decay spectral regimes, the authors propose Randomized Subspace Iteration (RSI).

Mechanism: RSI incorporates power iterations into the randomized projection process. Instead of a single multiplication ( $W\Omega$ $W Ω$ ), it iteratively applies the matrix and its transpose ( $W$ $W$ and $W^T$ $W^{T}$ ) $q$ $q$ times.
- Mathematically, applying $(WW^T)^{q-1}W$ to a random vector amplifies the contribution of larger singular values ( $s_i^{2q-1}$ ) while suppressing smaller ones.
Algorithm Flow:
1. Draw a random matrix $\Omega$ .
2. Perform $q$ iterations of matrix multiplication ($X = WY$, then $Y = W^T X$ ) with orthonormalization (QR decomposition) at each step.
3. Compute the exact SVD of the resulting small matrix to extract the low-rank factors.
Advantage: By increasing the iteration count $q$ , RSI enhances spectral separation, allowing it to accurately approximate the dominant subspace even when the singular values decay slowly.

3. Key Contributions

Theoretical Guarantee: A formal proof showing that the error in class probabilities is linearly bounded by the spectral approximation error of the weights, providing a rigorous foundation for evaluating compression reliability.
Algorithmic Innovation: The introduction of RSI as a superior alternative to RSVD for pretrained models. It offers a controllable mechanism (via parameter $q$ ) to trade off computational cost for approximation quality.
Empirical Validation: Comprehensive experiments demonstrating that RSI achieves near-optimal approximation quality (normalized error close to 1.0) where RSVD fails (normalized error > 2.0 or 4.0), particularly in aggressive compression scenarios.

4. Experimental Results

The authors evaluated the method on VGG19 (Convolutional) and ViT-B/32 (Transformer) using the Imagenette dataset. Crucially, no fine-tuning or retraining was performed; the compressed models were tested directly as off-the-shelf classifiers.

Single-Layer Analysis:
- VGG19 (4096 $\times$ 25088 layer): RSVD ( $q=1$ ) yielded a normalized error of $\approx 2.0$ . Increasing iterations to $q=4$ reduced the error to $\approx 1.1$ (near-optimal). RSI was 50–76 $\times$ faster than exact SVD.
- ViT (768 $\times$ 3072 layer): RSVD failed significantly (error > 4.0). RSI with $q=3$ or $4$ maintained errors below 1.2, offering an order-of-magnitude speedup over exact SVD.
End-to-End Compression:
- VGG19: At an aggressive compression ratio ( $\alpha=0.2$ , retaining only 20% of parameters), RSVD ( $q=1$ ) caused accuracy to drop to 59.27%. RSI with $q=4$ maintained 78.63% accuracy (close to the original 82.57%).
- ViT: Similar trends were observed. While extreme compression ( $\alpha=0.2$ ) hurt performance, RSI with higher $q$ significantly outperformed RSVD. At $\alpha=0.4$ , RSI achieved 86.37% Top-1 accuracy (vs. 76.17% for RSVD).
- Efficiency: Total compression time for all linear layers in both models was under 1 second on a single A100 GPU.

5. Significance and Future Directions

Practical Deployment: This work enables the deployment of massive pretrained models on edge devices without the computational overhead of retraining or the accuracy loss associated with standard randomized compression.
Complementarity: The method is compatible with LoRA (Low-Rank Adaptation). RSI can compress the backbone weights first, and LoRA can then be applied for fine-tuning, offering a dual benefit of reduced memory footprint and efficient adaptation.
Future Work: The authors suggest extending this to adaptive layer-wise rank selection, applying it to multimodal models, and integrating it into the low-rank adaptation framework for pretrained models.

In summary, the paper demonstrates that Randomized Subspace Iteration (RSI) is a theoretically grounded and empirically superior method for compressing pretrained deep learning models, effectively solving the "slow spectral decay" problem that plagues standard RSVD.