🤖 machine learning

Learning Fine-grained Parameter Sharing via Sparse Tensor Decomposition

This paper introduces Fine-grained Parameter Sharing (FiPS), a unified framework that compresses transformer MLPs by jointly optimizing cross-block parameter sharing, low-rank factorization, and sparsity, achieving significant model size reduction with minimal accuracy loss on both Vision Transformers and Large Language Models.

Original authors: Cem Üyük, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cem Üyük, Mike Lasby, Mohamed Yassin, Utku Evci, Yani Ioannou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of books (a large AI model) that is incredibly smart but takes up so much space that it won't fit on a regular bookshelf (a phone or small computer). You want to shrink the library down without losing the stories inside.

For a long time, scientists tried to compress these libraries by just cutting out pages (pruning) or writing the books in a smaller font (quantization). But this paper introduces a new, clever strategy called FiPS (Fine-grained Parameter Sharing).

Here is how FiPS works, explained through simple analogies:

1. The Problem: Too Many Identical Rooms

Think of a Transformer model (like the ones powering modern AI) as a giant hotel with many identical floors. On every floor, there is a "kitchen" (called an MLP layer) where the cooking happens.

The Old Way: Usually, every floor has its own unique set of 100 chefs, each with their own unique recipes. Even though the floors look the same, the chefs don't talk to each other. This wastes a lot of space.
The FiPS Idea: FiPS says, "Why does every floor need 100 unique chefs? Let's hire a Shared Master Chef (a shared basis) who knows the core recipes, and then have a few Local Assistants (sparse projection matrices) on each floor who tweak those recipes slightly for that specific floor."

2. The Secret Sauce: The "Shared Master Chef" and "Sparse Assistants"

FiPS does three things at once to shrink the model:

The Shared Master Chef (Low-Rank Factorization): Instead of 100 unique chefs per floor, FiPS creates one "Master Chef" who knows the fundamental techniques. This chef is shared across a group of floors.
The Sparse Assistants (Sparsity): The Local Assistants on each floor don't need to know every recipe the Master Chef knows. They only need to know a few specific ones to make the dish for that floor. FiPS forces these assistants to be "sparse," meaning they only keep the essential connections and ignore the rest. It's like giving an assistant a menu with only 3 items instead of 100.
The Teamwork (Cross-Block Sharing): FiPS groups these floors together. It looks at the Master Chef and the Assistants for a whole group of floors and figures out the best way to share the workload so the whole hotel runs efficiently.

3. How They Built It (The Construction Process)

The researchers didn't just guess; they used a mathematical tool called SVD (Singular Value Decomposition) to find the "Master Chef" automatically.

Imagine taking all the recipes from a group of floors, mixing them together, and finding the most common, essential ingredients.
They then assigned these ingredients to the Master Chef.
Finally, they trained the Local Assistants to use only the specific ingredients they needed, throwing away the rest (pruning).

4. The Results: Smaller, Faster, and Still Smart

The paper tested this on two types of AI:

Vision Transformers (ViTs): These are AIs that look at pictures.
- Result: They shrunk the model by up to 33% (and up to 57% with a little extra tuning) without the AI forgetting how to recognize objects. It's like shrinking a suitcase by a third but still fitting all your clothes.
Large Language Models (LLMs): These are AIs that write and talk.
- Result: They shrunk these models by 20% and made them smarter than other shrinking methods.
- The "Magic Trick": When they combined FiPS with a technique called "Quantization" (writing numbers in a very compact code), they achieved an 8x compression (making the model 8 times smaller) while keeping the AI's language skills much better than if they had just used compression alone.

5. Why This Matters

The paper claims that FiPS is a practical, "plug-and-play" way to make big AI models small enough to run on devices like phones or laptops without losing their intelligence. It proves that by sharing the "brainpower" (parameters) between different parts of the AI and being very selective about what information is kept (sparsity), we can build efficient AI that doesn't need a supercomputer to run.

In short: FiPS is like realizing that instead of every room in a house needing its own full set of tools, you can have one shared toolbox in the hallway and just a few specific tools in each room. The house works just as well, but it takes up much less space.

Technical Summary: Fine-grained Parameter Sharing via Sparse Tensor Decomposition (FiPS)

Problem Statement

Large neural networks, particularly Transformers, achieve state-of-the-art performance but face significant barriers to deployment on resource-constrained devices due to their sheer size. While various compression techniques exist (e.g., quantization, distillation, tensor decomposition), cross-layer parameter sharing remains relatively unexplored for Transformer models. Existing methods often share entire blocks (e.g., ALBERT), which constrains representational capacity, or rely on dense coefficients for sharing, which limits compression efficiency. There is a need for a method that reduces parameter counts significantly while maintaining high representational flexibility and minimal accuracy loss.

Methodology: Fine-grained Parameter Sharing (FiPS)

FiPS is a unified framework designed to compress Transformer Multi-Layer Perceptrons (MLPs) by combining cross-block parameter sharing, low-rank factorization, and sparsity into a single optimization process.

Core Mechanism

The method operates on the observation that Transformer MLPs consist of repeated, homogeneous blocks with identically shaped fully connected (FC) layers. Instead of learning independent weights for each layer, FiPS:

Concatenates weight matrices ( $W_1, \dots, W_N$ ) from a group of $N$ transformer blocks along their output dimension (the "long axis").
Factorizes this concatenated tensor into a shared basis $U \in \mathbb{R}^{d \times r}$ and layer-specific sparse projection matrices $V_i \in \mathbb{R}^{r \times p}$ .
Reconstructs the original weights as $W'_i = U V_i$ .

Key Technical Components

Initialization: The shared basis $U$ and projection matrices $V_i$ are initialized via Truncated Singular Value Decomposition (SVD) of the concatenated weights. If the target rank $r$ exceeds the model dimension $d$ (due to high parameter budgets), a "hybrid initialization" is used: new dimensions in $U$ are set to zero, and corresponding dimensions in $V$ are derived from top singular vectors scaled by a dampening factor $\tau$ to allow gradual learning.
Sparsity Induction: A critical insight of the paper is that enforcing sparsity on the projection matrices $V$ is essential. Unlike methods using dense coefficients, FiPS prunes low-magnitude values in $V$ . The optimal sparsity level was found to be around 75%, particularly when applied to the larger factor matrix.
Optimization Strategy:
1. Shared Initialization: SVD decomposition of concatenated weights.
2. Local Error Minimization: Using a small calibration dataset, $U$ and $V_i$ are optimized to minimize the $\ell_2$ reconstruction error between original and compressed activations. Sparsity is enforced via Gradual Magnitude Pruning (GMP).
3. Global Error Minimization (Optional): An end-to-end fine-tuning stage using dynamic sparse training (RigL) to recover performance at higher compression ratios.
Grouping Strategy: The paper identifies that adjacent layers share the most redundancy. Optimal performance is achieved by grouping consecutive blocks (e.g., groups of 4 for DeiT-B) and using a "tapered" grouping strategy for LLMs (smaller groups at the network extremes, larger in the middle).

Key Contributions

Systematic Analysis of Cross-Block Sharing: The authors systematically explored sharing granularity, concatenation schemes, and sparsity patterns, identifying that long-axis concatenation and sparsity on the projection factors yield the lowest reconstruction error.
The FiPS Algorithm: A novel algorithm that initializes via SVD and jointly optimizes a shared basis and sparse, layer-specific projections. It uniquely combines low-rank factorization with sparsity on the projection factors rather than the basis.
State-of-the-Art Compression Results:
- Vision Transformers (ViTs): Compresses DeiT-B and Swin-L by up to 33% with less than 1% top-1 accuracy loss on ImageNet-1k. With fine-tuning, compression reaches 57%.
- Large Language Models (LLMs): Compresses Llama-7B and Llama-3.1-8B by up to 20%, outperforming existing SVD-based baselines (ASVD, SVD-LLM, SVD-LLM V2) in perplexity and downstream benchmarks.
- Quantization Synergy: When combined with Quantization-Aware Training (QAT), 3-bit FiPS on Gemma-2-2B achieves the same 8× compression as 2-bit QAT alone but with significantly lower perplexity (35.43 vs. 41.86).
Hardware Efficiency: The method supports structured sparsity (e.g., 2:4 patterns), enabling measurable inference speedups (1.31× on NVIDIA A4000) and memory reductions (0.79× peak VRAM) without significant accuracy degradation.

Results

ViT Performance: On ImageNet-1k, FiPS consistently outperforms baselines like Adaptive Atomic Feature Mimicking (AAFM) and Global Feature Mimicking (GFM) across various parameter budgets (10% to 75%). For instance, at a 40% parameter budget, FiPS achieves 81.69% accuracy on DeiT-B, surpassing GFM (81.28%) despite GFM's higher computational cost.
LLM Performance: On WikiText-2 and C4 datasets, FiPS achieves the lowest perplexity among factorization-based methods at 20% compression. It also demonstrates superior transfer learning capabilities on CIFAR-100, Flowers102, and other vision benchmarks compared to GFM.
Ablation Studies: The paper confirms that:
- Sparsity on the projection matrix $V$ is critical; dense baselines collapse at low parameter budgets.
- SVD initialization outperforms random initialization.
- Global pruning (allocating parameters adaptively across layers) yields better results than local pruning.
- A 75% sparsity level is optimal for the projection matrices.

Significance and Claims

The paper establishes fine-grained parameter sharing as a practical and effective approach for Transformer MLP compression. The authors claim that FiPS is orthogonal to other compression techniques like Quantization-Aware Training (QAT) and LoRA, meaning it can be combined with them for further gains.

The significance lies in demonstrating that sharing parameters across layers does not require sacrificing representational capacity if done with a shared basis and sparse, learnable projections. This approach offers a competitive alternative to existing strategies, achieving substantial compression (up to 57% on ViTs and 20% on LLMs) with minimal performance loss, thereby facilitating the deployment of large models on resource-constrained devices. The method is particularly notable for not relying on distillation or auxiliary supervision signals, relying instead on block-wise reconstruction error minimization.