🤖 machine learning

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

SLOPE is a novel pretraining method for Large Language Models that combines double-pruned N:M sparse weights with a "lazy" low-rank adapter strategy applied only in the final 1% of training iterations, achieving significant speedups (up to 1.54×) and memory reductions (up to 0.63×) during both training and inference while maintaining model accuracy comparable to dense models.

Original authors: Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, Maryam Mehri Dehnavi

Published 2025-01-25✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

Original authors: Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, Maryam Mehri Dehnavi

✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The modern world of artificial intelligence is increasingly defined by large language models, vast digital brains trained on enormous libraries of text to understand and generate human language. These systems are powerful, but they come with a heavy price tag: they require immense amounts of computer power to train and vast quantities of memory to run. To make these models faster and cheaper to use, researchers have long tried to simplify them by removing parts of their internal structure that seem less important, a process known as pruning. Imagine a massive library where you decide to remove every book that isn't read often; the building becomes smaller and easier to navigate, but if you remove too many books or the wrong ones, the library loses its ability to answer questions. In the digital realm, this simplification often leads to a loss of accuracy, where the model becomes less reliable. Furthermore, while removing data saves space, the computer still has to do the complex work of checking which parts to keep and which to discard during every step of its learning, often negating the speed benefits.

A team of researchers has developed a new approach called SLOPE to solve this persistent trade-off between speed, memory, and accuracy. Their method allows these massive language models to be trained and used with significantly fewer resources without sacrificing their ability to understand language. The core of their innovation lies in how they handle the model's learning process. Instead of trying to keep the model simple from the very beginning, they allow it to learn using a standard, simplified structure for almost the entire training period. Then, in the final, tiny fraction of the training time, they introduce a small, flexible addition that helps the model recover any lost precision. This addition is not a heavy, full-sized component, but a lightweight, efficient layer that acts like a fine-tuning tool, allowing the model to regain its sharpness without needing to store or process a massive amount of extra data.

The researchers also tackled a specific bottleneck that has slowed down previous attempts at simplifying these models. In standard computer operations, the model must look at its internal data in two different directions: once to predict the next word and again to correct its mistakes. Previous methods of simplification worked well for the first direction but forced the computer to do extra, slow work when looking at the data in the second direction. The new method changes the order of operations so that the simplification works efficiently in both directions simultaneously. This allows the computer to process the data much faster, utilizing specialized hardware designed to handle these specific patterns of zeros and non-zeros. By rearranging how the data is pruned before it is processed in the correction phase, the researchers ensured that the model could learn just as effectively while moving through the data at a much higher speed.

When tested on models ranging from hundreds of millions to tens of billions of parameters, the results were substantial. The new method accelerated the training process by up to 1.25 times and made the final model run up to 1.54 times faster during use. Perhaps even more importantly, it reduced the amount of computer memory required to run these models by more than half in some cases. The researchers found that by adding their lightweight, flexible layer only at the very end of training, the model could achieve accuracy levels very close to those of the much larger, un-simplified versions. This approach avoids the common pitfall where simplified models require a final, expensive phase of retraining with full-sized data to become useful again. Instead, the model remains efficient throughout its entire lifecycle, from the moment it begins learning to the moment it answers a user's question.

The study demonstrates that it is possible to have both efficiency and high performance in artificial intelligence, provided the simplification is applied with a specific strategy. The researchers showed that the timing of when extra capacity is added matters greatly; introducing a small amount of flexibility at the very end of the learning process yields better results than trying to maintain that flexibility throughout the entire journey. This finding challenges the assumption that a model must be either fully complex or fully simple, suggesting instead that a hybrid approach, where the model is mostly simple but receives a targeted boost at the finish line, offers the best of both worlds. The work provides a clear path forward for making advanced artificial intelligence more accessible, faster, and less demanding on the hardware that powers it.

Technical Summary: SLOPE – Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

1. Problem Statement

Large Language Models (LLMs) require substantial computational resources, memory, and communication bandwidth for pretraining and inference. While model sparsity (pruning less important parts) offers a pathway to reduce these overheads, existing approaches face significant trade-offs:

Unstructured Sparsity: Removes elements from arbitrary locations but lacks hardware/software support for acceleration.
Structured Sparsity (N:M): Imposes constraints (e.g., at most $N$ zeros out of $M$ consecutive elements) to leverage hardware like NVIDIA's sparse tensor cores. However, the limited choice of sparsity patterns often leads to accuracy degradation compared to dense models.
Adaptive Methods: Dynamic mask adjustment improves accuracy but incurs high overheads (storing dense weights, searching for masks) and wastes computation on weights that will eventually be pruned.
Sparse-Dense Hybrid Approaches: Methods like Sparse-Dense Pretraining (FST) compensate for accuracy loss by switching to dense fine-tuning. However, this negates the memory and compute benefits of sparsity during the inference phase, as the final model remains dense.

The core challenge is to achieve a sparse model that is both accurate and efficient during both training and inference, without the overhead of dynamic mask searching or the memory cost of a final dense stage.

2. Methodology

The authors propose SLOPE (Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining), a method designed to improve the accuracy of sparsely pretrained models while accelerating training and inference. The methodology consists of three primary components:

A. Double-Pruned Backward Pass

Standard N:M sparsity accelerates the forward pass but fails in the backward pass because the weight matrix is transposed, destroying the row-wise N:M structure required for hardware acceleration.

Approach: Instead of enforcing the transposed weight matrix to be N:M sparse (which limits patterns and reduces accuracy), SLOPE first transposes the already sparsified N:M weight matrix and then imposes a second round of N:M sparsity.
Mechanism:
- Forward Pass: Uses a row-wise pruned weight matrix ( $W^R_i$ ).
- Backward Pass: Uses a double-pruned weight matrix ( $W^{R,C}_i$ ), where both row and column dimensions are pruned.
Theoretical Guarantee: The authors prove that this "lossy" materialization of pruned weights (where the backward pass uses a different sparsity mask than the forward pass) still satisfies convergence properties. Theorem 2.2 demonstrates that the dynamic alteration of the column-wise mask does not detrimentally impact optimizer convergence, provided the mask is fixed after initialization.

B. Lazy Low-Rank Adapters

To mitigate the accuracy loss inherent in aggressive pruning, SLOPE introduces low-rank adapters ( $L$ and $R$ ) to the weight decomposition: $W_{dense} = W_{sparse} + LR$ .

Timing: Unlike standard LoRA which is often used throughout training, SLOPE introduces these adapters only during the final 1% of pretraining iterations.
Rationale: Empirical results show that low-rank adapters converge significantly faster than sparse weights due to their lower parameter count. Adding them late in training allows the model to recover capacity and accuracy without incurring the computational cost of maintaining them for the entire pretraining duration.
Inference: The adapters are retained during inference, allowing the model to maintain dense-like capabilities without storing a full dense weight matrix.

C. Optimized CUDA Kernels

To realize the theoretical speedups, the authors developed highly optimized CUDA kernels:

Tiling: They tile upsample tensors to ensure they fit the performance sweet spots of the cuSPARSELt backend, preventing speedup degradation for large hidden dimensions.
Fused Operations: They combine sparse matrix multiplication (SpMM) with low-rank adapter computations into single kernels. This reduces kernel launch overhead and improves arithmetic intensity by concatenating downsample tensors with sparse weights and using fused multiply-add operations.

3. Key Contributions

Double-Pruned Backward Pass: A novel formulation that transposes an N:M sparse matrix before applying a second N:M sparsity mask. This enables accelerated sparse backward passes without the accuracy penalties associated with searching for transposable masks.
Lazy Low-Rank Adapters: A strategy to inject low-rank adapters only in the final 1% of training. This improves model capacity and convergence speed with minimal overhead, avoiding the need for a dense fine-tuning stage.
System Optimization: Joint optimization of NVIDIA 2:4 sparse kernels and low-rank calls through efficient tiling and scheduling, resulting in significant end-to-end speedups.

4. Experimental Results

The authors evaluated SLOPE on various LLMs (OPT, LLaMA-3, Mistral) and smaller models (GPT-2, BERT).

Speedup:
- Pretraining: Achieved up to 1.25× speedup (e.g., OPT-33B).
- Inference: Achieved up to 1.54× speedup (e.g., OPT-66B).
- SLOPE consistently outperformed the state-of-the-art FST method, which showed negligible inference speedup due to its requirement for a dense final stage.
Memory Footprint:
- Training: Reduced memory usage by up to 0.63× (approx. 68% reduction) by storing only non-zero weights, indices, and reduced optimizer states.
- Inference: Reduced memory usage by up to 0.61× (approx. 54% reduction).
Accuracy:
- On GPT-2 (Small/Large) pretraining, SLOPE achieved lower perplexity and higher zero-shot accuracy compared to Wanda and Extended SR-STE.
- On BERT-Large, SLOPE with low-rank adapters recovered performance close to dense models on SQuAD and GLUE tasks, whereas static sparse methods showed larger gaps.
- Experiments indicated that the first 12 blocks of a model are more sensitive to sparsity than the later blocks, suggesting that aggressive sparsity can be applied to the latter layers.

5. Significance and Claims

The paper claims that SLOPE successfully bridges the gap between the efficiency of sparse models and the accuracy of dense models. By combining a theoretically grounded double-pruned backward pass with a "lazy" low-rank adapter strategy, SLOPE eliminates the need for a dense fine-tuning stage that typically negates the benefits of sparse pretraining.

The authors emphasize that their method provides a practical solution for training and deploying large-scale LLMs with billions of parameters, offering substantial reductions in both training time and memory footprint while maintaining competitive model quality. The work demonstrates that it is possible to achieve accelerated sparse training and inference simultaneously without sacrificing the model's final performance capabilities.