💬 NLP

SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

SpenseGPT introduces a practical one-shot pruning method that utilizes a hybrid sparse-dense weight format to achieve up to 1.2x end-to-end LLM decoding speedup on B200 GPUs while maintaining model accuracy, overcoming the limitations of strict semi-structured sparsity without requiring custom compiler support.

Original authors: Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaeseong Lee, Seung-won Hwang, Samyam Rajbhandari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a Large Language Model, or LLM) that helps computers write code, solve math problems, and follow instructions. To make this library run fast on modern computers, engineers have been trying to "prune" it—removing unnecessary books to make it lighter and quicker.

However, there's a catch. The fastest way to read these books on new computer chips (like NVIDIA's B200) requires a very strict rule: For every four books on a shelf, exactly two must be empty. This is called "2:4 sparsity."

The Problem: The Strict Rule

The problem with this strict rule is that it's like trying to pack a suitcase where you must leave exactly half the space empty. If you just randomly throw away half the books to fit the rule, you lose important information, and the library stops working well. The computer becomes fast, but the answers it gives become silly or wrong.

Other researchers tried to fix this by making the rules more flexible, but their solutions were like building a custom, expensive engine that only works with a specific type of fuel, or they added so much extra work (overhead) that the speed gain disappeared.

The Solution: Spense (The Hybrid Bookshelf)

The authors of this paper propose a new method called Spense. Instead of forcing the entire library to follow the strict "two empty, two full" rule, they split the shelves into two zones:

The Sparse Zone: Here, they follow the strict rule (2 out of 4 books are removed). This zone gets the speed boost from the computer's special hardware.
The Dense Zone: Here, they keep all the books. No books are removed. This zone preserves the most important information.

Think of it like a hybrid car. The "Sparse Zone" is the electric motor (super efficient for cruising), and the "Dense Zone" is the gas engine (powerful when you need to climb a hill). By combining them, you get the best of both worlds: speed without losing power.

The Secret Sauce: Choosing What to Keep

The tricky part is deciding which books go into the "Dense Zone" and which get thrown out. The authors found that this choice is critical. If you pick the wrong books to keep, the library still fails.

They developed two strategies to make this choice:

SpenseGPT (The Simple Strategy): Imagine the books are arranged in a line. This method says, "Let's keep the last 25% of the books on the shelf as they are (Dense), and prune the first 75% (Sparse)." It turns out this simple "cut at the end" approach works surprisingly well because of how the pruning math is calculated.
SpenseGPT+ (The Smart Strategy): This is like hiring a librarian who reads every book to see which ones are most important. It calculates a "score" for every book based on how much it contributes to the final answer. It keeps the highest-scoring books in the Dense Zone, ensuring the most critical knowledge is never lost.

The Results: Fast and Accurate

The team tested this on two very large, smart models (Qwen3-32B and Seed-OSS-36B) using the latest super-fast computer chips (B200 GPUs).

Speed: They achieved a 1.2x speedup in real-world usage. This means the computer can generate answers noticeably faster than before.
Accuracy: Unlike other methods that made the models "dumb," Spense kept the models smart. They performed just as well on difficult tasks like math reasoning, coding, and following instructions as the original, unpruned models.
Practicality: Crucially, this method doesn't require building new, custom computer software (compilers). It works with the standard, highly optimized tools that already exist on these chips.

Summary

In short, the paper introduces a way to make AI models faster without making them less smart. Instead of forcing the whole model to be "half-empty" (which breaks it), they created a hybrid system: part of the model is stripped down for speed, and the most important part is kept full for accuracy. This allows modern computers to run these giant AI brains faster than ever before, using tools that are already available.

Technical Summary: SpenseGPT: Practical One-shot Pruning for Sparse and Dense GEMMs in LLM Inference

1. Problem Statement

Semi-structured 2:4 sparsity (where two out of every four consecutive weights are pruned to zero) is widely supported by modern accelerators (e.g., NVIDIA B200, AMD GPUs, Meta MTIAs) and offers a theoretical 2× speedup by allowing hardware to skip zero-weight computations. However, practical adoption for Large Language Model (LLM) inference faces two primary hurdles:

Strict Accuracy Degradation: The rigid 50% sparsity constraint often leads to significant accuracy loss in post-training pruning, particularly on reasoning and instruction-following tasks, unless extensive retraining is performed.
Impracticality of Existing Relaxations: Recent works attempting to relax the 2:4 constraint (e.g., PATCH, SlideSparse) suffer from critical limitations. PATCH requires specialized compiler support (Triton) not yet widely available, while SlideSparse necessitates input activation expansion, introducing runtime overheads that negate end-to-end speedups, especially for large sequence lengths ( $M$ ).

2. Methodology

The paper proposes Spense, a practical hybrid sparse-dense format, and SpenseGPT, a one-shot post-training pruning method built upon it.

A. The Spense Format
Instead of enforcing 2:4 sparsity across the entire weight matrix, Spense splits each weight matrix into two contiguous regions:

A dense region (retaining $p\%$ of weights).
A 2:4 sparse region (pruned with standard 2:4 sparsity).

This design leverages existing, highly optimized GEMM libraries (e.g., cuSPARSELt for sparse and cuBLASLt for dense on NVIDIA GPUs) without requiring custom compilers or input activation expansion. The authors implement two variants:

Output-split: Splits the output tensor, using sparse GEMM for one part and dense GEMM for the other.
Input-split: Splits the input tensor, processing the first part with sparse GEMM and the second with dense GEMM (with accumulation).
For GLU-variant MLPs, the authors use output-split for $W_{up}$ and $W_{gate}$ , and input-split for $W_{down}$ .

B. Dense Index Selection Strategies
A critical insight is that which intermediate indices are kept dense significantly impacts model quality. The paper identifies that intermediate indices in GLU-style MLPs can be permuted without changing the mathematical output, allowing the selection of specific indices to be dense before rearranging them into a contiguous block for execution. Two strategies are proposed:

SpenseGPT (Simple Strategy): Selects the last $p\%$ of intermediate indices to remain dense. This aligns with the column-wise processing order of SparseGPT, where processed columns are frozen, and reconstruction errors are compensated by updating remaining columns to the right. Keeping dense columns on the right provides a "compensation buffer."
SpenseGPT+ (Loss-Aware Strategy): Selects dense indices based on an estimated reconstruction loss. Inspired by Wanda, it computes an importance score for each intermediate index by aggregating contributions across the gate, up, and down projections. The score combines weight magnitude and input activation magnitude ( $S_{ij} = |W_{ij}|\sqrt{H_{j,j}}$ ). The top-scoring indices are selected to remain dense, then permuted to the contiguous region before applying SparseGPT to the remaining sparse regions.

3. Key Contributions

Spense Format: A hybrid sparse-dense format that relaxes strict 2:4 sparsity while maintaining compatibility with existing high-performance sparse and dense GEMM libraries, avoiding custom compiler support and activation expansion overheads.
Importance of Dense Selection: Demonstration that the selection of dense indices is a critical factor in maintaining model quality, with naive selection leading to severe performance drops.
Pruning Strategies: Introduction of SpenseGPT (contiguous tail selection) and SpenseGPT+ (reconstruction-loss-based selection) for one-shot pruning.
Real-World Speedup: The first demonstration of one-shot pruning achieving real-world end-to-end LLM decoding speedups on recent GPUs (B200) using semi-structured sparse tensor cores while preserving model quality.

4. Experimental Results

The methods were evaluated on Qwen3-32B and Seed-OSS-36B using FP8 precision on B200 GPUs with the VLLM framework. Benchmarks included AIME (math), GPQA (knowledge), IFEval (instruction following), and LiveCodeBench (coding).

Accuracy Preservation:
- Strict 2:4 pruning (50% sparsity) caused significant accuracy drops (e.g., Seed-OSS-36B average score dropped from 73.86 to 65.18 with SparseGPT).
- SpenseGPT+ at 25% sparsity achieved an average score of 73.12 on Seed-OSS-36B, nearly matching the dense baseline (73.86).
- On Qwen3-32B, SpenseGPT+ at 25% sparsity reached 70.76, close to the dense baseline of 71.68.
- Comparisons showed that SpenseGPT+ significantly outperformed SpenseGPT and other baselines (HyperPrune, Wanda) by correctly selecting dense indices.
End-to-End Speedup:
- Spense achieved up to 1.2× end-to-end decoding speedup over the dense baseline on B200 GPUs.
- In contrast, SlideSparse failed to provide significant speedup due to non-GEMM overheads from input activation expansion.
Efficiency:
- Pruning 32B–36B models took approximately 1.33–2.15 hours on a single B200 GPU, comparable to SparseGPT and much faster than HyperPrune. ProxSparse ran out of memory on these models.

5. Significance and Claims

The paper claims to present the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedups derived from semi-structured sparse tensor cores on recent hardware (specifically B200 GPUs) while maintaining model quality.

The authors emphasize that their approach bridges the gap between theoretical hardware acceleration and practical deployment. By avoiding the need for specialized compilers or input expansion, Spense enables immediate utilization of existing optimized libraries. The work highlights that simply relaxing the sparsity constraint is insufficient; the strategy for selecting which weights remain dense is paramount to preserving the model's reasoning and instruction-following capabilities.

Limitations Acknowledged:

Speedup is dependent on the efficiency of underlying GEMM libraries and varies with batch size, sequence length, and hardware.
SpenseGPT+ relies on calibration data; if the deployment distribution differs significantly from the calibration set, the selected dense indices may not be optimal.