💬 NLP

Sparser, Faster, Lighter Transformer Language Models

This paper introduces a new sparse packing format and optimized CUDA kernels that leverage unstructured sparsity in LLM feedforward layers to achieve over 99% sparsity with negligible performance loss, resulting in significant improvements in throughput, energy efficiency, and memory usage for large-scale models.

Original authors: Edoardo Cetin, Stefano Peluchetti, Emilio Castillo, Akira Naruse, Mana Murakami, Llion Jones

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC0 1.0

Original authors: Edoardo Cetin, Stefano Peluchetti, Emilio Castillo, Akira Naruse, Mana Murakami, Llion Jones

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a massive, high-end library (a Large Language Model or LLM) that can answer any question, write poetry, or solve complex problems. To do this, the library has millions of librarians (neurons) working in huge rooms (layers).

The Problem: The "Busy" Library
Currently, these libraries are incredibly expensive to run. Why? Because for every single question you ask, every single librarian in the room wakes up, grabs a book, and starts reading, even if 99% of them don't actually know the answer or need to do anything.

The Cost: This wastes massive amounts of electricity, takes up too much computer memory, and makes the library slow because everyone is moving at once, creating traffic jams.
The Paradox: You might think, "If 99% of the librarians are doing nothing, why not just tell them to go home?" The problem is that telling them to go home (making the system "sparse") usually creates more chaos. The computer has to constantly check a list of who is working and who isn't, which slows things down even more than just letting everyone work.

The Solution: The "Smart" Library
This paper introduces a new way to run the library that is sparser, faster, and lighter. The authors (from Sakana AI and NVIDIA) built a new system that lets the library work smarter, not harder.

Here is how they did it, using simple analogies:

1. The "Lazy" Librarian Trick (L1 Regularization)

First, they needed to train the librarians to actually be lazy. They added a tiny "penalty" to the training process (called L1 regularization).

The Analogy: Imagine a manager telling the librarians, "If you don't need to open a book to answer the question, you get a bonus."
The Result: The librarians quickly learned that they only needed to wake up for the specific questions they could actually answer. Suddenly, 99% of the librarians stayed asleep for most questions. The library became incredibly sparse.

2. The New "Smart Filing System" (TwELL Format)

Here was the big hurdle: Even though 99% of the librarians were asleep, the old computer systems were terrible at handling "who is awake." They tried to organize the sleeping librarians in a way that made the computer trip over its own feet.

The Analogy: Imagine trying to organize a party where 99% of guests are sleeping. The old method was to write down the name of every guest (sleeping or awake) on a giant list, then cross out the sleepers. This takes forever.
The Innovation: The authors invented a new filing system called TwELL (Tile-wise ELLPACK).
- Instead of one giant list, they broke the library into small, manageable "tiles" (like small rooms).
- Inside each small room, they only wrote down the names of the librarians who were actually awake.
- Because the computer is built to process these small "tiles" perfectly, it can now skip the sleeping librarians instantly without getting confused. It's like having a bouncer at every small room who only lets the awake people in, so the main hall never gets clogged.

3. The "Express Lane" (Fused Kernels)

Usually, in a computer, you have to do Step A (find the awake librarians), then Step B (get their books), then Step C (write the answer). This involves a lot of walking back and forth between the computer's brain (memory) and its muscle (processor).

The Innovation: The authors built a new "Express Lane" (called Fused Kernels).
The Analogy: Instead of the librarian walking to the shelf, walking back to the desk, and then walking to the printer, they built a conveyor belt that does all three steps in one smooth motion. The computer grabs the awake librarian, gets the book, and writes the answer all in one go, without ever stopping to check a list.

The Results: Why This Matters

By combining the "Lazy Librarian" training with the "Smart Filing System" and the "Express Lane," the authors achieved amazing results:

Speed: The library answers questions 20% faster.
Energy: It uses 15-17% less electricity.
Memory: It needs 20-28% less computer memory, meaning you can run these huge models on smaller, cheaper computers.
Scalability: The bigger the library gets, the more these savings add up.

The Bottom Line

This paper proves that you don't need to build a new type of library to make it efficient. You just need to teach the existing librarians to be lazy and give them a better way to organize their work.

They are releasing all their tools (the "filing system" and "conveyor belts") for free so that everyone can build faster, cheaper, and greener AI in the future. It's a win for the environment and a win for anyone who wants to use AI without needing a supercomputer in their basement.

1. Problem Statement

Despite the success of Large Language Models (LLMs), their scaling is hindered by massive computational costs, memory requirements, and energy consumption. While unstructured sparsity (where individual weights or activations are zero) offers a theoretical path to efficiency, it has historically failed to deliver practical speedups on modern GPUs.

The Paradox: Modern GPUs are heavily optimized for dense matrix operations. Unstructured sparsity introduces irregular memory access patterns and synchronization overheads (e.g., managing indices), often causing sparse kernels to run slower than dense ones.
The Bottleneck: Existing sparse formats (like ELLPACK) require pre-computation or separate kernels to convert dense outputs to sparse formats, breaking the execution pipeline and negating performance gains. Furthermore, training sparse models is difficult due to the unpredictable and non-uniform distribution of non-zero activations, which complicates memory allocation for backpropagation.

2. Methodology

The authors propose a holistic solution combining a new data format, specialized CUDA kernels, and a training recipe to leverage unstructured sparsity in the feed-forward layers (which account for ~80% of FLOPs in large models).

A. New Sparse Data Format: TwELL

The core innovation is Tile-wise ELLPACK (TwELL), a sparse format designed specifically for seamless integration with modern tiled matrix multiplication kernels.

Mechanism: Unlike standard ELLPACK which pads entire rows, TwELL divides columns into horizontal 1D tiles. Within each tile, it stores non-zero values and their indices in a local ELL-based format.
Advantage: TwELL can be materialized directly in the epilogue of a dense matrix multiplication kernel (e.g., the gate projection $xW_g$ ). This eliminates the need for a separate conversion kernel, removing synchronization overheads and global memory reads.
Hybrid Format for Training: To handle the high variance in non-zero counts during training (where some tokens activate hundreds of neurons), the authors introduce a Hybrid Format. It dynamically routes rows into an aggressively compact ELL matrix (for low sparsity rows) or a dense backup matrix (for high sparsity rows), trivializing storage costs for intermediate activations.

B. Kernel Design

The paper introduces custom CUDA kernels for H100 GPUs that leverage Tensor Memory Accelerator (TMA) and Warp Group Matrix Multiply-Accumulate (WGMMA) instructions.

Inference Kernels:
- Gate Projection: Computes $h_g = \text{ReLU}(xW_g)$ and simultaneously packs the output into TwELL format using warp-level atomic operations.
- Fused Up/Down Projection: A single kernel fuses the up-projection ( $xW_u$ ) and down-projection ( $hW_d$ ) operations. It iterates over the sparse TwELL data, performing dot products and scalar-vector multiplications without ever materializing the full dense hidden state in DRAM.
Training Kernels:
- Hybrid Conversion: Efficiently converts TwELL gate activations into the Hybrid format.
- Sparse MatMul: Performs dense-to-hybrid and hybrid-to-dense matrix multiplications, handling the "dense backup" rows using Tensor Cores while processing sparse rows via CUDA cores.
- Backpropagation: Efficiently injects L1 gradients and transposes the hybrid format for gradient computation without expensive dense reconstructions.

C. Training Recipe

Activation Function: Uses ReLU (instead of SiLU/GELU) to induce natural sparsity.
Regularization: Adds a simple L1 regularization term to the loss function to encourage sparsity in the feed-forward activations.
Strategy: The authors demonstrate that mild L1 regularization can induce >99% sparsity with negligible performance degradation.

3. Key Contributions

TwELL Format & Kernels: Introduction of a new sparse packing format and associated CUDA kernels that enable operator fusion (gate projection + packing) and fused inference (up/down projections), minimizing global memory access.
Hybrid Training Algorithm: A novel training pipeline that dynamically manages memory for highly non-uniform sparsity patterns, enabling efficient backpropagation without dense bottlenecks.
Quantitative Validation: Comprehensive empirical evidence showing that unstructured sparsity, when paired with these kernels, yields significant throughput, energy, and memory benefits that scale with model size.
Open Source: Release of all code and kernels to promote adoption.

4. Experimental Results

The authors evaluated models ranging from 0.5B to 2B parameters (trained on FineWeb) with varying L1 regularization coefficients.

Sparsity vs. Performance:
- Mild L1 regularization ( $L1 \approx 2 \times 10^{-5}$ ) achieves >99% sparsity (average non-zeros per token drops from ~5000 to ~29) with negligible impact on downstream task accuracy (e.g., ARC, HellaSwag).
- Performance degradation only becomes visible at extremely high regularization levels where average non-zeros drop below 1.
Efficiency Gains (Inference):
- Speedup: Up to 20.5% faster forward execution for 2B parameter models.
- Energy: Up to 17% reduction in energy per token.
- Scaling: Benefits increase with model scale; larger models see greater speedups due to higher absolute sparsity.
Efficiency Gains (Training):
- Speedup: Up to 21.9% faster training steps.
- Memory: Peak GPU memory usage reduced by >28% for 2B models, enabling larger batch sizes or training on smaller hardware.
Hardware Analysis:
- The benefits are even more pronounced on less powerful hardware (e.g., RTX 6000) compared to H100, suggesting sparsity can lower the barrier to entry for training efficient models.
Sparsity Patterns:
- Sparsity is not uniform: Early layers are less active, while middle layers show a "hump" of activity.
- Tokens with high information content (e.g., specific nouns/verbs) activate more neurons than predictable tokens (e.g., common prepositions), allowing the kernels to leverage inherent data unevenness.

5. Significance and Future Work

Practicality: This work bridges the gap between theoretical sparsity and practical GPU efficiency, proving that unstructured sparsity can be a viable "third axis" for scaling LLMs alongside parameter count and data volume.
Cost Reduction: By reducing memory and energy requirements, this approach makes training and deploying large foundation models more sustainable and accessible.
Future Directions: The authors suggest that dead neuron mitigation (re-initializing inactive weights) could further push sparsity levels higher without performance loss. They also note that their kernels can be applied to fine-tuning existing dense models.

In summary, this paper demonstrates that with the right data structures (TwELL) and hardware-aware kernels, unstructured sparsity can be transformed from a theoretical curiosity into a practical tool for building faster, lighter, and more energy-efficient LLMs.