PATCH: Learnable Tile-level Hybrid Sparsity for LLMs
PATCH is a hybrid sparsity framework that partitions LLM weight matrices into learnable tiles, dynamically assigning each tile to be either dense or 2:4 sparse to enable a continuous sparsity ratio between 0% and 50%, thereby achieving superior accuracy and practical hardware-accelerated speedups compared to existing uniform 2:4 pruning methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written by the author. For technical accuracy, refer to the original paper. Read full disclaimer
Large language models are the engines behind a new generation of artificial intelligence, capable of writing stories, answering complex questions, and translating languages with startling fluency. These systems work by processing vast networks of numbers, known as parameters, which are stored in the computer's memory. The more parameters a model has, the smarter it tends to be, but the more expensive and energy-hungry it becomes to run. To make these powerful tools useful on everyday devices or in data centers without burning through resources, scientists have long sought ways to shrink them. One of the most promising methods is pruning, which involves turning off the parts of the network that are not essential, effectively removing them to save space and speed up calculations. However, a stubborn problem has persisted: the most accurate way to cut these models often creates a messy, irregular pattern of cuts that modern computer chips cannot process quickly, while the patterns that chips can handle quickly often damage the model's intelligence.
A team of researchers at the University of Toronto has introduced a new approach called PATCH that bridges this gap. Instead of forcing the entire model into a single, rigid pattern of cuts, PATCH treats the model like a mosaic made of small tiles. The system learns to decide, for each individual tile, whether to leave it completely full of data or to apply a specific, hardware-friendly pattern of cuts that removes half the data in that tile. This allows the researchers to mix dense, uncut sections with sparse, cut sections in the same model. By doing so, they can tune the overall amount of data removed to be anywhere between zero and fifty percent, finding a sweet spot where the model remains highly accurate while still running fast on standard graphics cards.
The researchers tested this method on several large language models, ranging from small ones with half a billion parameters to massive ones with thirteen billion. They found that PATCH consistently outperformed existing methods that rely on fixed cutting patterns. For instance, when applied to a popular seven-billion-parameter model running on a consumer-grade graphics card, the new method made the model run between 1.18 and 1.38 times faster than the original, uncut version. Crucially, this speedup did not come at the cost of intelligence; in fact, the pruned models were slightly more accurate than those pruned by the previous best method. The system works by observing which parts of the model are critical for understanding language and protecting those areas, while aggressively cutting the parts that contain redundant information. This selective protection happens automatically as the system trains, learning to assign dense tiles to sensitive areas and sparse tiles to less important ones.
The success of this method relies on a specific type of hardware acceleration found in modern graphics cards, which are designed to process data in groups of four, removing two of them at a time. Previous attempts to use this hardware-friendly pattern forced the entire model to follow this rule, which often led to a loss of quality. PATCH avoids this by allowing some tiles to remain untouched, effectively creating a hybrid structure. This flexibility means the model can adapt its internal structure to the specific needs of the task, rather than being forced into a one-size-fits-all shape. The researchers also demonstrated that this approach works well when combined with other compression techniques, such as reducing the precision of the numbers used, suggesting it can be part of a larger toolkit for making artificial intelligence more efficient.
While the method requires a significant amount of computing power to train the initial cutting patterns, this is a one-time cost. Once the patterns are learned, the resulting models are ready for deployment and can run efficiently on widely available hardware. The researchers noted that the speed gains are most noticeable when the model is generating text, which is the primary bottleneck in many applications. However, they also observed that as the length of the text being processed grows, the relative speed advantage decreases slightly, because other parts of the calculation begin to dominate the time spent. Despite this, the ability to maintain high accuracy while achieving measurable speedups on standard consumer hardware marks a significant step forward in making powerful language models more accessible and sustainable.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.