💬 NLP

BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding

BLASST is a training-free, dynamic sparse attention mechanism that accelerates LLM inference across various architectures by using a fixed scalar threshold to skip negligible attention blocks, achieving up to 1.52x speedup with negligible accuracy loss.

Original authors: Jiayi Yuan, Cameron Shinn, Kai Xu, Jingze Cui, George Klimiashvili, Guangxuan Xiao, Perkz Zheng, Bo Li, Yuxin Zhou, Zhouhai Ye, Weijie You, Tian Zheng, Dominic Brown, Pengbo Wang, Markus Hoehnerbach

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiayi Yuan, Cameron Shinn, Kai Xu, Jingze Cui, George Klimiashvili, Guangxuan Xiao, Perkz Zheng, Bo Li, Yuxin Zhou, Zhouhai Ye, Weijie You, Tian Zheng, Dominic Brown, Pengbo Wang, Markus Hoehnerbach, Richard Cai, Julien Demouth, John D. Owens, Xia Hu, Song Han, Timmy Liu, Huizi Mao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Over-Thinker" AI

Imagine you have a brilliant assistant (a Large Language Model) who is great at writing stories or solving math problems. But there's a catch: this assistant has a terrible habit of over-thinking.

When you ask them a question, they don't just look at the important parts of your prompt. Instead, they read every single word you've ever written, compare it to every other word, and calculate how much each word relates to every other word.

Short conversation: This is fine. It's like reading a text message.
Long conversation (or a whole book): This becomes a nightmare. If you paste a 100-page novel, the assistant tries to compare every sentence to every other sentence. The number of comparisons grows so fast (quadratically) that the computer runs out of memory and gets incredibly slow. It's like trying to find a specific needle in a haystack by measuring the distance between every single piece of hay and every other piece.

The Solution: BLASST (The "Smart Filter")

The authors created a new method called BLASST (Dynamic Blocked Attention Sparsity via Softmax Thresholding). That's a mouthful, so let's call it "The Smart Filter."

Instead of the assistant reading the whole book word-for-word, BLASST gives them a simple rule: "If a part of the text seems boring or irrelevant, skip it entirely."

But here's the magic: The assistant doesn't need to be retrained to learn this rule, and it doesn't need to do a "practice run" before reading the real thing. It figures it out on the fly.

How It Works: The "Running Score" Analogy

Imagine you are a judge at a talent show, and you are watching acts one by one. You want to pick the best act, but you don't want to waste time watching the bad ones if you've already seen a superstar.

The Running Max: As soon as Act #1 performs, you give them a score (say, 8/10). You keep this as your "Running Max."
The Check: Act #2 comes on. Before they even finish their song, you peek at their first few notes. You realize their "peak potential" is only a 4/10.
The Decision: Since 4 is way lower than your current best (8), you know Act #2 isn't going to beat the record. You stop watching them. You don't listen to the rest of the song, you don't write down their score, and you don't give them a prize. You immediately move to Act #3.

BLASST does exactly this with computer code:

It looks at a small "block" of text.
It checks the highest "excitement score" (attention score) in that block.
If that score is too low compared to the best score it's seen so far, it skips the whole block.
It saves the computer from doing the math, loading the data from memory, and writing the result.

Why Is This a Big Deal?

Previous methods tried to solve this problem, but they had major flaws:

The "Pre-Game" Problem: Some methods required the computer to scan the whole text before starting to figure out what to skip. This wasted time.
The "Schooling" Problem: Other methods required the AI to go back to school (retraining) to learn how to ignore things. This is expensive and slow.
The "One-Sided" Problem: Some methods only worked when the AI was reading (prefill) but not when it was writing (decode).

BLASST fixes all of these:

No Pre-Game: It decides what to skip while it's reading, instantly.
No Schooling: It works with any existing AI model right out of the box.
Two-Way Street: It speeds up both the reading phase and the writing phase.

The "Auto-Calibration" Trick

You might ask, "How does the judge know what score is 'too low' to skip?" If the threshold is too high, they skip important stuff. If it's too low, they don't skip anything.

The authors discovered a simple mathematical relationship: The longer the text, the lower the threshold needs to be.

If you are reading a short email, you can be picky.
If you are reading a 100-page novel, the "average" excitement of words is lower, so you have to lower your bar to skip the boring parts.

They built an automatic calibration tool that sets this threshold perfectly for any length of text, so you don't have to guess.

The Results: Speed vs. Quality

The paper tested this on modern supercomputers (NVIDIA Blackwell and Hopper GPUs).

Speed: They achieved a 1.5x speedup. This means the AI finishes tasks 50% faster.
Quality: The AI got rid of about 70% of the work (skipped 70% of the text blocks), but the answers were almost exactly the same as if it had read everything. In some cases, by ignoring the "noise" (irrelevant words), the AI actually gave better answers!

Summary

BLASST is like giving a super-intelligent but slow reader a pair of "smart glasses." These glasses let them glance at a page, realize a paragraph is boring, and instantly skip over it without losing the main point.

It makes reading massive documents (like entire codebases or legal contracts) fast and cheap, without needing to rebuild the AI or slow it down with extra calculations. It's a "drop-in" upgrade that makes long-context AI actually usable in the real world.

1. Problem Statement

Large Language Models (LLMs) face a critical bottleneck in long-context inference due to the quadratic computational complexity ( $O(n^2)$ ) of the self-attention mechanism. As context lengths expand to 128K or even 1M tokens, both latency and memory consumption become prohibitive.

While existing solutions like FlashAttention optimize memory bandwidth via tiling, they still compute the full attention matrix. Previous sparse attention methods attempt to reduce computation by skipping tokens or blocks, but they suffer from five major adoption hurdles:

Pre-computation Overhead: Many require expensive pre-computation to determine sparsity patterns, negating speedups.
Training Requirements: Some require fine-tuning or architectural changes.
Phase Limitation: Most focus only on the prefill or decode phase, not both.
Hardware Compatibility: Lack of optimized kernels for modern GPUs (e.g., Blackwell, Hopper).
Integration Friction: They often require intrusive modifications to model architectures or APIs.

2. Methodology: BLASST

BLASST (BLocked Attention Sparsity via Softmax Thresholding) is a training-free, drop-in sparse attention mechanism that dynamically prunes negligible attention blocks during the forward pass.

Core Insight

The method leverages the online softmax computation used in FlashAttention. During block-wise processing, the algorithm maintains a running maximum ( $m_i$ ) of attention scores. For each new block $j$ , it computes the local maximum ( $\tilde{m}_i$ ) of the attention scores ( $QK^\top$ ) within that block.

If the difference between the local maximum and the running maximum is below a threshold $\ln(\lambda)$ :
$\tilde{m}_i - m_i < \ln(\lambda) \implies \exp(\tilde{m}_i - m_i) \approx 0$
The block's contribution to the final output after softmax normalization will be negligible. Consequently, the system can skip three expensive operations for that block:

Softmax Exponentiation: Skipping the exp() calculation.
Value Loading: Skipping the load of the Value block ( $V_j$ ) from High-Bandwidth Memory (HBM) to SRAM.
Matrix Multiplication: Skipping the $P_{ij}V_j$ multiplication.

Algorithm Design

Dynamic Pruning: The decision is made on-the-fly using statistics already computed during the standard FlashAttention loop.
Threshold Calibration: The paper identifies an inverse relationship between the optimal threshold $\lambda$ and context length $L$ : $\lambda = a/L$ . An automated calibration procedure determines the scaling factor $a$ to maintain a target sparsity level across varying context lengths without manual tuning.
Sparsity-Aware Training (Optional): While designed for inference, the authors propose fine-tuning models with BLASST active. This encourages the model to concentrate information in high-scoring blocks, further improving the accuracy-sparsity trade-off.

Kernel Optimization

The authors implemented optimized CUDA kernels for Blackwell (B200) and Hopper (H200) GPUs, tailored to the specific bottlenecks of each phase:

Prefill (Compute-Bound): The kernel skips both the softmax computation (CUDA cores) and the matrix multiplication (Tensor cores). Value blocks are still loaded to maintain predictable memory access patterns, as memory bandwidth is not the bottleneck here.
Decode (Memory-Bound): The kernel focuses on skipping the Value block load from HBM, which is the primary bottleneck. To avoid pipeline bubbles caused by conditional loading, the kernel uses batched load scheduling, processing multiple query-key products before deciding which Value blocks to fetch.

3. Key Contributions

BLASST Algorithm: A training-free, pre-computation-free method that dynamically skips attention blocks using online softmax statistics. It supports MHA, GQA, MQA, and MLA (Multi-head Latent Attention).
Automated Calibration: A robust procedure that establishes a simple inverse relationship ( $\lambda \propto 1/L$ ) between threshold and context length, ensuring consistent sparsity and predictable speedups across different sequence lengths.
Optimized Kernels: Highly optimized CUDA kernels for modern GPUs that minimize overhead by hiding skip-check logic behind existing operations.
Sparsity-Aware Training: A simple extension showing that models trained with sparse attention patterns are more robust to aggressive pruning.
Framework Integration: The kernels are integrated into TensorRT-LLM and FlashInfer, making them immediately usable in production.

4. Experimental Results

The authors evaluated BLASST on Llama-3.1-8B and Qwen3-8B across various benchmarks (RULER, LongBench, MATH500, AIME 2024, GPQA).

Accuracy: BLASST preserves benchmark accuracy with minimal degradation. At ~70% sparsity, accuracy drops are negligible. In some cases (e.g., Qwen3-8B on MATH500), BLASST even outperformed the dense baseline, likely due to implicit denoising by filtering out low-attention tokens.
Speedup (Prefill): Achieved up to 1.52× speedup at 71.9% sparsity on Blackwell GPUs.
Speedup (Decode): Achieved up to 1.48× speedup at 73.2% sparsity on Blackwell GPUs.
Comparison: BLASST outperformed other sparse methods (like MInference, FlexPrefill, XAttention) in accuracy while being the only method to accelerate both prefill and decode without training or pre-computation.
Long Context: Effective on extremely long sequences (up to 200K tokens), where natural sparsity increases, making the method even more effective.

5. Significance

BLASST represents a significant step toward practical, long-context LLM deployment. By removing the barriers of training, pre-computation, and architectural modification, it offers a universal optimization that works with existing models and frameworks. Its ability to dynamically adapt to context length via simple calibration and its optimization for the latest GPU hardware make it a viable solution for production inference systems requiring both high throughput and long context windows. The paper demonstrates that dynamic sparsity based on actual softmax statistics is a more robust and efficient approach than proxy-based or static sparsity methods.