💻 computer science

FlashAttention for Scalable Vector Architectures

This paper introduces FlashAttention-V, a blocked FlashAttention implementation optimized for scalable vector architectures that significantly accelerates transformer inference on CPUs by leveraging inter-head parallelism and efficient memory access, achieving up to 42x speedups in prefill and 11x in decode while also highlighting structural bottlenecks in current quantization formats for long-vector execution.

Original authors: Sonia Rani Gupta, Nikela Papadopoulou, Miquel Pericàs

Published 2026-08-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sonia Rani Gupta, Nikela Papadopoulou, Miquel Pericàs

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern artificial intelligence systems, from chatbots that draft emails to coding assistants that debug software, rely on a specific type of computer program called a transformer. These programs are built to understand language by looking at how words relate to one another in a sentence. To do this, they use a mechanism called attention, which acts like a spotlight, allowing the system to focus on the most relevant parts of the text while ignoring the rest. While these systems are often run on massive data centers with powerful graphics cards, there is a growing need to run them on smaller, everyday devices like laptops, tablets, and even specialized microchips found in the Internet of Things. These smaller devices often use a different kind of processor designed to handle many calculations at once by processing data in long lines, known as vector architectures. However, a major hurdle has emerged: the attention mechanism is incredibly hungry for memory, requiring the processor to constantly fetch and store large amounts of data, which slows everything down.

Researchers at Chalmers University of Technology and the University of Glasgow have tackled this problem by redesigning how the attention mechanism works on these vector processors. They created a new method called FlashAttention-V, which changes the way the computer organizes its work to fit the unique shape of these processors. Instead of trying to force the processor to handle one small piece of data at a time, the new method groups multiple independent calculations together into a single, wide line of data. Imagine a factory assembly line where workers usually handle one item at a time; this new approach allows them to grab a whole tray of items and process them all in one go, drastically reducing the time spent walking back and forth to the storage shelves. By rearranging the order of operations and packing data more tightly, the researchers found they could make these smaller devices run significantly faster, especially when dealing with short bursts of text or when the device is generating new words one by one.

The team tested their new approach on several different language models, including TinyLlama, Llama 3.2, and Qwen2.5, using both real hardware and detailed computer simulations. On a physical development board called the Banana Pi BPI-F3, which uses a RISC-V processor, the new method proved to be a massive improvement. When the device was preparing to read a short input, the new approach was between twelve and fourteen times faster than the standard, non-optimized version. When the device was generating text word by word, it was four to five times faster. The researchers also ran extensive simulations to see how the method would perform if the processors were built with even wider data lines, capable of handling much larger chunks of information at once. These simulations showed that as the data lines grew wider, the speed gains continued to climb, reaching up to forty-two times faster than the basic version in the best-case scenarios for preparing inputs.

However, the study also revealed a distinct limit to how much faster these devices can get. The researchers discovered that while the attention part of the program benefits greatly from these wider data lines, other parts of the system, specifically the layers that convert numbers into predictions, do not. These layers use a specific way of storing numbers called quantization, which compresses the data to save space. The way this data is currently packed creates a structural mismatch with the wide data lines, forcing the processor to do extra, inefficient work to separate and recombine the numbers. The simulations showed that for these specific layers, the time saved by processing more data at once was completely eaten up by the time spent rearranging it. This means that while the attention mechanism can be made incredibly fast, the overall speed of the system is currently held back by these other components, suggesting that future improvements will require a change in how these numbers are stored, not just how they are processed.

The findings offer a clear path forward for making artificial intelligence more accessible on everyday devices. The new method, FlashAttention-V, successfully bridges the gap between the design of modern language models and the capabilities of scalable vector processors. It proves that by simply reordering how the computer thinks about its tasks and packing data more efficiently, significant performance gains are possible without needing new hardware. The research confirms that for short tasks and real-time text generation, these optimized processors can be highly effective. Yet, it also serves as a cautionary note that the current ways of compressing data for these devices may be hitting a wall, and that unlocking the full potential of future, wider processors will likely depend on solving the puzzle of how to store and move these compressed numbers more intelligently.

Technical Summary: FlashAttention for Scalable Vector Architectures

Problem Statement

Inference with transformer models on CPUs is increasingly critical, particularly for Small Language Models (SLMs) deployed on edge devices. While vector architectures like RISC-V Vector Extension (RVV) and ARM Scalable Vector Extension (SVE) offer promising execution substrates, the attention module remains a primary bottleneck due to high memory bandwidth requirements.

Existing implementations of FlashAttention on vector CPUs (e.g., in llama.cpp) suffer from a fundamental scalability limitation: they tie the vector length (VL) to the head dimension ( $D$ ), typically 64 or 128. Consequently, when hardware supports vector lengths significantly larger than $D$ (e.g., 512-bit, 2048-bit, or 8192-bit), these implementations cannot utilize the full SIMD width. They saturate at $VL \le D$ , leaving wider vector registers underutilized and failing to exploit the potential of long-vector architectures. Furthermore, standard quantization formats (specifically Q8_0) introduce structural incompatibilities with long-vector execution in linear projection and feed-forward layers, creating additional bottlenecks.

Methodology

The authors propose FlashAttention-V, a redesigned FlashAttention algorithm specifically optimized for scalable vector architectures. The core innovation is the exploitation of parallelism across attention heads rather than just within a single head.

Key Algorithmic Transformations

Inter-Head Packing: When $VL > D$, FlashAttention-V maps multiple attention heads into a single vector register. By reordering loops over attention heads and applying loop unrolling, the algorithm packs $Q$ , $K$ , and $V$ elements from multiple heads into contiguous vector registers. This allows the system to utilize vector lengths exceeding the head dimension.
Loop Reordering and Unrolling: The algorithm reorders loops to improve spatial locality and enable unit-stride memory accesses. It applies loop unrolling over attention heads to increase instruction-level parallelism (ILP) and maximize vector register occupancy.
GQA/MHA Awareness: The design natively supports Grouped-Query Attention (GQA) and Multi-Query Attention (MQA). For shared key-value heads, the algorithm loads data once and replicates it across the vector register, avoiding redundant memory accesses.
Blocked Execution: The implementation uses cache-aware blocking strategies (inspired by FlashAttention-2) to keep intermediate data in registers and reduce memory traffic.

Experimental Setup

Implementation: Integrated into the ggml framework within llama.cpp using RISC-V and ARM SVE intrinsics.
Models Evaluated: TinyLlama, Llama 3.2, Qwen2.5 (GQA models), and Pythia-410M (MHA model).
Hardware/Simulation:
- Real Hardware: Banana Pi BPI-F3 (RVV 256-bit).
- Simulation: gem5 configured with RISC-V in-order RiscvMinorCPU. The simulator was extended to model realistic vector latencies proportional to vector length and lane count (based on Vitruvius+ microarchitecture), scaling from 512-bit to 8192-bit vector lengths and 8 to 64 lanes.
Baselines: Non-vectorized FlashAttention (ggml-scalar) and existing vectorized FlashAttention (ggml-vec).

Key Contributions

FlashAttention-V Algorithm: A novel approach that enables effective utilization of vector lengths beyond the head dimension ($VL > D$) by exploiting inter-head parallelism. It achieves 22×–42× speedup over scalar FlashAttention in the prefill stage and 8×–11× in the decode stage at 512-bit vector lengths (gem5 simulation).
Scalability Analysis: The paper provides a quantitative analysis of FlashAttention-V scaling up to 8192-bit vector lengths. It identifies that while theoretical speedups increase with vector width, realistic performance is governed by vector functional unit latencies. The authors find that configurations with 64 vector lanes and 4096-bit vectors achieve optimal scalability (2×–2.5× gain in prefill), while gains diminish beyond this point due to latency overheads.
Identification of Quantization Bottlenecks: The authors characterize a structural incompatibility between Q8_0 quantization and long-vector execution in linear projection and feed-forward layers. They demonstrate that the interleaved layout of weights and scales in Q8_0 forces explicit packing and masked reduction operations, which consume up to 60% of execution cycles at 2048-bit VL. This overhead negates the arithmetic benefits of longer vectors, a finding consistent across both RVV and Arm SVE.

Results

Real Hardware (Banana Pi BPI-F3): FlashAttention-V achieved 12×–14× speedup over scalar FlashAttention. It was most effective for short contexts ( $N \le 128$ ), which are common in edge inference.
Prefill Stage (Simulation):
- At 512-bit VL, speedups ranged from 22× (TinyLlama) to 42× (Qwen2.5) over scalar baselines.
- Scaling to 8192-bit VL provided additional gains (up to ~3×) for models with smaller head dimensions that could pack more heads per register.
- Under realistic latency scaling (64 lanes), the 4096-bit configuration sustained a 2×–2.5× speedup over the 512-bit baseline, while 8192-bit configurations saw diminishing returns due to latency penalties.
Decode Stage: FlashAttention-V achieved 8×–11× speedup at 512-bit VL. However, due to the single-token nature of decoding, performance showed diminishing sensitivity to vector width and lane count beyond 512 bits, as the workload per iteration was insufficient to amortize packing/unpacking overheads.
Quantized Layers: Microbenchmarks revealed that packing and masked reduction overheads in Q8_0 linear layers dominate execution time at high vector lengths, preventing arithmetic amortization.

Significance and Claims

The paper claims to bridge the gap between FlashAttention and scalable vector architectures by enabling efficient execution on vector lengths far exceeding the head dimension. The authors assert that their optimizations (loop reordering, inter-head packing, and unrolling) are transferable principles that significantly accelerate transformer inference on edge devices.

Crucially, the paper highlights a fundamental limitation in current quantization strategies for long-vector execution. It argues that while FlashAttention-V successfully scales the attention mechanism, the structural layout of Q8_0 quantized linear layers poses a barrier to similar scalability in feed-forward networks. The authors conclude that future work must explore alternative quantization formats or memory layouts to eliminate explicit packing overheads, as the current arithmetic gains of long vectors are currently offset by data movement costs in quantized models.

The work provides actionable insights for hardware designers and software engineers, suggesting that 64-lane, 4096-bit vector configurations offer a "sweet spot" for scalability before latency overheads dominate, and that optimizing attention kernels alone is insufficient for end-to-end performance gains without addressing quantization-induced bottlenecks.