🤖 machine learning

EVA: Accelerating LLM Decoding via an Efficient Vector Quantization Architecture

This paper introduces EVA, a hardware-software co-optimized architecture that accelerates LLM decoding by transforming memory-bound vector quantization lookups into efficient, conflict-free GEMM computations, achieving up to 11.17× speedup and 7.17× higher energy efficiency compared to state-of-the-art methods.

Original authors: Bowen Duan, Cong Guo, Chiyue Wei, Haoxuan Shan, Yuzhe Fu, Xinhua Chen, Yifan Xu, Ziyue Zhang, Changchun Zhou, Hai Li, Yiran Chen

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bowen Duan, Cong Guo, Chiyue Wei, Haoxuan Shan, Yuzhe Fu, Xinhua Chen, Yifan Xu, Ziyue Zhang, Changchun Zhou, Hai Li, Yiran Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of knowledge (a Large Language Model, or LLM) that can write stories, solve math problems, and chat with you. To make this library work, the computer has to do two main things: reading a huge chunk of text at once (the "prefill" stage) and writing one word at a time, over and over again (the "decoding" stage).

The paper argues that while reading is fast, writing one word at a time is incredibly slow and wasteful on current computers. The authors, a team from Duke University, have built a new system called EVA to fix this.

Here is how EVA works, explained through simple analogies:

The Problem: The "One-Word-at-a-Time" Traffic Jam

Think of the computer's brain (the processor) as a giant factory with thousands of workers (processing units) ready to do math.

The Prefill Stage (Reading): Imagine the factory gets a huge shipment of 1,000 boxes. All the workers can grab a box and work on it simultaneously. This is fast and efficient.
The Decoding Stage (Writing): Now, imagine the factory has to produce just one tiny item, then wait, then produce one more, then wait. Even though the factory has thousands of workers, only one or two are ever busy. The rest are standing around doing nothing. This is the "GEMV" problem mentioned in the paper: the computer is memory-bound (waiting for data) rather than compute-bound (doing math), leading to a massive traffic jam.

The Old Solution: The "Dictionary Lookup" Bottleneck

To make the factory faster, engineers tried to shrink the "instruction manual" (the model weights) using a technique called Vector Quantization (VQ).

The Analogy: Instead of writing out the full instruction for every single word, they replaced long instructions with short codes (like "A1," "B2") that point to a shared dictionary (the Codebook).
The New Problem: While this shrinks the manual, it creates a new traffic jam. Every time the factory needs to make a word, it has to run to the dictionary, look up the code, and grab the instruction.
The Conflict: Imagine 100 workers all running to the same shelf in the dictionary at the exact same time. They bump into each other, causing a memory conflict. They have to wait in line, slowing everything down. The paper calls this "Memory Inefficiency."

The EVA Solution: Changing the Workflow

The authors of EVA realized they didn't need to change the dictionary; they just needed to change how the workers used it. They introduced a two-step magic trick:

Step 1: Do the Math Before Looking Up the Code

Instead of looking up the code first and then doing the math, EVA flips the script.

The Analogy: Imagine the workers don't wait for the dictionary. Instead, they take the input (the question) and run it against the entire dictionary all at once.
The Result: This turns the "one-by-one" math problem into a "big batch" math problem. In computer terms, they turned a slow GEMV (Matrix-Vector) operation into a fast GEMM (Matrix-Matrix) operation. Now, all the factory workers are busy again, doing math in parallel.

Step 2: The "Conflict-Free" Lookup

Once the math is done, the workers have a list of "intermediate results" (an Output Codebook).

The Analogy: In the old system, everyone ran to the same shelf. In EVA, the results are pre-sorted into different, separate bins. When a worker needs a specific result, they go to their own dedicated bin. No one bumps into anyone else.
The Result: The "memory conflict" disappears completely. The lookup becomes instant and parallel.

The Hardware: A Smart Factory Floor

The paper also describes the physical machine (the chip) built to run this system:

Reconfigurable Workers: The factory workers are smart. They can switch modes. When the computer is "reading" (prefill), they work with simple, fast 8-bit numbers. When it's "writing" (decoding), they switch to more precise 16-bit numbers to keep the quality high.
Specialized Adders: The final step of writing a word just involves adding numbers together. EVA adds a special "adder" station at the end of the line that is super fast and doesn't need complex math tools, keeping the line moving smoothly.

The Results: Speed and Efficiency

The paper tested EVA against the best existing systems (like FIGLUT and standard GPUs) using popular AI models (like LLaMA).

Speed: EVA was up to 11 times faster at generating text than the best existing lookup-based systems.
Energy: It used 7 times less energy to do the same job.
Quality: Despite compressing the model so heavily (down to 2-bit precision, which is like compressing a high-res photo into a tiny icon), the quality of the text remained excellent, with almost no loss in accuracy.

Summary

EVA is like redesigning a factory so that instead of having workers stand in line to grab instructions one by one, they process the instructions in a massive, organized batch where everyone has their own lane. This eliminates the traffic jams, keeps all the workers busy, and makes the AI write text significantly faster and more efficiently.

Technical Summary: EVA - Accelerating LLM Decoding via an Efficient Vector Quantization Architecture

1. Problem Statement

Large Language Models (LLMs) suffer from significant inference inefficiencies during the autoregressive decoding phase. Unlike the prefill stage, which utilizes compute-bound General Matrix Multiplication (GEMM) operations, decoding generates tokens sequentially, resulting in a series of small-scale General Matrix-Vector Multiplication (GEMV) operations. This shift leads to two primary bottlenecks on modern hardware accelerators:

Compute Inefficiency: GEMV operations underutilize processing elements (PEs) in accelerators optimized for large GEMM workloads, leaving most lanes idle.
Memory Inefficiency: While Weight-Only Vector Quantization (VQ) effectively compresses model weights (down to 2-bit) to reduce memory bandwidth, the standard decoding process requires codebook lookups. These lookups involve irregular, uncoalesced memory access patterns that cause frequent memory bank conflicts, serialization, and reduced cache efficiency. Consequently, conventional VQ implementations often fail to outperform, and can even be slower than, standard FP16 inference.

2. Methodology: The EVA Architecture

The paper proposes EVA, a hardware-software co-designed architecture that restructures the VQ decoding process to eliminate both computational and memory bottlenecks. The core insight is to transform the decoding workflow from a memory-bound GEMV with conflict-prone lookups into a compute-efficient, conflict-free GEMM.

A. Algorithmic Reformulation: From GEMV to GEMM

EVA introduces a two-step reformulation of the decoding computation:

Input-Codebook Dot Products (GEMV $\to$ GEMM): Instead of reconstructing the full weight matrix from indices and then performing GEMV, EVA directly computes dot products between the input activation vectors and the Weight Codebook (WC).
- The input is reshaped into vectors of size $d$ .
- These vectors are multiplied by the WC (containing $2^n$ centroids) to generate an intermediate Output Codebook (OC).
- This transforms the operation from $M=1$ GEMV to a compact GEMM ( $O = X \times B$ ), significantly increasing arithmetic intensity and parallelism.
Conflict-Free Lookup: The final output is obtained by looking up values in the OC using the Weight Index (WI) matrix.
- Unlike standard VQ where lookups into the WC cause bank conflicts, the OC is structured such that different rows (derived from different input vectors) map to different memory banks.
- This ensures that simultaneous index accesses (e.g., indices 5 and 3) map to distinct banks, eliminating memory conflicts entirely.

B. Hardware Architecture

EVA implements this reformulation on a specialized accelerator:

Reconfigurable Mixed-Precision GEMM Unit: The core utilizes a systolic array that supports both INT8 (for the prefill stage) and FP16 (for VQ decoding). It achieves this by reconfiguring a $32 \times 32$ INT8 array into a $32 \times 8$ FP16 array, decomposing FP16 multiplications into four INT8 operations. This allows seamless switching between prefill and decoding modes without duplicating hardware.
Epilogue Unit (EU): A dedicated, lightweight unit handles the post-GEMM operations. It performs conflict-free lookups on the OC and accumulates the results using adder trees. The EU is designed to be scalable and operates in parallel with the GEMM pipeline to hide latency.
Tiling Strategy: The architecture employs a weight-stationary approach where the Weight Codebook and Output Codebook remain on-chip, while input and index tensors are streamed in tiles. This minimizes off-chip memory traffic.

3. Key Contributions

Codebook-Driven GEMM Formulation: The paper proposes a novel mathematical reformulation that replaces weight decoding with direct input-codebook dot products. This shifts the decoding paradigm from memory-bound GEMV to compute-efficient GEMM.
Conflict-Free Output Codebook Lookup: A mechanism is developed that reorganizes memory access to eliminate bank conflicts, a primary limitation of existing lookup-based quantization methods.
Hardware-Software Co-Optimized Architecture: EVA is designed specifically for LLM decoding while maintaining compatibility with conventional prefill execution. It features a reconfigurable systolic array and a specialized epilogue unit to maximize utilization across different quantization precisions.

4. Experimental Results

The authors evaluated EVA using LLaMA-1, LLaMA-2, LLaMA-3, Mixtral-8x7B, and Qwen3-30B-A3B models against state-of-the-art baselines (Systolic Array, ANT, FIGNA, and FIGLUT) on a TSMC 28nm process at 500MHz.

Speedup: EVA achieves up to 11.17× speedup in decoding latency compared to the SOTA lookup-based architecture (FIGLUT).
Energy Efficiency: EVA demonstrates 7.17× higher energy efficiency compared to FIGLUT.
Accuracy: EVA preserves arithmetic precision after vector quantization. At 2-bit precision, EVA (using AQLM) maintains competitive accuracy, outperforming non-analytic baselines like GPTQ by significant margins (e.g., ~19 percentage points on average across downstream benchmarks for 2-bit).
End-to-End Performance: On decode-heavy datasets (e.g., Dolly), EVA achieves an average 8.20× to 24.49× end-to-end speedup over baselines.

5. Significance and Claims

The paper claims that EVA represents a fundamental shift in how vector-quantized LLM inference is executed on hardware. By addressing the root causes of inefficiency—specifically the GEMV underutilization and memory bank conflicts inherent in lookup-based quantization—EVA enables the practical deployment of highly compressed (2-bit) models without sacrificing inference speed.

The authors emphasize that their approach is algorithm-agnostic regarding the specific quantization method (e.g., AQLM, GPTVQ) but relies on the structural properties of vector quantization. They assert that EVA is the first architecture-level accelerator specifically designed to resolve the memory-system bottlenecks of vector-quantized LLM inference, offering a scalable solution that bridges the gap between aggressive model compression and high-performance hardware execution.