🤖 AI

TileFuse: A Fused Mixed-Precision Kernel Library for Efficient Quantized LLM Inference on AMD NPUs

This paper introduces TileFuse, a close-to-metal mixed-precision kernel library for AMD XDNA2 NPUs that fuses unpacking, dequantization, and matrix operations to enable efficient, native support for popular AWQ-style quantized LLM inference, achieving significant performance and energy efficiency gains over existing baselines on client edge devices.

Original authors: Wesley Pang, Gregory Hyegang Jun, Feiyang Liu, Deming Chen

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wesley Pang, Gregory Hyegang Jun, Feiyang Liu, Deming Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brand new, super-fast delivery truck (the NPU) sitting in your garage, but the only road map you have (the software) tells you how to drive a slow, old bicycle. You can't use the truck's speed because the map doesn't know how to navigate its lanes.

This is the problem with running modern AI chatbots (LLMs) on new laptop chips today. The chips have powerful "AI engines" (NPUs) designed to save battery and run fast, but the software usually forces the AI to change its shape to fit the engine, rather than letting the engine adapt to the AI.

TileFuse is a new set of tools that fixes this. It's like building a custom, high-speed highway specifically for the truck, allowing it to carry heavy loads of compressed data without stopping to repackage them.

Here is how the paper explains this solution using simple analogies:

1. The Problem: The "Repacking" Bottleneck

Usually, to run an AI on these new chips, you have to take the AI's "compressed" weights (which are like books packed tightly into a small suitcase) and unpack them into a bulky format (like taking the books out and laying them flat on a table) just so the chip can read them.

The Old Way: The chip reads the suitcase, unpacks the books, puts them back in a different suitcase, and then starts reading. This wastes time and energy.
The TileFuse Way: The chip reads the suitcase, opens it, and reads the books while they are still being unpacked. It does everything in one smooth motion.

2. The Solution: "Fused" Kernels

The authors created a library called TileFuse. Think of a "kernel" as a specific instruction manual for the chip.

Fusion: Instead of having three separate workers (one to unpack, one to convert, one to calculate), TileFuse combines them into one super-worker. This worker grabs the compressed data, converts it on the fly, and does the math all in one go.
The Result: This is like a chef who doesn't just chop vegetables; they chop, season, and cook them in a single, continuous motion. The paper claims this makes the "unpacking" part of the process up to 2.8 times faster for certain tasks compared to older methods.

3. The "Interleaved" Layout: Organizing the Warehouse

Large AI models have massive lists of numbers (weights). The chip's memory system has a limit on how far it can reach to grab the next piece of data. If the data is stored in a messy, scattered way, the chip has to make long, slow trips to get the next piece.

The Analogy: Imagine a warehouse where boxes are stacked randomly. A forklift has to drive 50 feet to get the next box.
TileFuse's Fix: They rearrange the warehouse (called "Interleaved Pre-tiling") so that the boxes the forklift needs next are right next to each other. This allows the chip to grab huge chunks of data in one smooth sweep, supporting much larger AI models (up to 32,000 items wide) that previously wouldn't fit.

4. The "GEMV" Problem: The Traffic Jam

AI chatbots work in two phases:

Prefilling: Reading a long prompt all at once (like reading a whole book). This is fast and easy for the truck.
Token Generation: Writing one word at a time (like writing a sentence). This is slow and tricky.

The Issue: When writing one word at a time, the chip's "truck" often sits idle because it's designed to carry big loads, not small ones. It's like using a semi-truck to deliver a single letter; the engine is running, but the truck is empty.
The Fix: TileFuse redesigned the traffic flow for this phase. Instead of sending data to just one lane of the highway, it distributes the work across all 32 lanes of the chip simultaneously. This keeps the whole engine busy, even when the "load" is small.

5. The Real-World Results

The team tested this on real AMD laptops (Ryzen AI) and compared it to using the laptop's standard graphics card (iGPU).

Speed: For reading long prompts (prefilling), the NPU with TileFuse was up to 2 times faster than the graphics card.
Battery: Because the NPU is more efficient, it used 64% less energy to do the same job.
The Catch: For writing one word at a time (token generation), the NPU was sometimes slower than the graphics card. This is because the "setup time" to configure the NPU is too long for such tiny, quick tasks.
The Hybrid Solution: The paper suggests a "best of both worlds" approach: Use the NPU for the heavy lifting (reading long prompts) and the graphics card for the quick tasks (writing words). This combination gives the best speed and battery life.

Summary

TileFuse is a bridge. It takes the popular, compressed AI formats that developers already use (like AWQ) and makes them run natively on AMD's new AI chips without needing to change the AI models themselves. By fusing the unpacking and math steps, organizing the data perfectly, and utilizing the full power of the chip's lanes, it makes running AI on laptops significantly faster and more energy-efficient.

Technical Summary: TileFuse

Problem Statement

The deployment of Large Language Models (LLMs) on edge devices is increasingly reliant on Neural Processing Units (NPUs) to meet strict power and thermal budgets. However, practical LLM inference on client NPUs, specifically AMD's XDNA2 architecture, faces significant barriers. The primary obstacle is the fragmentation of software stacks and the lack of low-level programmability. Most NPU frameworks (e.g., Apple Core ML, Intel OpenVINO, Qualcomm QNN) expose only high-level interfaces or proprietary deployment paths that do not natively support widely used, off-the-shelf quantization formats like AWQ (Activation-Weight Quantization).

Consequently, developers are often forced to reshape models to fit NPU-specific quantization schemes or rely on less efficient full-precision execution. While some prior works have explored hardware-aware quantization for specific platforms (e.g., Qualcomm's ScalingLLM), they often fail to support standard formats like $W4A16$ or cannot handle the large matrix dimensions (e.g., >8K) common in modern LLMs. Furthermore, existing NPU optimizations often lack the ability to fuse unpacking and dequantization directly into the compute kernel, leading to unnecessary memory traffic and inefficient data movement.

Methodology

TileFuse is a close-to-metal mixed-precision kernel library designed for AMD XDNA2 NPUs, targeting the linear layers (GEMM and GEMV) that dominate LLM inference. The system leverages the open MLIR-AIE (IRON) programming flow to co-design weight layouts, metadata placement, microkernels, and array-level dataflows.

1. Fused Mixed-Precision Microkernels

TileFuse fuses three distinct operations into a single kernel flow:

Unpacking: Extracting packed INT4/INT8 weights.
Dequantization: Converting low-bit integers to BF16 using per-group or per-channel scales and zero-points.
Execution: Performing the matrix multiplication (GEMM) or matrix-vector multiplication (GEMV).

By keeping weights in compressed format until they reach the compute core, TileFuse avoids the global memory round-trip required to materialize full-precision (BF16) weights. The microkernels are designed to unpack and dequantize each weight tile once, store the resulting BF16 values in local compute-core memory, and reuse them for subsequent matrix operations, thereby preserving the reuse patterns essential for tiled GEMM performance.

2. Interleaved Pre-Tiling

To address the limitations of runtime DMA streaming constraints (which typically cap supported dimensions below 8K), TileFuse employs an offline interleaved pre-tiling strategy.

Metadata Packing: Quantization metadata (scales and zero-points) is packed directly into the weight tiles in memory, ensuring that all required data for dequantization is available contiguously.
Layout Reorganization: The weight matrix is reorganized into an interleaved column-major layout. This ensures that weight tiles assigned to the same AIE column are contiguous in memory, eliminating large memory strides that would otherwise force a single GEMM operation to be split across multiple kernel invocations. This approach supports GEMM dimensions up to 32K.

3. GEMV Dataflow Co-Design

Token generation in edge LLMs is typically a batch-size-1 GEMV operation, which is often memory-bound in full precision but becomes compute-bound when quantization and dequantization are introduced.

Baseline Limitation: Standard GEMV mappings often utilize only a single row of the 4×8 AIE compute array, leaving the majority of the hardware idle.
Optimization: TileFuse redesigns the dataflow to distribute weights across the full 4×8 array. It utilizes memory cores as an intermediate distribution layer, streaming bundles of weight tiles from shim cores to memory cores, which then fan out to all four compute rows. This maximizes spatial utilization and arithmetic throughput for token generation.

4. Reconfigurability-Aware Optimization

The system explicitly leverages the reconfigurability of the XDNA2 architecture to customize compute-core microkernels and DMA access patterns for specific workloads. However, the authors acknowledge that this flexibility introduces runtime dispatch and fabric reconfiguration overhead. TileFuse is optimized to amortize this overhead during large prefilling GEMMs but recognizes its limitations for latency-sensitive, small-scale GEMV operations.

Key Contributions

Fused Kernels for Practical Quantization: Implementation of fused AIE kernels that directly consume AWQ-style $W4A16$ and $W8A16$ weights, eliminating the need for model reshaping or proprietary quantization schemes.
Interleaved Pre-Tiling: A layout strategy that supports GEMM dimensions up to 32K and enables efficient runtime streaming by aligning memory access with DMA buffer descriptor constraints.
GEMV Dataflow Redesign: A new dataflow that distributes weights across the full 4×8 AIE array, significantly improving token-generation throughput.
End-to-End Edge Efficiency: Integration of these kernels into a hybrid runtime (NPU for linear layers, iGPU for attention/non-linear layers) that demonstrates substantial gains in latency and energy efficiency.

Results

Evaluations were conducted on Ryzen AI laptops (Ryzen AI 7 350 and Ryzen AI 9 HX 370) using models like Llama3-8B, Gemma-2B, and Qwen2.5-3B.

Kernel Performance:
- GEMM: TileFuse achieves up to 121.6% performance improvement over full-precision baselines.
- GEMV: TileFuse achieves up to 281% performance improvement over full-precision baselines.
- Comparison: On GEMM, TileFuse delivers more than 2× performance and energy-efficiency gains over strong iGPU baselines (llama.cpp fused kernels).
End-to-End LLM Inference:
- Prefilling: On the Ryzen AI 7 350, TileFuse achieves up to 2.0× lower prefilling latency (e.g., 2.00× speedup for Llama3-8B at 2048 tokens) compared to the iGPU baseline.
- Energy: The NPU path reduces energy consumption by more than 64.6% on the Ryzen AI 7 350 and 52.8% on the Ryzen AI 9 HX 370.
- Token Generation: The NPU underperforms the iGPU for token generation due to the millisecond-scale overhead of kernel dispatch and reconfiguration relative to the microsecond-scale execution of GEMV operators.
Comparison with Prior Work: TileFuse outperforms recent NPU-driven frameworks like FastFlowLM and Scaling NPU in prefilling throughput (e.g., 1.45× and 1.61× speedups respectively on Llama3-8B and Qwen2.5-3B).

Significance and Claims

The paper claims that TileFuse demonstrates that AMD XDNA2 NPUs are a practical target for efficient, off-the-shelf quantized LLM inference. By bringing standard formats like AWQ directly onto the hardware through close-to-metal optimization, the work bridges the gap between raw NPU potential and real-world client deployment.

The authors emphasize that the success of TileFuse relies on a hybrid deployment strategy: utilizing the NPU's reconfigurable spatial array for large-context prefilling (where overhead is amortized) while relying on the iGPU for latency-sensitive token generation. This approach validates that NPUs can substantially improve edge LLM usability when hardware is adapted to user-reliant quantization formats, rather than forcing models to conform to proprietary constraints. The work establishes that with appropriate kernel and dataflow co-design, NPUs can deliver significant performance and energy benefits for edge AI, even in the presence of software stack fragmentation.