Technical Summary: Llamas on the Web

Problem Statement

Running Large Language Models (LLMs) locally in web browsers offers significant advantages for privacy, latency, and energy efficiency, particularly as model developers release open-weight models designed for edge deployment. However, realizing this potential is hindered by three primary constraints in existing browser-based inference frameworks (e.g., WebLLM, Transformers.js):

1. Memory Inefficiencies: Existing engines often dynamically allocate GPU memory and load weights inefficiently. This leads to memory growth during execution, causing slowdowns or crashes due to hard memory caps enforced by browsers per tab.
2. Lack of Performance-Portable Design: While WebGPU provides functional portability across diverse hardware, existing frameworks lack sufficient kernel specialization. They fail to tune for specific device capabilities (e.g., subgroup operations, tensor cores), resulting in suboptimal performance across the heterogeneous hardware landscape of browsers.
3. Limited Quantization Support: To support the wide variety of quantization formats used in the llama.cpp ecosystem (ranging from legacy schemes to advanced K-quants and I-quants), inference engines require flexible kernel designs. Existing frameworks often lack the extensibility to support these diverse formats efficiently or to integrate new ones rapidly.

Methodology: LlamaWeb

The authors present LlamaWeb, a WebGPU backend for llama.cpp designed to address these constraints. The system is built within the llama.cpp codebase to leverage its extensive model support and active ecosystem. The design focuses on three core pillars:

1. Memory-Efficient Local Inference

To minimize memory overhead and ensure stability on memory-constrained devices:

• Static Memory Allocation: All memory required for model execution, including intermediate data structures for operations like FlashAttention and fixed-size slotted regions for kernel parameters, is statically allocated at startup. This avoids dynamic buffer creation and fragmentation.
• Optimized Loading: The system utilizes the wllama library to stream weights asynchronously from the Origin Private File System (OPFS) directly to WebGPU buffers. It avoids materializing weights in the WebAssembly heap, which is grow-only and cannot release memory during a tab's lifetime.
• Parameter Buffer Arena: Instead of maintaining dynamic pools of small buffers for kernel parameters (a common source of fragmentation), LlamaWeb allocates a single static buffer with configurable slots, rotating through them to guarantee parameters are not overwritten until kernel execution completes.

2. Performance-Portable Kernel Library

To maximize performance across heterogeneous hardware:

• Templated Shader Generation: The backend uses a custom WGSL preprocessor (pre-wgsl) and a kernel library that generates shaders based on device-specific features (e.g., subgroup support) and data types.
• Tunable Parameters: Key parameters, such as tile sizes for matrix multiplication and workgroup sizes, are tunable. The authors derived "performance-portable" defaults through an empirical study sweeping thousands of configurations across four GPUs from different vendors.
• Execution Scheduling: The runtime efficiently groups kernel submissions and minimizes CPU-GPU synchronization. It caches compiled pipelines to avoid redundant compilation costs after the initial forward pass.

3. Quantization-Aware Shader Design

To support the diverse quantization formats of llama.cpp:

• Integrated Dequantization: Dequantization routines are co-designed with computation kernels. For matrix multiplication, quantized blocks are loaded, dequantized into shared memory, and reused. For matrix-vector multiplication (decode phase), dequantization occurs directly into registers to improve memory access patterns.
• Extensibility: The system represents quantization formats as flat buffers of unsigned 32-bit integers, with dequantization logic selected at compile time. This allows new formats (e.g., the recently introduced q1_0 for 1-bit models) to be integrated by adding a dequantization routine without rewriting core matrix operation implementations.

Key Contributions

1. A Memory-Efficient Inference Engine: A WebGPU implementation that statically allocates memory and optimizes data movement, enabling the execution of useful models on memory-constrained edge devices without unexpected crashes.
2. A Flexible Shader Library: A library for LLM inference that supports external tuning, specializes shaders based on available hardware features (e.g., subgroup matrix operations), and utilizes an optimized scheduling runtime.
3. Co-Designed Quantization Kernels: A set of core kernels (matrix operations) designed to work with llama.cpp's extensive quantization formats, enabling the deployment of models optimized for different sizes and facilitating the integration of emerging quantization methods.

Experimental Results

The authors evaluated LlamaWeb on 16 devices from 8 vendors (including NVIDIA, AMD, Intel, Apple, Qualcomm, Samsung, ARM, and Imagination Technologies) running 10 language models across four weight formats.

• Memory Efficiency: Compared to WebLLM and Transformers.js, LlamaWeb reduced peak browser memory usage by 29–33% on average across various device/browser/OS combinations. In specific tests (e.g., Safari on Apple M4), it used up to 59% less memory than Transformers.js, which suffered from memory leaks in those configurations.
• Decode Throughput: LlamaWeb increased decode throughput by 45–69% compared to WebLLM and Transformers.js across four GPUs from separate vendors.
• Prefill Throughput: During the prefill phase, LlamaWeb initially underperformed compared to WebLLM (by 21–51%) and Transformers.js. The authors attribute this to WebLLM's use of TVM's kernel fusion and Transformers.js's subgroup-optimized implementations. However, when running LlamaWeb natively (outside the browser) with safety checks disabled, it outperformed other frameworks, suggesting that browser-specific safety checks and the lack of stable subgroup matrix support in browsers are the primary bottlenecks.
• Native vs. Browser Performance: On some devices (e.g., Intel Arc B580), LlamaWeb's WebGPU backend was competitive with or even faster than native backends (e.g., SYCL) for specific tasks. However, native backends (CUDA, Metal) generally outperformed WebGPU by up to 10x in prefill and 2.5x in decode, largely due to better utilization of vendor-specific features like tensor cores and asynchronous memory loads.
• Quantization Impact: The study found that while quantization reduces memory requirements, it does not always linearly improve performance due to the computational cost of dequantization. For instance, moving from q8_0 to q4_k_m improved throughput, but moving to q2_k decreased it on mid-tier clusters.

Significance and Claims

The paper claims that LlamaWeb represents a significant step toward making browser-based LLM inference accessible, private, and performant. By integrating with the llama.cpp ecosystem, it supports a vast library of models (177,691 compatible models in GGUF format) that other browser frameworks cannot match.

The authors emphasize that their work demonstrates that performance portability is achievable in the browser through careful kernel tuning and static memory planning. While they acknowledge that native backends currently hold a performance advantage, LlamaWeb provides a competitive cross-platform alternative that avoids the need for platform-specific native installations. The work also highlights the need for further improvements in WebGPU specifications, such as better support for smaller integer types and the removal of runtime safety checks that hinder performance, to fully realize the potential of browser-based AI.

The code is open-source and has been integrated into the llama.cpp codebase via over 100 pull requests, supporting 72 core ML operators and 23 data formats.