🤖 machine learning

Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

This paper reveals that WebGPU dispatch overhead, rather than kernel quality, is the primary bottleneck for single-batch LLM inference in browsers, demonstrating that simple measurements overestimate costs due to synchronization conflation and concluding that reducing dispatch count through amortization is the most effective optimization strategy.

Original authors: Jędrzej Maczan

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jędrzej Maczan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to run a massive, complex video game on a computer, but you have to do it through a very strict, security-conscious manager who doesn't let you touch the hardware directly. This is the world of running Artificial Intelligence (specifically Large Language Models, or LLMs) inside a web browser. These models are the brains behind chatbots that can write stories, solve math, and hold conversations. To make them run fast on your laptop or phone without needing a supercomputer, developers use a special tool called WebGPU. Think of WebGPU as a universal translator that lets your browser talk to your computer's graphics card (the part that usually renders video games) so it can do the heavy math for AI.

However, there's a catch. In the past, when developers tried to make these AI models faster, they focused on making the individual math steps (called "kernels") more efficient, like polishing the engine of a car. But this paper asks a different question: What if the car is fine, but the driver is spending too much time getting in and out of the vehicle? In the browser world, every single math step requires a "dispatch"—a request sent from the browser to the graphics card to start working. The big mystery was: How much time is actually wasted just sending these requests, versus doing the actual math? Understanding this is crucial because if we waste too much time just asking the computer to work, the chatbot will feel slow and sluggish, no matter how smart the math is.

The "Stop-and-Go" Traffic Jam

The researcher in this paper discovered that everyone had been measuring the speed of these AI requests all wrong. Imagine you are timing how long it takes a delivery driver to drop off a package. If you time them from the moment they leave the warehouse, drive to the house, drop the package, and then drive all the way back to the warehouse to get the next one, you are measuring the whole round trip. But in the real world of AI, the driver doesn't drive back to the warehouse after every single package. They drop off a whole stack of packages in one go, and only drive back once at the very end.

The paper shows that previous measurements were like timing that full round trip for every single package. They were mixing up the time it takes to send the request (the dispatch) with the time it takes to wait for the computer to say "Okay, I'm done" (synchronization). This "wait time" is huge—it's like a 450 microsecond pause. When researchers added this wait time to every single step, they thought the cost of sending a request was about 20 times higher than it really was.

By using a new method called "sequential-dispatch," the author figured out how to time just the act of sending the request, without the long wait in between. They found the true cost is much lower: between 24–36 microseconds on some systems (Vulkan) and 32–71 microseconds on others (Metal). Interestingly, this cost is the same whether the computer is using "float32" or "float16" numbers (two different ways of storing decimal numbers), proving that the delay comes from the browser's rules, not the math itself.

The Real Bottleneck: Too Many Stops

Once they knew the true cost of a single request, the team asked: "Does this actually matter?" To find out, they ran a controlled experiment. They took a standard AI model and changed how it was packaged. Instead of sending 876 tiny requests to the graphics card to process one word of text, they "fused" (glued together) some of the steps so the card only had to receive 564 requests.

Here is the kicker: they didn't make the math inside the requests any faster. They didn't change the code to be smarter or use less memory. They just reduced the number of times the browser had to knock on the graphics card's door.

The result? The AI got 53% faster. The time it took to generate the first word of a response dropped from 71.4 ms to 41.6 ms.

This experiment proved that at the most common setting (processing one word at a time, known as "batch size 1"), the biggest problem isn't that the math is too slow or the memory is too full. The problem is simply that there are too many "knocks on the door." The author explicitly ruled out the idea that better math code or less memory usage was the reason for the speedup. The only thing that changed was the number of dispatches.

What This Means for the Future

The paper concludes that if we want browsers to run AI chatbots smoothly, we need to stop trying to perfect every single math step and start focusing on grouping them together. It's like realizing that to get a delivery truck to a house faster, you shouldn't just make the driver run faster; you should just make sure they carry a bigger box so they don't have to make as many trips.

The author suggests that the solution lies in "dispatch amortization"—a fancy way of saying we need to spread the cost of those "knocks" over many tasks so the delay doesn't hurt as much. They point out that this might require changes not just in the software that runs the AI, but potentially in the WebGPU rules themselves, perhaps allowing the browser to accept a "command graph" (a pre-planned route) rather than checking every single step individually.

While these findings are based on specific hardware (like the NVIDIA RTX 5090) and one specific way of running the AI, the message is clear: for now, the secret to faster AI in your browser isn't a faster engine; it's fewer stops.

Technical Summary: Measuring and Reducing WebGPU Dispatch Overhead for LLM Inference

Problem Statement

Large Language Models (LLMs) are increasingly deployed in browser and edge environments using WebGPU as a cross-platform standard. While existing performance engineering for these systems focuses on quantization, kernel optimization, and memory efficiency, the cost of WebGPU operation dispatch remains poorly characterized.

A critical issue identified is that standard "single-operation" performance measurements—timing a single operation end-to-end—conflate the cost of the dispatch itself with the cost of full GPU-CPU synchronization. In real-world autoregressive decoding, synchronization occurs only once per token, not once per operation. Consequently, single-operation measurements overestimate the per-dispatch cost by approximately 20x, obscuring the true performance bottleneck. Furthermore, it remains unclear whether throughput limitations at batch size 1 are caused by kernel quality (shader efficiency) or the sheer number of dispatches required.

Methodology

The author introduces a sequential-dispatch measurement method to isolate the true marginal cost of a single dispatch.

Sequential Measurement: Instead of timing a single operation with immediate synchronization, the method submits $N$ identical operations to the queue and performs a single synchronization at the very end. The total time is divided by $N$ , amortizing the one-time synchronization cost and revealing the true dispatch overhead.
Controlled Fusion Experiment: To determine the impact of dispatch count versus kernel quality, the author conducted a kernel fusion experiment. They fused groups of operations to reduce the dispatch count per forward pass from 876 to 564. Crucially, the underlying WGSL shaders (computation logic) remained unchanged, and the reduction in intermediate memory traffic was negligible (<1.8 MB, <1 µs). This setup isolates the dispatch count as the sole variable.
Implementation: Measurements were performed using torch-webgpu, a custom PyTorch backend, across multiple hardware vendors (NVIDIA, AMD, Apple, Intel) and backends (Vulkan, Metal, Dawn, wgpu).

Key Results

1. True Per-Dispatch Cost

Using the sequential method, the author measured the actual per-dispatch cost, finding it to be independent of data type (float32 vs. float16):

Vulkan: 24–36 µs per dispatch.
Metal: 32–71 µs per dispatch.
Overestimation: Single-operation measurements overestimate these costs by ~20x (e.g., measuring ~497 µs on Dawn, where ~450 µs is attributed to synchronization).

2. Dispatch Count as the Bottleneck

The controlled fusion experiment demonstrated that reducing dispatch count significantly improves throughput without altering kernel quality:

Reducing dispatches from 876 to 564 resulted in a 53% increase in throughput (tokens per second).
Time to First Token (TTFT) dropped from 71.4 ms to 41.6 ms.
Since computation and memory traffic were held constant, the author concludes that dispatch count, not kernel quality, is the primary bottleneck for LLM inference at batch size 1.

3. Overhead Breakdown

Analysis of the dispatch overhead components revealed that command buffer submission is the dominant factor:

Submit: ~12.9 µs (40% of total CPU time).
Encoder Create/Finish: ~12.5 µs combined (39%).
Pipeline/Bind Group Setup: <1.5 µs each.
This indicates that the primary bottleneck lies in the API overhead of submitting commands, rather than the setup of pipelines or binding resources.

Significance and Claims

The paper claims to fill a gap in WebGPU performance engineering by characterizing dispatch overhead, a factor previously left as "future work" in state-of-the-art studies like LlamaWeb.

Optimization Strategy: For batch size 1 inference in WebGPU, the effective optimization strategy is reducing dispatch count (via fusion or amortization) rather than optimizing individual kernels.
Specification Implications: The author suggests that deeper performance gains may require changes to the WebGPU specification itself. They propose dispatch amortization mechanisms analogous to CUDA Graphs (command-graph capture and replay) to avoid re-validating and re-submitting every operation, though they acknowledge this must be weighed against browser security models that mandate per-operation validation.
Scope Limitations: The author modestly notes that their causal fusion results are specific to the tested platform (RTX 5090, Dawn, Vulkan) and batch size 1. They do not claim these results are universal for all batch sizes, noting that kernel compute begins to dominate at larger batch sizes.

In conclusion, the work establishes that single-operation benchmarks are misleading due to synchronization conflation and identifies the reduction of dispatch count as the critical path to practical, high-performance browser-based LLM inference.