🤖 AI

Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

This paper reveals that while Physical AI's batch-1 LLM inference is memory-dominated, faster GPUs suffer from disproportionate launch-side overheads that prevent proportional latency gains, and that standard quantization methods often fail to achieve expected speedups unless paired with highly optimized kernels like GPTQ+ExLlamaV2.

Original authors: Josef Chen

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Josef Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Fast Car" vs. The "Traffic Jam"

Imagine you are trying to drive a car from point A to point B. You have two types of cars:

The L4: A reliable, standard sedan.
The H100: A hyper-fast Formula 1 race car with a massive engine.

The paper asks a simple question: If you are driving alone (Batch-1) and only need to move a tiny amount of luggage (a single robot or user session), which car gets you there faster?

The common belief was: "The Formula 1 car (H100) is 11 times faster because it has a much bigger engine (memory bandwidth). It should win every time."

The paper's finding: The Formula 1 car actually loses, or at least doesn't win by much. Why? Because the race car is so fast that the time it takes to start the engine and put it in gear (CPU launch overhead) becomes the biggest delay. On the slower sedan, the engine is the bottleneck, so starting it doesn't matter as much.

The Core Problem: "The Launch Tax"

In the world of AI, when a computer generates one word (token) at a time, it has to do a specific set of tasks over and over again.

The Old View: The speed depends entirely on how fast the computer can read its memory (like how fast a librarian can run to the shelves to grab books). The H100 has super-fast shelves, so it should be instant.
The New View: The speed depends on how many times the computer has to say, "Okay, start this task!" to the hardware.

The Analogy:
Imagine a delivery driver (the GPU) who needs to drop off a package.

On the L4 (Slow Driver): The driver spends 20 minutes driving to the house and 1 minute parking. The drive is the bottleneck.
On the H100 (Fast Driver): The driver can drive to the house in 1 minute. But, every time they drop off a package, they have to spend 30 seconds walking from the truck to the door, signing a form, and walking back.
- Because the drive is so fast, that 30-second "walk to the door" (the Launch Tax) becomes the main reason the delivery is slow.
- The H100 is so powerful that it sits idle, waiting for the driver to finish the paperwork.

The Experiment: "The Graphs" Fix

To prove this, the researchers tried a trick called CUDA Graphs.

The Analogy:
Instead of the driver asking, "Can I start? Okay, go. Can I start? Okay, go," for every single package, they write down a master script (a graph) that says: "Drive, Park, Drop, Return, Repeat." They hand this script to the driver once, and the driver just follows the script without asking for permission every time.

The Results:

On the H100 (Race Car): This script made a huge difference. The car sped up by 26%. This proved that the "paperwork" (launch overhead) was indeed the problem.
On the L4 (Sedan): The script made almost no difference (only 3% faster). This is because the sedan was already spending most of its time driving (reading memory), not waiting for paperwork.

The "Inverted" Cost Ladder

Here is the most surprising part of the paper.

Usually, companies think: "If I want to save money, I should buy the cheapest, slowest chip (L4). If I want speed, I buy the expensive, fast chip (H100)."

The paper says: For single-stream AI (like a robot talking to you), this is backwards.

The H100 is expensive and fast, but it wastes a lot of its speed on "paperwork."
The L4 is cheap and slow, but it uses 100% of its speed on the actual work.

The "Magic" Trick:
The researchers found that if you take the cheap L4 and use a specific type of software "compression" (called ExLlamaV2), the L4 becomes almost as fast as the H100.

H100 with tricks: 11.78 milliseconds per step.
L4 with tricks: 17.36 milliseconds per step.

Even though the H100 is 11 times more powerful on paper, the L4 (with the right software) is only about 1.5 times slower, but it costs 10 times less to run.

Summary of Key Takeaways

Fast isn't always faster: Just because a chip has a bigger "memory highway" (bandwidth) doesn't mean it will finish the job faster if the "start-up time" (launch overhead) is too long.
The Bottleneck shifts:
- On cheap chips (L4), the bottleneck is moving data (memory bandwidth).
- On expensive chips (H100), the bottleneck is starting the tasks (CPU launch overhead).
Software matters more than hardware: For these specific "one-at-a-time" AI tasks, choosing the right software (like ExLlamaV2) on a cheap chip is a better deal than buying the most expensive chip.
Who is this for? This applies to Physical AI: robots, self-driving cars, and personal assistants that talk to you one sentence at a time. It does not apply to chatbots that talk to thousands of people at once (batch processing), where the rules are different.

In short: If you are building a robot that needs to think and talk to you in real-time, don't just buy the most expensive supercomputer. Buy a cheaper computer and tune the software to stop it from wasting time on "paperwork." You'll get better performance for less money.

Technical Summary: Memory-Bound but Not Bandwidth-Limited: The Physical AI Inference Gap in Batch-1 LLM Decode

Problem Statement

Physical AI systems (robots, autonomous vehicles, edge copilots) typically operate under a single-stream, batch-1 autoregressive decode workload. Unlike cloud serving which optimizes for throughput via large batches, physical AI waits on the next token for a single user or agent. The prevailing industry assumption is that this workload is strictly HBM-bandwidth-bound, meaning latency scales inversely with peak memory bandwidth ( $B_{peak}$ ). Consequently, the deployment ladder is assumed to be $L4 \to L40S \to A100 \to H100$ , where higher bandwidth directly yields lower latency and cost-per-token.

This paper challenges that assumption, arguing that for batch-1 decode on fast silicon (e.g., H100), the bottleneck shifts from memory bandwidth to per-kernel CPU launch overhead. The authors posit that faster memory does not translate to proportional latency gains because the fixed cost of launching kernels becomes the dominant fraction of the step time on high-bandwidth GPUs, while slower GPUs remain truly bandwidth-bound.

Methodology

The study is a controlled measurement experiment, not a theoretical model, conducted on Modal cloud hosts using four NVIDIA GPUs: H100 SXM5, A100-80GB SXM4, L40S, and L4.

Workload: Three 7–8B parameter GQA transformers (Qwen-2.5-7B, Mistral-7B-v0.3, Llama-3.1-8B) with bf16 weights and SDPA attention.
Scope: 44 valid cells covering 4 context lengths (2048 to 16384) and 4 GPUs.
Metrics:
- $R_{floor}$ : The ratio of the analytic memory floor ( $t_{floor} = (W+K)/B_{peak}$ ) to the observed step time ( $t_{obs}$ ). A value of 1.0 indicates perfect bandwidth utilization.
- CUDA Graphs A/B Test: A falsification experiment comparing "eager" execution against "graphed" execution (captured DAG replay) to isolate launch overhead.
- Software Stack Matrix: Comparisons of attention backends (SDPA, FlashAttention-2/3, FlashInfer) and quantization paths (bf16, bnb-nf4, AutoAWQ+Marlin, GPTQ+ExLlamaV2).

Key Results

1. The Inverted Cost-Per-Token Ladder

The study reveals a stark divergence in bandwidth utilization:

L4 (300 GB/s): Achieves ~81% of peak HBM bandwidth ( $R_{floor} \approx 0.81$ ). The workload is truly memory-bound.
H100 (3350 GB/s): Achieves only ~27% of peak HBM bandwidth ( $R_{floor} \approx 0.27$ ).
Implication: The H100's 11x bandwidth advantage over the L4 does not yield an 11x speedup. Instead, the H100 is stalled by launch overhead, while the L4 is stalled by memory traffic.

2. CUDA Graphs Falsification Test

The authors isolated the mechanism by removing CPU launch overhead via CUDA Graphs:

On H100 (ctx=2048): CUDA Graphs improved latency by 1.259× (95% CI [1.253, 1.267]). This confirms that launch overhead accounts for roughly 20% of the step time.
On L4 (ctx=2048): The same intervention yielded only a 1.028× improvement.
Conclusion: The "launch tax" is the binding constraint on H100 for batch-1 decode, whereas bandwidth is the binding constraint on L4.

3. Attention Kernel Selection

Contrary to the belief that FlashAttention variants are always superior:

On H100 batch-1, PyTorch's default SDPA dispatcher (36.05 µs/layer) outperformed explicit FlashAttention-3 (79.25 µs), FlashInfer (48.20 µs), and even the FlashAttention-2 context (44.35 µs).
The binding constraint is the sequence of kernel launches, not the throughput of the attention kernel itself.

4. Quantization and Kernel Implementation on L4

On the bandwidth-bound L4, quantization alone is insufficient; the kernel implementation matters:

bf16: 62.32 ms/step.
bnb-nf4: 59.36 ms/step (negligible gain due to dequantization overhead).
AutoAWQ + Marlin: 45.24 ms/step (1.38× speedup). Marlin is tuned for Ampere (SM80), not Ada (SM89).
GPTQ + ExLlamaV2: 17.36 ms/step (3.59× speedup). ExLlamaV2's kernels are tuned for Ada (SM89), successfully realizing the theoretical bandwidth savings.
Result: An L4 with ExLlamaV2 (17.36 ms) is significantly faster than an H100 with standard eager execution (14.83 ms) and competitive with an H100 using CUDA Graphs (11.78 ms), despite the H100 having 11x the peak bandwidth.

Significance and Claims

The paper claims to demonstrate that the standard deployment ladder ( $L4 \to H100$ ) is inverted for physical AI workloads:

Launch Overhead Dominance: On fast GPUs (H100) with short contexts, the CPU launch overhead per kernel dominates step time, rendering peak memory bandwidth largely irrelevant.
Kernel Implementation > Hardware Upgrade: For batch-1 streaming decode, optimizing the kernel implementation (e.g., using ExLlamaV2 on L4) yields greater latency reductions than upgrading to a higher-bandwidth GPU.
Cost Efficiency: At Modal's published rates (May 2026), an L4 with optimized kernels serves tokens at roughly 6x lower cost than an H100 with CUDA Graphs, despite the H100's superior raw specs.
Scope Limitation: The authors explicitly state these findings apply to 7–8B GQA transformers, batch-1, bf16, and single-stream decode. They do not claim these results generalize to large batches (where launch tax diminishes), different architectures (MHA/MQA), or other hardware (AMD, Blackwell).

The paper concludes that for physical AI inference, the "memory-bound" label is incomplete; the system is often launch-bound on high-end silicon and bandwidth-bound on edge silicon, requiring different optimization levers (CUDA Graphs vs. specialized kernels) rather than a simple hardware upgrade path.