When NPUs Are Not Always Faster: A Stage-Level Analysis of Mobile LLM Inference

Technical Summary: When NPUs Are Not Always Faster

Problem Statement
The deployment of Large Language Models (LLMs) on mobile devices increasingly relies on heterogeneous execution involving both CPUs and Neural Processing Units (NPUs). However, prior research has failed to systematically characterize NPU effectiveness at the operator and pipeline levels. Existing benchmarks often provide only aggregate throughput or compare CPUs and GPUs, leaving a gap in understanding how NPUs perform under specific LLM workloads, particularly regarding the distinct computational characteristics of the Prefill (compute-intensive) and Decode (memory-bound) stages. There is a lack of data on whether offloading to NPUs actually improves latency or energy efficiency when accounting for communication, quantization, and scheduling overheads.

Methodology
The authors present a stage-aware, multi-level benchmarking framework on a heterogeneous CPU–NPU System-on-Chip (Snapdragon 8 Gen 3). The study utilizes the llama.cpp inference engine and evaluates four quantized models (Llama-3.2-3B, Llama-3.1-8B, Qwen3-4B, and Qwen3-8B).

Key methodological innovations include:

1. Stage-Level Analysis: Separating inference into the Prefill stage (batched GEMM) and the Decode stage (autoregressive GEMV) to analyze performance divergence.
2. Operator-Level Profiling: Distinguishing between op-usec (pure computation time) and call-usec (total invocation time including scheduling and synchronization) to quantify orchestration overhead.
3. OPMASK-Based Pipeline Decomposition: A controlled methodology using four execution modes to isolate and measure the latency contributions of three specific NPU pipeline components:
   • Communication: CPU–NPU request dispatch and response handling.
   • Quantization: Dynamic data format conversion (e.g., F32 to INT8).
   • Computation: Actual execution on NPU units.
     By enabling these components incrementally, the authors calculate the specific overhead of each stage without hardware modification.

Key Results
The study reveals a counter-intuitive performance reversal where NPUs do not universally outperform CPUs:

• Prefill Stage (CPU Superiority): In the compute-intensive Prefill stage, the CPU consistently outperforms the NPU by 1.27× to 1.62×. This is attributed to the NPU's limited on-chip memory (8 MiB VTCM) constraining data tiling for large matrices, less optimized HVX intrinsics compared to mature CPU GEMM libraries, and overhead from unsupported operators falling back to the CPU.
• Decode Stage (Limited NPU Gain): While the NPU shows a 1.55×–1.67× advantage on the core MUL_MAT operator (GEMV) due to efficient software-managed scratchpad memory and DMA, the end-to-end speedup is marginal (1.05×–1.20×).
• Overhead Bottlenecks: Two primary factors erode NPU benefits:
  1. Scheduling Tax: For lightweight operators (e.g., RMS_NORM), the call-usec is 8×–22× higher than op-usec due to CPU–NPU round-trip communication costs.
  2. Cross-Backend Fallback: Operators lacking native NPU support (e.g., FLASH_ATTN_EXT) fall back to the CPU with a 1.5× latency penalty due to cache coherency synchronization and tensor reordering.
• Energy Consumption: Contrary to expectations, increasing NPU offloading leads to higher total energy consumption, with battery drain increasing by up to 51% for longer prompts. This is driven by the increased execution time in the Prefill stage and the cumulative overhead of scheduling and fallbacks.
• Pipeline Decomposition: Communication overhead is negligible in Prefill (<3.2%) but surges in Decode (9.9%–13.0%) due to the high-frequency dispatch of fine-grained operations.

Significance and Design Guidelines
The paper claims to be the first study to combine competitive CPU–NPU profiling with stage-awareness, operator-level granularity, and pipeline decomposition. Based on these findings, the authors derive three design guidelines for NPU architects and system integrators:

1. Stage-Aware Backend Selection: Route the Prefill stage to multi-core CPUs (leveraging mature libraries and larger caches) and the Decode stage to the NPU.
2. Reduce Dispatch Latency: NPU dispatch latency for lightweight operators must be reduced below 10 µs. Techniques such as batch dispatch, persistent command queues, or operator fusion are necessary to mitigate the scheduling tax.
3. Improve Operator Coverage: Critical operators like FLASH_ATTN must be implemented natively on the NPU to eliminate the fallback penalty and fully realize acceleration benefits in the decoding phase.

The authors conclude that realizing efficient on-device LLM inference requires moving beyond simple offloading strategies toward stage-aware execution and improved system–hardware co-design.