Technical Summary: Efficient On-Device Diffusion LLM Inference with Mobile NPU

1. Problem Statement

While Diffusion Large Language Models (dLLMs) offer a promising alternative to autoregressive models by enabling parallel token generation, their deployment on mobile devices faces significant hurdles. Unlike autoregressive decoding, which generates tokens sequentially, dLLMs iteratively denoise masked sequences. This process introduces three specific challenges when targeting mobile Neural Processing Units (NPUs):

1. Token Commitment and Shrinking Workloads: In block-wise dLLM decoding, as denoising progresses, the number of remaining masked tokens in a block decreases. This leads to "tail steps" where the effective workload is too small to fully utilize the NPU's high-throughput tensor engines, resulting in poor hardware amortization.
2. Token Revision and KV Cache Reuse: dLLMs require iterative refinement of token states. Tokens committed early in a block may still need revision as more context becomes available. This creates sparse, irregular updates to the Key-Value (KV) cache that disrupt the dense, regular execution patterns preferred by NPUs.
3. Memory Access and Address Space Constraints: Mobile NPUs have a limited virtual address (VA) space. dLLM inference requires frequent swapping of model weights, KV caches, and activations between system memory and the NPU. This incurs costly remapping and data transfer overheads that can negate the computational benefits of NPU acceleration.

Existing solutions often rely on naive NPU offloading, which fails to address the dynamic nature of dLLM inference against the static, tile-based execution model of mobile NPUs.

2. Methodology: llada.cpp

The authors propose llada.cpp, the first NPU-aware inference framework designed to align the parallel decoding nature of dLLMs with the execution characteristics of mobile NPUs. The framework integrates three core techniques:

A. Multi-Block Speculative Decoding

To address the shrinking workload problem in late-stage decoding, llada.cpp dynamically expands the execution window.

• Mechanism: Instead of limiting the NPU forward pass to the current block, the system speculatively incorporates tokens from future blocks into the current decoding window.
• Strategy: It employs Continuous Sliding (shifting the window rightward as tokens commit) and Adaptive Lookahead (proactively extending the window to include future blocks when the current block's masked token count drops below a threshold).
• Benefit: This maintains a high density of active tokens for the NPU, ensuring that fixed overheads (kernel launches, data transfers) are amortized over a larger, productive workload without violating the original decoding semantics.

B. Dual-Path Progressive Revision

To handle the irregularity of token revisions without stalling the NPU, the framework decouples dense computation from sparse updates.

• Staged Token Stabilization: Tokens transition through three states: Invisible (masked/low confidence), Visible (concrete prediction but revisable), and Stable (reliable for long-term reuse).
• Dual-Path Execution:
  • NPU Path: Dedicated to dense denoising of the current block.
  • CPU Path: Handles sparse updates for "Visible" tokens that require revision. The CPU asynchronously recomputes confidence, refreshes KV states, and updates token identities using specialized kernels that interpret NPU-specific tiled layouts (HMX).
• Selective Logits Skipping: To further reduce CPU load, the system skips output projection and confidence estimation for tokens that have already reached the "Stable" state.
• Delayed Merge: Refreshed KV entries from the CPU are merged into the prefix cache only at step boundaries, avoiding fine-grained synchronization.

C. Swap-Optimized Memory Runtime

To mitigate the limited NPU address space and transfer overheads, the framework optimizes memory management.

• Graph-Guided Buffer Mapping: The runtime analyzes the computational graph to classify tensors as "long-lived" (weights, KV cache) or "short-lived" (activations). It uses profiled metrics to determine which objects remain resident in the NPU-visible VA space and which are mapped temporarily, minimizing fragmentation.
• Pipeline-Based Data Transfer: The system employs double-buffered staging. While the NPU executes operators, the host prepares buffer descriptors and transfers data for the next operator in a separate staging slot. This overlaps data movement and mapping costs with computation, hiding latency.

3. Key Contributions

The paper claims the following contributions:

• Identification of Mismatch: It characterizes the fundamental misalignment between static mobile NPU execution models and dynamic block-wise dLLM inference, specifically highlighting challenges in token commitment, revision, and memory access.
• llada.cpp Framework: It introduces the first NPU-aware inference framework for dLLMs on smartphones, combining algorithmic innovations (speculative decoding, progressive revision) with system-level optimizations (memory runtime).
• Comprehensive Evaluation: It validates the framework across diverse hardware platforms (Qualcomm SM8650, SM8750, SM8850) and workloads (LLaDA-8B, Dream-7B), demonstrating effective translation of dLLM parallelism into reduced latency.

4. Results

The evaluation was conducted on smartphones including the OnePlus Ace5 Pro (SM8750). Key findings include:

• Latency Reduction: llada.cpp reduces the end-to-end generation latency of LLaDA-8B by 17× to 42× compared to a CPU baseline with prefix KV cache reuse.
• Comparison to Autoregressive Models: On newer SoCs (SM8750, SM8850), llada.cpp achieves up to 3.9× speedup over an equal-sized autoregressive model (Llama-3-8B) running on the same hardware, highlighting the advantage of diffusion parallelism on mobile NPUs.
• Quality Preservation: The framework maintains generation quality comparable to CPU baselines. Ablation studies show that while multi-block speculative decoding alone can slightly reduce accuracy due to premature future-token usage, the dual-path progressive revision mechanism recovers quality by keeping early tokens revisable.
• Energy Efficiency: By providing sufficient parallel work to the NPU, llada.cpp reduces both average power and energy per request. In contrast, running autoregressive models on NPUs showed higher power consumption due to the overhead of scheduling small, sequential workloads.
• Scalability: The system successfully handles long-output runs (up to 512 tokens) and different dLLM architectures (Dream-7B), proving the robustness of the memory management and decoding strategies.

5. Significance and Claims

The paper positions llada.cpp as a critical step toward making dLLMs practical for latency-sensitive mobile applications. It argues that the "repeated parallel computation" of dLLMs, often viewed as a liability, can be converted into an advantage by leveraging the tensor-computation capabilities of modern mobile NPUs.

The authors emphasize that simply offloading dLLMs to NPUs is insufficient; success requires a co-design approach that respects the hardware's constraints (static tile shapes, limited address space) while exploiting the algorithm's strengths (parallel denoising). The work demonstrates that with the right system optimizations, dLLMs can outperform traditional autoregressive models on mobile devices, offering a path toward low-latency, privacy-preserving, and accessible AI on smartphones.