Technical Summary: Lever – Speculative LLM Inference on Smartphones

Problem Statement

The deployment of high-quality Large Language Models (LLMs) on smartphones is fundamentally constrained by limited DRAM capacity. While state-of-the-art models contain billions of parameters, commodity smartphones typically offer only ~10 GB of DRAM, much of which is shared with the OS and other applications. Although offloading model weights to flash storage allows for larger models, it introduces a severe performance bottleneck: autoregressive decoding requires repeated model invocations, and loading weights from flash for every token generation results in prohibitive I/O latency.

Existing speculative decoding methods, which use a small draft model to propose tokens and a larger target model to verify them, are ill-suited for this environment. They are designed for server-class hardware where both models reside in high-bandwidth memory and verification exploits abundant parallelism. On mobile devices, these methods fail because:

1. Flash-Bound Communication: Verification requires loading target weights from flash, making I/O latency the dominant cost rather than computation.
2. Parallelism-Limited Computation: Mobile NPUs offer significantly less parallelism than server GPUs, making the verification of large token trees computationally expensive.
3. Irregular Execution: Token-tree speculation introduces irregular branching and dynamic structures that do not align well with the fixed, regular tensor operations optimized for mobile NPUs.

Methodology: The Lever System

The authors present Lever, an end-to-end system that jointly optimizes the three stages of speculative decoding (drafting, verification, and execution) specifically for the DRAM-flash heterogeneous memory hierarchy and mobile CPU-NPU hardware.

1. Mobile-Optimized Drafting Construction

Lever redefines the token tree construction objective. Instead of maximizing the acceptance rate of draft tokens (as in server-side methods), Lever optimizes for expected output tokens per unit of speculative-cycle latency.

• Gain-Cost Modeling: Lever estimates the marginal gain (probability of a token being accepted) and marginal cost (latency of draft generation + target verification) for each potential tree expansion.
• Greedy Expansion: Using a greedy algorithm, Lever expands the token tree by selecting frontier nodes with the highest gain-to-cost ratio. This ensures the tree size is balanced: large enough to amortize expensive flash I/O, but small enough to avoid overwhelming mobile compute resources with excessive verification overhead.
• Adaptive Stopping: Expansion stops when the marginal gain of the next best candidate falls below the average gain rate of the current tree, avoiding fixed budget constraints.

2. Predictor-Based Verification Pruning

To mitigate the compute bottleneck of verifying large token trees on mobile SoCs, Lever introduces an early-exit mechanism.

• Intermediate Predictor: A lightweight neural network predictor is attached to an intermediate layer of the target model. It scores draft branches based on hidden states to predict which branches are likely to survive final verification.
• Shadow Candidates & Normalization: To ensure scoring reliability even when a parent node has few children, Lever includes "shadow candidates" (draft tokens not in the tree) in the normalization process. This prevents trivial high scores for single-child nodes.
• Conservative Pruning: Branches with low predicted scores are pruned before traversing the remaining target layers. A "backbone path" is preserved to ensure the tree maintains depth, and safeguards prevent over-aggressive pruning that would discard valid sequences. This reduces verification computation while maintaining output correctness, as final token acceptance is still determined by the full target model.

3. Hardware-Hybrid Execution Acceleration

Lever restructures the execution flow to maximize the utilization of both the CPU and NPU.

• Batch-Aware Drafting: While drafting is traditionally CPU-bound, Lever identifies that as the token tree grows, multiple high-value frontier nodes can be expanded simultaneously. It schedules these expansions in batches on the NPU to improve throughput, while handling small, dynamic expansions on the CPU.
• Layer-Wise Separation in Verification: Lever decouples the transformer layers from the output projection.
  • NPU: Executes the transformer layers for all preserved nodes in a batched manner.
  • CPU: Performs output projection (logits generation) on-demand only along the actual accepted path. This avoids the redundant computation of calculating logits for branches that are ultimately rejected, a significant source of wasted cycles on mobile NPUs with fixed batch shapes.

Key Contributions

1. System-Level Analysis: The paper identifies that speculative decoding on mobile devices requires a shift from algorithmic optimization to algorithm-system co-design, specifically addressing the trade-offs between flash I/O latency, limited parallelism, and irregular execution patterns.
2. The Lever System: A novel system integrating three optimized components:
   • A gain-cost objective for mobile-aware token tree construction.
   • A predictor-based pruning mechanism to reduce verification compute.
   • A CPU-NPU hybrid execution strategy that minimizes redundant computation and maximizes hardware utilization.
3. Comprehensive Evaluation: Extensive experiments demonstrating Lever's effectiveness across multiple models (Llama-3.1-8B, Qwen3 variants), tasks (code, math, dialogue), and hardware platforms (Snapdragon 8 Gen 3, 8 Elite).

Results

Evaluations were conducted on commercial smartphones (OnePlus 12, Honor 500 Pro, Honor Magic V5) using Q4_0 quantized models.

• Latency Reduction: Lever reduces inference latency by an average of 2.93× compared to baseline flash-offloaded autoregressive inference (Flash-AR).
• Improvement over Speculative Decoding: Lever achieves a 1.50× speedup over conventional speculative decoding methods (Chain-SD) and 1.27× over EdgeLLM, a mobile-specific baseline.
• Efficiency: The system successfully narrows the latency gap between flash-backed and memory-resident inference. Ablation studies confirm that all three components (drafting optimization, pruning, and hybrid execution) are necessary to achieve these gains.
• Robustness: Lever maintains performance across varying DRAM budgets (0% to 100% of layers in DRAM), different draft model qualities, and varying output lengths.

Significance

The paper claims that Lever demonstrates the viability of running high-quality, large-parameter LLMs directly on smartphones without relying on cloud servers. By treating speculative decoding as a systems problem rather than just an algorithmic one, Lever effectively manages the constraints of mobile storage hierarchies and heterogeneous compute units. This enables interactive, privacy-preserving, and cost-effective LLM applications (such as typing assistance and interactive editing) that were previously impractical due to memory and latency limitations.