Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM

1. Problem Statement

Deploying Large Language Models (LLMs) on mobile edge devices (specifically Samsung Galaxy S24/S25 with Qualcomm SM8650/SM8750 chipsets) faces significant engineering hurdles:

• Resource Constraints: Strict limitations on memory (RAM), latency, and power consumption.
• Inflexibility vs. Adaptability: Server-side LLMs often use Parameter-Efficient Fine-Tuning (PEFT) like LoRA to adapt to specific tasks. However, on-device deployment typically requires frozen inference graphs to ensure performance guarantees.
• The Conflict: Traditional approaches require recompiling or re-quantizing the model for every new task or LoRA adapter, which is infeasible for real-time, multi-use-case mobile applications.
• Latency Bottlenecks: Generating multiple stylistic variations (e.g., formal vs. jovial) or running multiple tasks sequentially multiplies latency. Standard speculative decoding often requires separate "draft" models, which consume scarce memory.

2. Methodology

The authors propose a hardware-aware framework that enables a single frozen inference graph to dynamically handle multiple LoRA adapters and tasks without recompilation. The solution integrates four core innovations:

A. Multi-LoRA Runtime Switching (One-for-All Graph)

Instead of merging LoRA weights statically or maintaining multiple graphs, the authors treat LoRA weights as runtime inputs.

• Mechanism: The foundation LLM graph is compiled with placeholders for LoRA weights. At runtime, specific LoRA weights (for different tasks/languages) are passed as inputs alongside the token data.
• Benefit: This allows dynamic task switching (e.g., switching from "Translation" to "Summarization") instantly without reloading the model or re-compiling the graph, significantly reducing memory footprint and storage requirements.

B. Architectural & Quantization Optimizations

To fit the model on mobile NPUs (Neural Processing Units):

• Attention Parallelism: Multi-Head Attention (MHA) is decomposed into parallel Single-Head Attention (SHA) operations to better utilize heterogeneous NPU cores.
• Linear-to-Convolutional Transformation: Dense linear layers are reinterpreted as $1\times1$ convolutions. This leverages highly optimized convolution kernels (Winograd, tiling) native to mobile NPUs, which are originally designed for vision tasks.
• Mixed-Precision Quantization:
  • Weights: Quantized to INT4 (per-channel).
  • Activations: Quantized to INT8 (per-tensor).
  • Training: Uses Quantization-Aware Training (QAT) to maintain accuracy despite low precision.

C. Concurrent Token Generation (CTG)

To address the latency of generating multiple stylistic outputs (e.g., 8 different tones for a "Smart Reply"):

• Mechanism: Instead of running the model 8 times sequentially, the system uses a masked decoding scheme. It modifies the first-token sampling to drive multiple parallel streams within a single forward pass.
• Efficiency: All streams share the same KV-cache and tensor layout. This reduces latency by up to 6× for stylistic variations compared to sequential inference.

D. Dynamic Self-Speculative Decoding (DS2D)

To accelerate token generation without the memory overhead of a separate draft model:

• Mechanism: The model is fine-tuned using Prefix Tuning to become "semi-autoregressive." It appends "forecast embeddings" to the prompt, allowing it to predict $1+m$ tokens in a single step (1 verified token + $m$ draft tokens).
• Tree-Based Verification: The generated draft tokens form a tree structure. A verification step accepts or rejects these tokens based on the model's logits.
• Optimization: The branch configuration (how many tokens to predict) is optimized per use-case to maximize acceptance rates while keeping input batch sizes aligned with NPU power-of-2 constraints (e.g., 32).

3. Key Contributions

1. Runtime LoRA Switching: A novel framework enabling a single frozen graph to dynamically load multiple LoRA adapters as inputs, eliminating the need for model recompilation.
2. Hardware-Aware Architecture: Transformations (MHA $\to$ SHA, Linear $\to$ Conv) specifically tailored for Qualcomm NPUs to maximize throughput.
3. Concurrent Multi-Stream Inference: A method to generate multiple stylistic outputs simultaneously in one pass, achieving 6× latency reduction.
4. Draft-Free Speculative Decoding: A "Self-Speculative" approach (DS2D) that uses internal prefix tuning to accelerate decoding by up to 2.3× without requiring a separate, memory-heavy draft model.
5. Commercial Deployment: Successful integration of this framework into the Samsung Galaxy S24 and S25 series, supporting 9 languages and 8 distinct use cases.

4. Results

The system was evaluated on Samsung Galaxy S24 (SM8650) and S25 (SM8750) devices:

• Performance Gains:
  • Memory: Reduced peak memory usage by 4–6× compared to baseline approaches.
  • Latency: Achieved up to 6× reduction for multi-style generation and 2.3× speedup in decode time using DS2D.
  • Throughput: Increased token generation rate to 41.32 tokens/sec on the 1B model and significantly improved tokens/sec on the 3B model.
• Accuracy:
  • Maintained high accuracy across 9 languages (Korean, English, German, Spanish, French, Italian, Portuguese, Chinese, Japanese) and 8 tasks (Correction, Style, Smart Reply, Summarization, etc.).
  • Quantized (INT4/INT8) models showed Relative Accuracy (RA) often exceeding 97-100% compared to full-precision (FP32) baselines (e.g., 99.6% RA for Korean Correction).
• Scalability: Demonstrated successful scaling from a 1B parameter model to a 3B parameter model with the DS2D optimization.

5. Significance

• Commercial Viability: This work proves that complex, multi-task Generative AI can run fully on-device without cloud reliance, addressing privacy concerns and enabling offline functionality.
• Paradigm Shift: It moves away from the "one model per task" or "static merging" paradigm toward a dynamic, input-driven inference graph, setting a new standard for edge AI deployment.
• Sustainability: By optimizing for edge hardware, it reduces the energy and computational cost associated with cloud-based LLM inference.
• Future Impact: The framework provides a blueprint for deploying adaptable foundation models on resource-constrained devices, paving the way for more sophisticated on-device AI assistants.

Limitations Noted

• LoRA Dimensionality: All LoRA adapters must share identical dimensions to fit the fixed graph placeholders.
• Static Forecast Embeddings: In DS2D, forecast embeddings are static and not conditioned on evolving input sequences, which may limit acceptance rates for highly variable inputs.
• Fixed Styles: Concurrent generation relies on pre-defined stylistic variants; dynamic, user-specified styles require recompilation.