Technical Summary: Energy-Efficient On-Device RAG on a Mobile NPU

Problem Statement

The proliferation of Large Language Models (LLMs) has created a demand for on-device AI systems that preserve privacy, reduce latency, and enable offline functionality. Retrieval-Augmented Generation (RAG) is the dominant paradigm for grounding LLM outputs in external knowledge, yet deploying a complete RAG pipeline on mobile devices remains challenging. A full RAG workflow involves multiple compute-intensive neural stages: embedding generation, semantic retrieval, cross-encoder reranking, and autoregressive LLM generation. These stages collectively impose significant power and thermal constraints on mobile and laptop platforms, particularly when relying on CPU inference, which is energy-prohibitive for sustained operation. While dedicated Neural Processing Units (NPUs) are increasingly integrated into System-on-Chips (SoCs) to address this, prior work has largely focused on single-model inference (e.g., LLM prefilling) rather than end-to-end multi-model RAG pipelines. Furthermore, the energy savings and performance trade-offs of offloading a complete, multi-stage RAG pipeline to a mobile NPU have not been quantified.

Methodology

The authors present a system design and benchmark study executed on a Dell XPS 13 laptop equipped with a Qualcomm Snapdragon X Elite SoC, featuring the Hexagon NPU (45 TOPS INT8).

System Architecture:
The system implements an end-to-end RAG pipeline comprising two distinct workflows:

1. Indexing Pipeline: Transforms a document corpus into a searchable vector index. It involves document parsing/chunking (1,000-character chunks with 500-character overlap, constrained by NPU context limits), embedding generation using EmbeddingGemma 300M, and FAISS index construction.
2. Query Pipeline: Processes user queries through query rewriting (LLM), hybrid retrieval (combining SQLite FTS5 and FAISS), cross-encoder reranking using Jina Reranker v2, and answer generation using Qwen3-4B-Instruct.

Implementation Details:

• NPU Backend: All neural components (embedding, reranking, LLM) are compiled ahead-of-time into static computation graphs and executed via the Qualcomm AI Runtime (QAIRT/QNN) SDK.
• Baselines: The same model architectures and pipeline logic are compared against CPU and integrated GPU (Adreno X1-85) baselines using llm.cpp with GGUF quantization. The reranker is pinned to the NPU in all configurations due to a batched-scoring defect in the OpenCL backend.
• Engineering Constraints: The NPU implementation required specific engineering adaptations, including a strict model loading order (largest model first: LLM → Embedding → Reranker) to satisfy contiguous memory allocation requirements and reduced context windows (4,096 tokens) compared to CPU/GPU deployments.

Evaluation Metrics:

• Performance: Wall-clock time, throughput (tokens/s), and latency (Time-to-First-Token, decoding speed).
• Energy: System-level power draw and total energy consumption (Joules) measured via HWiNFO64 telemetry at 500ms granularity.
• Quality: Answer quality evaluated using GPT-4.1 as an LLM-as-judge on a 1–10 rubric, alongside failure rate and correctness rate analysis.
• Workloads: Indexing on 10 SEC 10-K filings (9,324 chunks) and query processing on 120 queries from the wiki-minirag and FinDER datasets.

Key Contributions

1. First End-to-End NPU RAG Pipeline: The authors implement what is believed to be the first system where all three neural inference stages (embedding, reranking, and LLM generation) run entirely on a mobile NPU (Qualcomm Hexagon).
2. Comprehensive Benchmarking: The study provides a granular comparison of NPU, CPU, and GPU backends for both indexing and query workloads, measuring not just speed but also per-component power and total system energy.
3. Quantitative Sustainability Analysis: The paper quantifies the "super-linear" energy benefits of NPU offloading, demonstrating that speedups translate into disproportionate energy savings due to reduced average system power draw during execution.
4. Quality Parity Validation: Through rigorous evaluation, the paper establishes that the constraints imposed by the NPU (smaller context windows, specific chunk sizes) do not materially degrade answer quality compared to CPU or GPU baselines.

Results

Performance:

• Indexing: The NPU achieved 9.1× higher embedding throughput (3,325 vs. 367 tokens/s) and a 7.9× total pipeline speedup over the CPU baseline.
• Query Processing: On the wiki-minirag benchmark, the NPU delivered 18.1× faster LLM prefilling and 4.0× lower end-to-end query latency compared to the CPU.
• GPU Performance: Contrary to expectations for x86 platforms, the integrated Adreno GPU was 1.7× slower than the CPU baseline for this workload and consumed significantly more energy.

Energy Efficiency:

• Indexing: The NPU reduced total system energy by 12.3× compared to the CPU. This was driven by both speed and a 35% reduction in average system power (27.6 W vs. 42.7 W) as CPU clusters operated near idle.
• Query Processing: The NPU reduced total system energy by 4.0× compared to the CPU and 6.5× compared to the GPU.
• Scaling: For a hypothetical daily workload of 1,000 queries, the NPU saves approximately 260 Wh per day compared to the CPU baseline.

Answer Quality:

• A GPT-4.1 judge evaluation found answer quality to be on par across all backends. The NPU scored a mean of 9.32, compared to 8.95 (CPU) and 9.03 (GPU).
• 86.7% of queries received identical scores across all three backends. Statistical analysis indicated that minor score differences were dominated by evaluator noise rather than substantive backend performance gaps.
• In the domain-specialized FinDER test, the NPU exhibited conservative refusal behavior (refusing to answer when context was insufficient) similar to the CPU, though with a lower substantive answer rate due to stricter context window constraints.

Significance and Claims

The paper claims to demonstrate that practical, energy-efficient on-device RAG is achievable on mobile NPUs without quality regression.

• Green Edge Intelligence: The study positions the Hexagon-class NPU as a sustainable path for edge AI, offering massive energy reductions (up to 12.3×) that make continuous on-device RAG viable for battery-powered devices.
• Hardware Reality Check: The results challenge the assumption that integrated GPUs are always the optimal accelerator for LLM workloads on mobile SoCs, showing that for this specific 4B parameter workload, the NPU outperforms both the CPU and the GPU in speed and energy efficiency.
• Generalizability: While the study is specific to the Snapdragon X Elite, the authors expect these findings to generalize to comparable mobile NPUs (Apple Neural Engine, Intel NPU, MediaTek APU) as their software stacks and model ecosystems mature.
• Engineering Insights: The paper highlights critical, non-trivial engineering constraints of NPU deployment (static graphs, memory allocation order, context limits) that must be addressed to realize these benefits.

The authors conclude that NPU acceleration enables a "sustainable path toward green edge intelligence," allowing for privacy-preserving, low-latency RAG systems that are energy-competitive with traditional CPU-based approaches.