Technical Summary: Toward Sustainable On-Device Intelligence

Problem Statement

The rapid proliferation of Large Language Models (LLMs) has created a tension between the computational intensity required for their training and inference and the resource-constrained reality of edge devices (smartphones, IoT gateways, embedded systems). While Retrieval-Augmented Generation (RAG) has emerged as a dominant paradigm for grounding LLM outputs in external knowledge and mitigating hallucinations, standard RAG pipelines assume cloud-scale resources (large vector indices, GPU-backed search, generous memory). Deploying RAG on edge devices introduces a complex, multi-dimensional trade-off among accuracy, latency, memory footprint, energy consumption, and carbon emissions.

Currently, research in on-device AI, RAG, and Green AI (sustainability) exists in three largely separate silos. Existing surveys address pairs of these domains (e.g., on-device AI + RAG, or Green AI + model compression) but fail to integrate all three into a unified system-design perspective. Consequently, design decisions—such as quantization bit-width, retrieval depth, and hardware selection—are often made in isolation, ignoring their interdependencies on the overall sustainability and performance of the system. There is a lack of unified evaluation frameworks that explicitly measure energy and carbon alongside traditional metrics like accuracy and latency.

Methodology

This paper employs a structured literature review approach inspired by PRISMA 2020 guidelines.

• Search Strategy: Systematic searches were conducted across five databases (IEEE Xplore, ACM Digital Library, Scopus, Google Scholar, arXiv) using Boolean queries combining terms from three domains: On-device AI (e.g., "small language model," "NPU"), RAG (e.g., "retrieval-augmented generation"), and Green AI (e.g., "energy-efficient inference," "carbon footprint").
• Selection Criteria: The review focused on peer-reviewed articles, conference papers, and well-cited preprints from 2017 to August 2026. Papers were included if they addressed at least one of the three domains and presented novel methods, systems, or benchmarks. Cloud-only LLM serving without edge or sustainability considerations was excluded.
• Corpus: The initial search yielded 1,247 papers. After deduplication and screening (title/abstract and full-text), 108 papers were selected from the systematic search. An additional 34 papers were identified through backward and forward citation chaining. The final corpus analyzed in the survey totals 142 papers.
• Classification: The literature was classified into a hierarchical taxonomy of nine categories spanning three pillars:
  1. On-Device AI: Foundation models, Small Language Models (SLMs), Quantization/Compression, NPU Hardware.
  2. RAG: Core RAG fundamentals, On-Device RAG systems.
  3. Green AI: Sustainability measurement, Evaluation/Benchmarks, Applications/Case Studies.

Key Contributions

1. Unified Synthesis of Three Pillars

The paper provides the first review to jointly cover on-device AI inference, retrieval-augmented generation, and Green AI/sustainability as a single design space. It analyzes the interdependencies between these domains, such as how quantization affects NPU throughput and energy, and how retrieval strategies impact both answer quality and computational overhead.

2. Systematic Taxonomy of 142 Papers

The authors classify the reviewed literature into nine specific categories, mapping the evolution of the field from foundational Transformer architectures to specialized NPU hardware, efficient RAG pipelines, and carbon-aware inference strategies.

3. The ALEMC Evaluation Framework

A core contribution is the proposal of ALEMC (Accuracy, Latency, Energy, Memory, Carbon), a unified evaluation framework that extends conventional performance metrics.

• Metrics: It defines formal metrics for Accuracy (harmonic mean of task score and faithfulness), Latency (Time-to-First-Token and End-to-End), Energy (Joules per query), Memory (Peak RSS), and Carbon (gCO2e per query).
• Scoring: It introduces a weighted geometric mean composite score to enable holistic comparison of system configurations, ensuring that poor performance in one dimension (e.g., high energy) cannot be masked by strong performance in another.
• Application: The framework is applied to synthesize design principles and compare different deployment patterns.

4. Integration Patterns for Sustainable On-Device RAG

The paper synthesizes three distinct design patterns that occupy different points in the accuracy-efficiency-sustainability trade-off space:

• Pattern 1: Quantized SLM + Lightweight Retriever + NPU: A fully offline, single-device configuration using 4-bit quantized SLMs (e.g., Phi-3-mini, Gemma-2B) and graph-based or lightweight retrieval (e.g., MiniRAG) to minimize memory and energy.
• Pattern 2: Speculative Edge–Cloud Hybrid: A confidence-gated architecture where a local SLM drafts responses, and a cloud-based large model verifies only low-confidence queries. This balances local privacy/efficiency with cloud-level accuracy.
• Pattern 3: Carbon-Aware Adaptive RAG: A dynamic system that adjusts runtime parameters (DVFS, LoRA adapter rank, retrieval depth) based on real-time grid carbon intensity and device battery state to optimize for sustainability.

5. Identification of Open Challenges and Roadmap

The survey identifies six critical open challenges:

1. Hardware Gaps: Current NPUs are optimized for CNNs and lack native support for variable-length sequences, sparse operations, and dynamic graphs required by LLMs/RAG.
2. Standardized Benchmarks: A lack of reproducible energy benchmarks for on-device RAG, making cross-study comparisons difficult.
3. Lifecycle-Aware Model Selection: The need to account for training carbon and retraining costs, not just inference energy, when selecting models.
4. Federated RAG: Challenges in secure index aggregation and differential privacy for collaborative on-device knowledge sharing.
5. Multi-Modal Edge RAG: The difficulty of deploying multi-modal retrieval (text, image, audio) within strict edge memory and energy budgets.
6. Policy and Regulation: The emerging need for standardized carbon labeling and energy reporting for AI systems.

Results and Empirical Observations

The paper does not present a single new experimental result but synthesizes findings from the 142 reviewed papers and includes a controlled empirical study (Section 7.3) to validate the ALEMC framework.

• Feasibility: The synthesis confirms that sustainable on-device RAG is technically achievable. Systems using 4-bit quantized SLMs (1–3B parameters) with lightweight retrieval can operate within 2–4 GB of memory on modern mobile NPUs, delivering answers in under 2 seconds.
• Energy Savings: On-device inference can reduce energy consumption by 10–100× compared to cloud inference for small models, primarily by eliminating network transfer and infrastructure (PUE) overhead. However, this advantage diminishes for larger models or high-frequency queries where embodied carbon and local grid intensity become dominant factors.
• Trade-offs: The empirical study and literature synthesis demonstrate steep diminishing returns: moving from a 4-bit 1B-parameter model to a full-precision 7B model may improve accuracy by only 3–5% while increasing energy consumption by an order of magnitude.
• Context Compression: Techniques like SparseRAG and Selective Content Reduction can reduce context length by 30–55% with negligible accuracy loss, directly lowering prefill compute and KV cache energy.

Significance and Claims

The paper claims its primary significance lies in the synthesis of three adjacent research communities into a coherent framework. It argues that the design of sustainable on-device RAG systems cannot be achieved by optimizing any single pillar (compression, retrieval, or sustainability) in isolation.

• Shift in Discourse: The work aims to shift the research question from "Can we run RAG on-device?" (which has largely been answered affirmatively) to "Can we run RAG on-device sustainably?"
• Unified Perspective: By exposing interdependencies—such as how quantization bit-width dictates memory available for retrieval indices, or how retrieval depth impacts energy consumption—the paper provides a principled basis for system design.
• Actionable Framework: The ALEMC framework and the identified integration patterns offer practitioners a concrete methodology to navigate the multi-dimensional Pareto frontier of accuracy, latency, energy, memory, and carbon.
• Future Directions: The paper outlines a research roadmap, calling for standardized energy benchmarks, NPU hardware co-design for LLMs, and the development of lifecycle-aware model selection tools to guide the field toward truly sustainable edge intelligence.