Technical Summary: SmartRAG – Native Graph-Based RAG for Mobile Devices

Problem Statement

Deploying Large Language Models (LLMs) as personal assistants on mobile devices faces a fundamental tension: the need for privacy, low latency, and offline availability versus the computational and memory constraints of edge hardware. While Retrieval-Augmented Generation (RAG) offers a solution by integrating external non-parametric memory, existing on-device approaches rely heavily on "naive vector retrieval." This paradigm struggles with complex queries requiring cross-segment evidence aggregation, entity linking, and multi-hop reasoning. Conversely, cloud-based graph RAG methods, which excel at multi-hop reasoning, depend on massive models (tens of billions of parameters) for graph construction and reasoning—resources that are unavailable on mobile devices.

The core challenge is that lightweight on-device models lack the capacity to identify unknown entities, organize long-tail concepts, or perform complex graph-based reasoning without retraining. Furthermore, relying solely on the limited parametric memory of small models is insufficient for continuous updates and knowledge-intensive tasks.

Methodology: SmartRAG Architecture

SmartRAG addresses these limitations by decomposing on-device intelligence into four coordinated modules, shifting the burden of structured memory construction from the LLM to a specialized system architecture. The framework operates entirely on commodity smartphones using a quantized 1.7B-parameter backbone.

1. Perception: EvoNER (Evolutionary Named Entity Recognition)

To handle the dynamic nature of user data without retraining the backbone LLM, SmartRAG employs EvoNER, a continually learnable named-entity recognizer.

• Span-based Detection with Character Injection: EvoNER encodes input sentences into sub-word representations and enumerates candidate spans. It augments contextual embeddings with character-level features to improve robustness against misspellings and rare surface forms.
• Reserved-Label Mechanism: To support continual expansion, the classifier is partitioned into a base head (for deployed entity types) and reserved rows for future types. New entity types are mapped to unused reserved rows without changing the output dimensionality.
• Incremental Learning: When new entity types emerge, EvoNER updates via teacher-guided distillation restricted to the base-label subspace. This allows the system to absorb unseen concepts while preserving performance on existing types.
• Three-Stage Updates: To accommodate mobile scheduling constraints, training follows a three-stage schedule: adapting only heads, adding partial encoder layers, and finally allowing brief full-model consolidation.
• Triplet Discovery: Extracted entities are categorized into coarse types. Unseen entities are clustered via agglomerative clustering on semantic vectors, with the LLM assigning labels to coherent clusters.

2. Memory: MRGraph (Multi-layer Provenance Graph)

Extracted knowledge is stored in MRGraph, a three-layer knowledge graph designed for efficient retrieval and explicit provenance.

• Layer 1 (Entity–Paragraph Graph): The paragraph is the atomic unit. Entity nodes are deduplicated and linked to source paragraphs. Relations are stored as keyed edges with observation counts to prevent uncontrolled growth.
• Layer 2 (Abstraction Layer): Semantic clusters are maintained above entities, with centroids updated via smoothing. This layer supports topic drift detection and incremental summary refinement.
• Layer 3 (Runtime Raw-Text View): At query time, retrieved paragraphs and local graph neighborhoods are assembled into a context bounded by a token budget, providing both structured and unstructured evidence.

3. Focus: Hybrid Retrieval

The Focus module retrieves evidence through a hybrid pipeline combining graph traversal, lexical matching, and dense semantic search.

• Plan-Guided Multi-hop Text Retrieval: The system decomposes questions into a retrieval plan (seed entity, relation phrases, target types). It retrieves candidate paragraphs using fused lexical and dense scores, selecting evidence via Maximal Marginal Relevance (MMR).
• Graph-Structured Augmentation: Bridge entities from the text retrieval seed a graph traversal. The system compares node concatenations against abstraction-layer summaries and uses Personalized PageRank to estimate node importance, expanding the subgraph.
• Full-Document Fallback: If initial stages yield insufficient evidence, a global retrieval over documents recovers cross-boundary references.

4. Thinking: LLM-Driven Semantic Operations

The on-device LLM is invoked sparingly for high-value semantic operations, keeping inference costs bounded:

• Labeling: Assigning names to emerging entity clusters.
• Summarization: Periodically clustering the knowledge graph and generating summaries to form a self-compressing memory loop.
• Planning: Decomposing complex questions into multi-hop retrieval plans to reduce search ambiguity.
• Answering: Generating responses conditioned on retrieved evidence, constrained to structured schemas to reduce hallucination.

Key Contributions

1. Problem Identification: The paper identifies that complex reasoning requires structured, updatable memory, but existing graph-construction pipelines are too dependent on powerful LLMs for edge sustainability.
2. SmartRAG Framework: A novel on-device RAG method that reallocates high-frequency memory writing to lightweight trainable components (EvoNER) while reserving the LLM for sparse, high-value operations.
3. EvoNER: A continually learnable entity recognizer that absorbs unseen semantic categories through lightweight incremental updates and teacher distillation, eliminating the need to retrain the backbone LLM.
4. End-to-End On-Device Pipeline: The integration of provenance-preserving structured memory and hybrid retrieval into a full pipeline demonstrated on real smartphones, achieving strong performance under strict device constraints.

Experimental Results

Experiments were conducted on four QA benchmarks (TriviaQA, Natural Questions, HotpotQA, MultiHopQA) using a quantized Qwen3-1.7B backbone running entirely on OnePlus 13 and OnePlus 15 smartphones.

• Performance vs. Larger Models: SmartRAG with a 1.7B parameter model achieved multi-hop reasoning performance competitive with models up to 18× larger (e.g., Qwen3-32B and Ministral3-14B) on Natural Questions, HotpotQA, and MultiHopQA.
• Comparison with RAG Baselines: SmartRAG significantly outperformed other retrieval-augmented methods (Naïve RAG, InstructRAG, StableRAG, TruthfulRAG) instantiated with the same 1.7B/3B backbones. For instance, on MultiHopQA, SmartRAG achieved a Correctness (Cr) of 50.17% compared to 45.66% for Naïve RAG and 19.95% for TruthfulRAG.
• Ablation Studies: Removing the MRGraph component caused the largest performance drop (Token-F1 from 0.5484 to 0.4151), confirming the critical role of structured memory. Disabling the Thinking planner and hybrid retrieval also resulted in significant degradation.
• On-Device Feasibility: While 4B models are technically executable, the 1.7B backbone remains in the "practically deployable regime" regarding memory (approx. 4–5.5 GB PSS) and latency. The 4B model's memory footprint (approx. 6.5–6.8 GB) and longer Time-to-First-Token (TTFT) make it less suitable for interactive on-device RAG.

Significance and Claims

The paper claims that SmartRAG transforms the capability for structured knowledge from a model-centric problem into a system-architecture problem. By decoupling knowledge extraction, memory organization, and reasoning support, the framework enables lightweight components to build a structured knowledge foundation that supports complex reasoning without the computational overhead of massive models.

The authors emphasize that this approach provides a practical implementation path for on-device RAGs that differs from the cloud paradigm. It demonstrates that under strict device constraints, improving memory construction and evidence organization can be at least as important as increasing parameter counts. The system achieves privacy-preserving, offline, and low-latency personal assistance by running entirely on commodity smartphones, avoiding the need to send user data to the cloud.

The paper remains modest regarding limitations, noting that the current planner is heuristic, latency is dominated by retrieval-conditioned prefilling, and evaluation is currently limited to English benchmarks. It also acknowledges that while the system builds detailed personal knowledge graphs, appropriate encryption and access control are necessary to mitigate data governance concerns if a device is compromised.