Technical Summary: From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG

Problem Statement

The rapid emergence of personal AI agents based on Large Language Models (LLMs) necessitates on-device deployment to ensure privacy and responsiveness. However, these agents face a critical bottleneck: handling the inherently personal and context-dependent nature of real-world requests under strict memory budgets.

Traditional Retrieval-Augmented Generation (RAG) approaches often rely on indiscriminately storing all incoming data (e.g., conversation histories, web footprints, static documents) to ensure retrieval coverage. On personal devices, this approach is infeasible due to limited storage. Furthermore, standard retrievers optimize for query-text similarity rather than user preferences, leading to retrieved content that may be factually correct but misaligned with the user's constraints (e.g., recommending seafood to a user with an allergy). The core challenge identified is determining what to store in the first place to create a compact, preference-aligned memory that remains effective under tight resource constraints.

Methodology: EPIC

The authors propose EPIC (Efficient Preference-aligned Index Construction), a framework designed to build compact, preference-aligned on-device memory from raw data. EPIC shifts the focus from "how to use personal information at retrieval time" to "what to store" by integrating user preferences throughout the RAG pipeline via three core components:

1. Semantic-Based Coarse Filtering

This stage performs high-recall pruning of preference-irrelevant content.

• Mechanism: Data items ($x$) and user preferences ($P$) are embedded into a shared semantic space using a sentence encoder (e.g., Contriever).
• Process: The system computes cosine similarity between item embeddings and preference embeddings. Items are retained only if they match at least one preference above a threshold $\tau$.
• Goal: Rapidly discard clearly irrelevant items without requiring expensive model inference, significantly reducing the candidate set for subsequent stages.

2. Preference-Aligned Fine Verification

This stage bridges the gap between approximate latent similarity and explicit semantic relevance.

• Decision Module (DM): A language model verifies whether a candidate item genuinely aligns with the user's preferences. It outputs a structured decision (Keep or Discard) and a rationale. Items marked Discard are removed.
• Instruction Generator (IG): For items marked Keep, the IG synthesizes preference-conditioned anchor instructions. These are concise directives explaining how and when the content should be used given specific preferences.
• Memory Construction: The on-device memory is indexed not by raw text, but by the embeddings of these generated instructions. Each memory entry stores the instruction embedding, the instruction text, the associated raw item, and the preference metadata.

3. Preference-Guided Query Steering

This component ensures that retrieval queries are aligned with the preference-conditioned memory.

• Mechanism: Given a user query $q$, the system identifies the most similar preference $p^*$ in the embedding space.
• Steering: The query embedding is shifted toward the preference direction: $\tilde{q} = \text{normalize}(E(q) + E(p^*))$.
• Retrieval: The steered query $\tilde{q}$ is used to perform nearest-neighbor search over the instruction embeddings in the FAISS index, retrieving the top-k instructions which point to the grounded, preference-aligned raw items.

Key Contributions

1. Shift in Focus: The paper argues for a fundamental shift in on-device RAG from optimizing retrieval mechanisms to optimizing memory construction (deciding what to store) under tight budgets.
2. EPIC Framework: Introduction of a three-stage pipeline (Coarse Filtering, Fine Verification, Query Steering) that constructs instruction-centric memory, reducing storage to preference-relevant data only.
3. New Benchmarks: The authors introduce three new benchmarks to evaluate preference-aligned memory beyond conversational history:
   • PrefWiki: Static knowledge corpora (Wikipedia) for recommendation tasks.
   • PrefRQ: Subjective debate questions.
   • PrefELI5: Noisy, web-derived digital footprints for explanation tasks.
   • These are built upon the PrefEval protocol, which uses LLM-based judges to evaluate preference-following accuracy.
4. On-Device Validation: Experiments conducted on edge devices (Jetson Orin Nano, MacBook Pro M4, Galaxy Z Flip 6) demonstrating practical deployability.

Results

EPIC was evaluated across four benchmarks (PrefWiki, PrefRQ, PrefELI5, PrefEval) against standard RAG, indexing-enhanced RAG (e.g., RAPTOR, HippoRAG), and preference-conditioned RAG baselines.

• Memory Efficiency: EPIC reduces indexing memory usage by 2,404× compared to the best-performing baseline (NV-Embed-v2). In on-device experiments, EPIC maintains a memory footprint of under 1 MB.
• Accuracy: EPIC improves preference-following accuracy by 20.17 percentage points on average compared to the best baseline (NV-Embed-v2).
• Latency:
  • Retrieval: Achieves 33.33× lower retrieval latency than the best baseline. In on-device streaming, retrieval latency is 29.35 ms/query, with the query steering overhead being only 0.18 ms.
  • Indexing: Reduces indexing latency compared to LLM-augmented indexing methods by limiting the number of items passed to the LLM for fine verification.
• Robustness: In streaming experiments with stochastic preference drift, EPIC maintains high accuracy and constant memory usage, whereas baselines that store all data exhibit linearly growing memory usage and declining performance.

Significance and Claims

The paper claims that EPIC establishes a solid technical foundation for privacy-preserving on-device personal AI agents. By prioritizing the storage of information relevant to user preferences and constraints, EPIC enables agents to remain effective under tight resource budgets without compromising on personalization quality.

The authors emphasize that their approach is distinct from:

• Parametric Personalization: EPIC uses non-parametric memory construction rather than per-user model fine-tuning.
• Memory-Side Personalization: Unlike methods that assume memories are already available, EPIC addresses the construction stage from raw, heterogeneous data.
• Index Compression: While compression methods (e.g., vector quantization) reduce the size of existing indexes, EPIC reduces the memory footprint at the content level by deciding which items to store in the first place.

The work suggests that preference-aligned memory construction is a viable direction for bringing personalized RAG from server-side settings to resource-constrained personal devices, balancing the trade-off between volume (storing everything) and value (storing what matters).