Technical Summary: MEMARENA

Problem Statement

The rapid adoption of agentic AI necessitates robust capabilities for retaining and acting upon personal memory. However, existing benchmarks for conversational memory fail to adequately evaluate on-device personal memory assistants due to three structural gaps:

1. Perspective: Most benchmarks utilize an "omniscient" transcript where all information is globally visible to the model. Real-world personal agents (e.g., a specific user's assistant) operate under ego-centric constraints, only observing conversations the user participated in or witnessed.
2. Global Coherence: Existing datasets often consist of isolated, task-based episodes lacking a persistent social graph, shared timeline, or consistent behavioral constraints (e.g., social norms regarding time of day or relationship dynamics).
3. Activity Density: Current benchmarks provide sparse data compared to human-level interaction volumes. A typical person generates ~10K conversational tokens daily; existing benchmarks often fall orders of magnitude short, failing to test persistent memory capacity, interference, and temporal provenance at scale.

Furthermore, there is a lack of evaluation for permission-aware access, where agents must not only recall information but also determine who is authorized to receive it based on implicit social norms and explicit access rules derived from interaction history.

Methodology

To address these gaps, the authors introduce MEMARENA, the first ego-centric benchmark designed for on-device personal memory agents.

1. Data Generation: MASIM Simulator

MEMARENA utilizes MASIM, an evaluation-aware multi-agent simulator, to generate a synthetic, world-grounded corpus.

• World Construction: MASIM simulates a shared social world with 50 agents over 15 days. Agents possess RIASEC-typed personalities, structured memories, and persistent social ties on a Dunbar-layered graph.
• Interaction Streams: The simulator generates two complementary dialog streams:
  • Person-Person (PP): Builds the social world through scheduled events, groups, and locations, creating source-attributed evidence under partial observability.
  • Person-Agent (PA): Models the user-assistant interface, including activity narration, memory reflection, and memory probes.
• Temporal Batching: Time is discretized into daily intervals. Sessions within a day share a fixed memory snapshot, with updates committed only at day boundaries to ensure temporal consistency and prevent causal leakage.
• Scale: The resulting MEMARENA-L dataset contains 10.3M dialog-text tokens, averaging ~24.1K text-only ego-observed tokens per agent per day.

2. Benchmark Design

MEMARENA defines six evaluation dimensions across three categories, with ground truth derived directly from the simulator's logged world state (eliminating the need for manual annotation):

• Memory Recall:
  • D1 Cloze Fidelity: Filling blanked dialogue content.
  • D2 Metadata Completeness: Recovering speaker, timestamp, and participant metadata.
• Memory Reasoning:
  • D3 Factual QA: Answering standard, adversarial (counterfactual), and temporal questions.
  • D4 Cross-Session Reasoning: Detecting conflicts and resolving anaphora across sessions.
• Memory Trustworthiness:
  • D5 Calibrated Abstention: Refusing fabricated-premise queries while answering grounded ones.
  • D6 Permission-Aware Access: Respecting access rules (e.g., refusing to disclose private info to unauthorized third parties).

3. Experimental Setup

The authors evaluated five open-weight reader models (Qwen3-0.6B, Qwen3-8B, Qwen3-32B-AWQ, Llama-3.2-3B, Mistral-7B) paired with five memory backends:

• Vanilla: Context-window only (no retrieval).
• BM25-RAG: Keyword-based retrieval.
• Oracle: Retrieval of exact ground-truth evidence sessions (diagnostic upper bound).
• Memobase: Structured memory with schema-extracted profiles.
• MemSearch: Hybrid index (markdown + Milvus-Lite) over raw session chunks.

Experiments were conducted on an NVIDIA Spark GB10 edge node to measure both accuracy and on-device latency (Time-to-First-Token, TTFT).

Key Results

1. Memory Backend Choice > Reader Scale

The choice of memory backend has a significantly larger impact on content accuracy (Recall and Reasoning) than scaling the reader model size.

• Backend Impact: At the smallest model size (Qwen3-0.6B), switching from Memobase to MemSearch improved Recall by +32.5 percentage points (pp) and Reasoning by +19.2 pp.
• Model Scaling: Scaling Qwen3 from 0.6B to 32B under the same MemSearch backend yielded only +10.6 pp (Recall) and +6.8 pp (Reasoning).
• Oracle Gap: Even with the Oracle backend (perfect retrieval), the largest reader (Qwen3-32B) lagged behind the Oracle ceiling by significant margins in cross-session reasoning, indicating that evidence selection is the primary bottleneck, not just model capacity.

2. Universal Failure in Permission-Aware Access

Current memory backends universally fail to correctly gate information disclosure based on permissions.

• The Trade-off: Systems either fail to retrieve the secret (leading to "I don't know" responses) or retrieve it and leak it. The "ideal" corner of retrieving and gating is empty.
• Performance: No deployable backend exceeded an F1PU score of 44.
  • Oracle: Leaked secrets heavily when permission was denied (high utility, low privacy).
  • Deployable Backends (MemSearch, RAG): Were too timid, withholding answers even when permission was granted (high privacy, low utility), often due to a lack of cross-session context or metadata to make confident permission judgments.
• Model Scale: Larger models tended to disclose more information but were not better at making permission-aware judgments; they were simply better at extraction.

3. Latency Overhead is Negligible on Edge

On the NVIDIA Spark GB10 edge node, memory search adds a fixed, moderate cost that constitutes a small fraction of the total Time-to-First-Token (TTFT) for most combinations.

• Search Latency: BM25-RAG adds ~87ms, Memobase ~7ms, and MemSearch ~48ms.
• Impact: For larger models (e.g., Qwen3-32B), the search overhead is negligible (e.g., <4% of total TTFT). Only the smallest model (Qwen3-0.6B) paired with the slowest backend (BM25) saw search time occupy ~54% of the TTFT.
• Conclusion: Memory backends are accuracy-critical but relatively resource-light components for improving agentic capability on personal edge devices.

Significance and Claims

The paper claims that MEMARENA reframes the design space for open-source memory systems. The authors argue that:

1. Evidence Selection is Paramount: The primary lever for improving agent performance is not increasing model scale or context window size, but rather improving the matched-evidence selection provided to the model.
2. Access Control is a Critical Failure Mode: Existing systems are inadequate for long-horizon, ego-centric context extraction, particularly regarding permission-aware access gating. The inability to distinguish between "I don't know" and "I can't tell you" remains a significant unsolved challenge.
3. Feasibility on Edge: Despite the complexity of managing dense, multi-day personal histories, the computational overhead of memory search is manageable on modern edge AI platforms, shifting the focus to backend accuracy rather than hardware constraints.

The authors note limitations, including the synthetic nature of the corpus (English-only, lacking real-world conversational noise) and the reliance on LLM-generated ground truth for factual QA, though they report high agreement between LLM judges and human annotators. They conclude that while current backends are insufficient, the benchmark provides a necessary framework for evaluating and improving the next generation of personal memory assistants.