Technical Summary: Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents

Problem Statement

Existing memory systems for Large Language Model (LLM) agents predominantly rely on universal, static policies to determine what information to store. These "one-size-fits-all" approaches overlook a fundamental reality: the contexts worth storing vary significantly across different users and their specific interaction patterns.

Current systems often waste limited memory budgets on transient, one-off interactions while failing to preserve critical context for long-horizon tasks. For instance, a user engaging in "Recipe Advice" for a long-term business venture requires persistent memory of project progress, whereas another user seeking the same advice for a single meal has a transient need. Universal policies cannot distinguish these heterogeneous use cases, leading to the eviction of essential user-specific contexts and the retention of irrelevant noise. The paper posits that an ideal memory system must infer user-specific "worth-storing" contexts and selectively store them, bypassing unnecessary operations for transient sessions.

Methodology

1. PerMem-Bench: A Benchmark for Personalized Memory

To rigorously evaluate memory personalization, the authors introduce PerMem-Bench, the first benchmark designed specifically for this purpose. It features multi-year, multi-domain interaction histories across 20 diverse user personas. The dataset is constructed via a fully automated, three-stage pipeline:

• User-Specific Agent Use Profiling: The system defines a user profile as a joint configuration of domain participation and memory necessity. Using the Nemotron-Persona-USA dataset, the pipeline assigns specific attributes (participation, frequency, and memory necessity) to each of 20 domains (e.g., Travel, Coding, Recipe Advice) for every persona. Crucially, memory necessity is determined by user intent (e.g., long-term project vs. one-off query) rather than inherent domain properties.
• Life Skeleton and Timeline Construction: For domains requiring memory, the system generates a structured "life skeleton" comprising interconnected projects and events (dialogue sessions). Each event includes "reference memories" (ground truth facts like user states or project milestones) that the memory system is expected to retain. For transient domains, independent, self-contained events are generated without project dependencies. These are arranged into a coherent, multi-year timeline.
• Dialogue Generation: A dual-simulator framework synthesizes realistic dialogues. A user simulator generates context-driven utterances based on the life skeleton, while an agent simulator responds solely to user input and its internal memory, without access to the ground-truth skeleton.

The benchmark also includes PerMem-Benchd, a dynamic variant that simulates profile shifts (e.g., a long-term project concluding or a new hobby beginning) to test system adaptability.

2. Session-Level Storage Gating

The authors propose session-level storage gating as a lightweight personalization framework. Instead of modifying the underlying memory system, a gating module inspects each session (and optionally prior context) to predict whether it is part of a long-horizon task or a transient interaction. If classified as transient, memory operations (storage, update, deletion) are skipped entirely, conserving the memory budget for high-value sessions.

Three gating methods are evaluated as baselines:

• Greedy: Predicts based solely on the current session's dialogue.
• Context-aware: Uses a sliding window of recent session summaries to capture sequential patterns.
• Structure-aware: Explicitly models the relational structure of sessions, clustering them into projects or isolated interactions to infer domain-level usage patterns.

3. Evaluation Protocol

Performance is measured using Memory Retention Rate (RR), which quantifies how consistently reference memory units are preserved in the memory bank throughout their required lifespan. The evaluation compares the gating methods against:

• Universal: No gating (standard baseline).
• Oracle: Perfect knowledge of which sessions are worth storing (upper bound).

Experiments utilize three existing memory systems (Mem0, Memory-R1, RMM) and two backbone LLMs (Qwen3-14B, gpt-5-mini).

Key Results

1. Potential of Personalization: Under "Oracle" conditions (perfect gating), personalized memory systems yield substantial retention gains over universal policies. These gains are most pronounced under tighter memory budgets (e.g., 100–200 entries), where avoiding wasteful storage on transient sessions is critical.
2. Limitations of Current Gating: While the Structure-aware method shows promise in recovering domain-level usage patterns (achieving up to 0.844 F1 on static data), current gating methods are not yet accurate enough to fully realize the benefits of personalization.
   • Greedy and Context-aware methods often underperform the Universal baseline due to high false negative rates (missing worth-storing sessions).
   • Structure-aware methods achieve higher accuracy but still yield only marginal improvements over Universal in practice.
3. Sensitivity to Profile Shifts: Structure-aware methods struggle significantly when user profiles shift (e.g., a long-term project ending). They tend to over-rely on established project structures, continuing to classify sessions as "worth-storing" even after the usage pattern has fundamentally changed. In contrast, session-level methods (like Greedy) remain more robust to such shifts but lack the ability to infer complex long-term structures.

Significance and Contributions

The paper makes three primary contributions:

1. Formalization of the Need: It identifies and formalizes the critical gap in current memory systems: the failure to account for heterogeneous user-specific patterns, leading to resource misallocation.
2. PerMem-Bench: It presents the first benchmark specifically designed to evaluate memory personalization, featuring diverse personas and multi-year, multi-domain dialogues generated via a validated, automated pipeline.
3. Empirical Study on Personalization: It provides the first empirical study on memory personalization, proposing session-level storage gating as a novel paradigm. The study establishes that while personalization offers significant theoretical gains, the primary challenge lies in developing accurate gating mechanisms that can dynamically infer user intent and adapt to changing profiles.

The authors conclude that while the potential for personalized memory is substantial, the field must focus on improving session-level gating accuracy to bridge the gap between current baselines and the theoretical upper bound of the Oracle.