Technical Summary: DynamicMem

Problem Statement

Large Language Model (LLM) agents are increasingly deployed as personal assistants requiring the ability to maintain and update a user's profile over extended periods (months). This profile encompasses attributes (who the user is), habits (what they routinely do), and preferences (what they prefer). However, evaluating this "long-horizon memory" capability is hindered by the lack of real-world data due to privacy constraints. Existing synthetic benchmarks fall short in four critical areas:

1. Lack of Cross-App Evidence: Prior benchmarks often treat history as isolated dialogue sessions or disjointed events, failing to model how a single user intent drives coordinated behaviors across multiple applications.
2. Oversimplified User Modeling: They often treat the user profile as a monolithic, static entity, ignoring that different aspects (attributes, habits, preferences) evolve on different timescales and via different mechanisms.
3. Unmotivated Drift: Existing benchmarks either freeze profiles or allow them to drift without causal drivers (e.g., seasons, life events), failing to capture how external context triggers state changes.
4. Single-Snapshot Evaluation: They report accuracy at a single fixed history length, obscuring how performance scales as the trajectory grows and whether systems correctly overwrite superseded facts.

Methodology

The authors introduce DynamicMem, a synthetic benchmark designed to generate 15-month, fine-grained user trajectories across 16 applications (e.g., e-commerce, fitness, social media) without concatenating existing chat histories. The system employs a three-stage, top-down synthesis pipeline:

1. Multi-Timescale User Profile Construction

The user profile is decomposed into three categories with distinct dynamics:

• Attributes: Discrete facts (e.g., job title, vehicle) that change abruptly with life events.
• Habits: Medium-term routines (e.g., weekly grocery shopping) that adapt to context and may revert.
• Preferences: Long-term inclinations (e.g., health management style) that drift gradually.
  The evolution is driven by a World Background (seasonality, major events, external context) which triggers Typed Deltas (ADD, MODIFY, SHIFT, etc.). These changes are causally grounded; a single external driver (e.g., a new health initiative) can induce coupled changes across attributes, habits, and preferences simultaneously.

2. Intent-Conditioned Event Chain Generation

To ensure cross-app consistency, the system generates Event Chains. Instead of isolated events, a single user intent (e.g., obtaining a certification) unfolds as an ordered sequence of actions across different apps (e.g., reading an email, updating a LinkedIn profile, messaging a client). This ensures that entity references remain consistent across the trajectory and that every state change is supported by observable behavioral traces.

3. State-Consistent Multi-App Log Generation

Event chains are grounded into concrete API interaction logs for 16 simulated applications. Each app is a stateful environment where API calls read and update an internal state (e.g., shopping carts, playlists, message threads). This guarantees that later steps in a chain are logically dependent on earlier outputs, preventing hallucinated identifiers and ensuring behavioral consistency.

Evaluation Protocol

DynamicMem utilizes a checkpoint-based evaluation at five quarterly intervals (C1–C5) over the 15-month horizon. At each checkpoint, systems are evaluated on two task families:

• State Completion: The system must reconstruct the user's current profile (attributes, habits, preferences) from the available logs up to that point.
• Personalized Service: The system must use the recovered state to perform a downstream task (e.g., setting up a smart home device, drafting a reminder) given a scenario that omits the specific user details.

Key Contributions

1. DynamicMem Benchmark: The first synthetic benchmark providing 15-month, multi-app trajectories (averaging 2.2M tokens per user) with explicit causal drivers for profile evolution.
2. Fine-Grained Evaluation Schema: A decomposition of user profiles into attributes, habits, and preferences, probed by both reconstruction (State Completion) and application (Personalized Service) tasks.
3. Scalable Pipeline: A method to generate coherent, state-consistent logs across 16 apps and 66 APIs, ensuring that evidence for user states is distributed and implicit rather than explicit.
4. Diagnostic Analysis: A framework for analyzing memory failures by distinguishing between retrieval issues (what the memory system provides) and generation issues (how the model uses it).

Results

The authors benchmarked five representative memory systems (Vanilla RAG, HippoRAG2, A-Mem, MemoryOS, SimpleMem) against an Oracle upper bound. Key findings include:

• Divergent Scaling: As the history horizon grows, State Completion performance degrades significantly (dropping up to 16.7 points for SimpleMem from C1 to C5), while Personalized Service performance remains stable or even improves slightly. This suggests that while retrieving specific state facts becomes harder with more data, the growing corpus provides sufficient context for service-oriented tasks.
• Failure Modes by State Type:
  • Preferences drive the decline in State Completion. Because preferences are implicit and inferred from scattered behaviors, they are most susceptible to being "drowned out" by unrelated logs as the horizon expands.
  • Habits remain relatively stable in State Completion due to their repetitive nature, but they collapse in Personalized Service (dropping ~43–50 points). Current models fail to proactively select the correct routine from a scenario without explicit cues.
  • Attributes show early degradation that plateaus; systems struggle to overwrite changed facts (e.g., a new job) while retaining old ones.
• Retention vs. Update Trade-off: No system successfully retains stable facts while simultaneously overwriting changed ones. Mechanisms that preserve stable facts (e.g., recursive consolidation) often prevent the adoption of new, changed facts.
• Bottleneck Location: Over 93% of failures are traced to the memory system's retrieval (irrelevant evidence, identity misses, or conflated data) rather than the answer-generation model's reasoning. This indicates that the primary bottleneck lies in memory architecture, not the LLM's generation capabilities.

Significance

The paper claims that DynamicMem serves as a controlled testbed for diagnosing specific weaknesses in long-horizon memory architectures that aggregate accuracy scores hide. It highlights that:

• Long-term memory is not merely a retrieval problem but a dynamic state tracking problem requiring distinct strategies for retention and updating.
• Current systems fail to handle implicit preferences and proactive routine selection, which are critical for real-world personal assistants.
• The field must move beyond simple recall metrics to evaluate how systems manage causal attribution and state evolution over time.
• Improvements in memory retrieval and organization offer the most immediate "headroom" for performance gains, as the answer-generation models are currently limited by the quality of evidence they receive.