Technical Summary: MEME (Multi-entity & Evolving Memory Evaluation)

Problem Statement

As Large Language Model (LLM) agents increasingly operate in persistent, multi-session environments, they must accurately store, update, and reason over information that evolves over time. While existing benchmarks evaluate memory systems on static fact retention or independent entity updates, they fail to address dependency reasoning. In real-world interactions, an update to one entity often triggers a "ripple effect," requiring the agent to propagate changes to logically dependent facts or recognize when a previously valid answer has become uncertain due to a missing update rule.

Current memory benchmarks (e.g., RULER, LoCoMo, MemBench) treat updates as isolated, single-entity events. They do not evaluate:

1. Cascade: Inferring that a dependent entity's value must change based on a stated dependency rule and an upstream update.
2. Absence: Recognizing that a dependent entity's value is now uncertain when an upstream entity changes but no replacement rule exists.
3. Deletion: Verifying that a removed fact is no longer reported, rather than persisting as a stale value.

This leaves a critical blind spot in evaluating how memory systems handle stateful, interdependent knowledge.

Methodology

The MEME Framework

The authors propose a principled evaluation taxonomy organized along two orthogonal dimensions:

1. Entity Scope: Single-entity vs. Multi-entity.
2. Temporal Dynamics: Static vs. Evolving.

This framework defines six distinct task types distributed across the resulting quadrants:

• Retrieval (Static, Single/Multi): Exact Recall (verbatim reproduction) and Aggregation (combining scattered static facts).
• State Management (Evolving, Single/Multi): Tracking (reconstructing revision history) and Deletion (verifying removal).
• Dependency Reasoning (Evolving, Multi): Cascade (propagating changes via rules) and Absence (identifying uncertainty).

Dataset Generation

The MEME dataset consists of 100 controlled episodes (50 per domain: Personal Life and Software Project) generated from a Directed Acyclic Graph (DAG) knowledge graph.

• Structure: Each domain uses a hand-crafted DAG where nodes are entities and edges represent conditional dependency rules (e.g., "If the team lead changes, the code reviewer will be X").
• Episodes: Each episode involves a chronological sequence of conversations (~35K tokens) interleaved with filler sessions to simulate realistic noise.
• Verifiability: Gold answers for dependency tasks are computed algorithmically by propagating updates through the DAG. If a rule exists, the new value is derived; if not, the answer is "Uncertain."
• Validation: An in-context baseline (feeding gold facts directly to the LLM) confirms the tasks are solvable in principle (0.91 accuracy with Claude Opus 4.7).

Experimental Setup

The study evaluates six memory systems spanning three architectural paradigms:

1. Raw Retrieval: BM25 and text-embedding-3-small.
2. LLM-Processed Memory: Mem0 (fact extraction) and Graphiti (temporal knowledge graph).
3. File-Based Agents: Karpathy Wiki and MD-flat (minimal single-file baseline).

All systems use gpt-4.1-mini as the internal LLM (for ingestion/retrieval planning) and the answering LLM, ensuring a fair comparison of architectural differences. The evaluation employs a "trivial-pass" filter for dependency tasks, requiring the system to correctly recall the pre-change state before correctly handling the post-change state.

Key Results

1. Universal Failure on Dependency Reasoning

Under default configurations, all six systems collapse on dependency reasoning tasks:

• Cascade Accuracy: 3% average.
• Absence Accuracy: 1% average.
• Deletion Accuracy: 17% average.

In contrast, static tasks (Exact Recall, Aggregation) show significantly higher performance. The failure is consistent across all three architectural paradigms.

2. Diagnosis of Failure Modes

The authors trace the failure to the retrieval and answering stages, not encoding or maintenance:

• Retrieval Failure: In raw retrieval and tool-based systems, the "change event" (the update) is either out-ranked by the pre-change value (in vector retrievers) or never surfaced (in graph/tool-based systems).
• Answering Failure: Even when the change event is retrieved, the answering LLM often fails to propagate the dependency, defaulting to the pre-change value.
• Structural Gap: The gap is not due to a lack of instructions or noise; prompt optimization (DSPy SIMBA), increased retrieval depth (top-k), and noise reduction failed to close it.

3. The Cost of Closure

The only configuration that partially closes the gap is MD-flat paired with Claude Opus 4.7 as the internal LLM.

• Mechanism: Opus 4.7 actively restructures the memory file at ingestion, writing explicit "Contingency" entries. When an upstream change occurs, it scans for these contingencies and writes the propagated value directly into the store, allowing the retriever to surface the correct value immediately.
• Performance: Cascade accuracy rises to 32%, and Absence to 59%.
• Cost: This configuration costs approximately 70× the baseline cost (due to Opus 4.7's high token pricing and complex processing). It also degrades performance on Exact Recall and Tracking tasks.

Significance and Claims

The paper argues that current memory systems are insufficient for real-world agentic applications where knowledge is interdependent. The key contributions are:

1. A New Evaluation Standard: MEME introduces the first benchmark to score Cascade, Absence, and Deletion tasks, exposing a critical blind spot in existing memory research.
2. Empirical Evidence of a Structural Gap: The study demonstrates that dependency reasoning is not merely a retrieval noise issue but a fundamental architectural limitation. Practical-cost systems (using smaller, cheaper LLMs) fail to propagate updates or recognize uncertainty, regardless of prompt engineering or retrieval depth.
3. Feasibility vs. Practicality: While closure is theoretically possible (demonstrated by the Opus 4.7 configuration), it currently depends on configurations that are not practical at scale due to prohibitive costs and performance trade-offs on other tasks.
4. Future Direction: The authors suggest that the path forward lies in memory architectures that natively propagate updates through dependent facts during the maintenance phase, rather than relying on costly internal LLMs to perform this reasoning at inference time.

The paper concludes that for dependency-heavy workloads, no practical-cost option exists today, and developers must rely on careful upstream design (e.g., explicit dependency rules in logs) as a near-term workaround.