EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

1. Problem Statement

Large Language Model (LLM) assistants are increasingly expected to function as persistent collaborators that retain and reason over information accumulated across weeks or months. Current approaches face a fundamental trade-off:

• Full-Context Prompting: Providing the model with the entire interaction history. While effective, this scales poorly in cost and treats memory as a flat transcript rather than a structured representation.
• Retrieval-Augmented Systems: Using vector databases or flat retrieval to fetch relevant snippets. These often struggle with cross-space integration (connecting information from distinct life domains), temporal reasoning, and emergent synthesis (deriving conclusions not explicitly stated in the text).

Existing benchmarks often fail to isolate these specific memory behaviors or conflate memory construction, retrieval, and answer generation, making it difficult to determine when structured memory offers a genuine advantage over brute-force context inclusion.

2. Methodology

EngramaBench (The Benchmark)

The authors introduce EngramaBench, a benchmark designed to evaluate long-term conversational memory under controlled conditions.

• Structure: Built around 5 canonical personas (e.g., a professor, a startup founder, an artist) spanning diverse professional and personal domains.
• Data: 100 multi-session conversations (20 per persona) and 150 queries (30 per persona).
• Query Taxonomy: The benchmark tests five distinct memory behaviors:
  1. Single Space: Factual recall within one semantic domain.
  2. Cross-Space: Integrating information across two or more distinct life domains.
  3. Temporal Cross-Space: Reasoning over time and space simultaneously.
  4. Adversarial: Questions based on false premises requiring the system to abstain.
  5. Emergent Insight: Synthesizing multiple evidence points into a higher-level conclusion.
• Scoring: Uses a composite score weighted to balance factual memory, adversarial abstention, and synthesis.

Systems Evaluated

To isolate the effect of memory architecture, all systems use the same answering model (GPT-4o):

1. GPT-4o Full-Context: Concatenates all history into the prompt (Brute-force baseline).
2. Mem0: An open-source vector-retrieval system (Flat retrieval baseline).
3. Engrama: A graph-structured memory system that organizes interactions into entities, semantic spaces, temporal traces, and cross-space associative links.

Engrama Architecture

• Ingestion: Converts conversations into a graph where nodes represent entities/events and edges represent relationships, preserving "space" (domain) and "time" as first-class properties.
• Retrieval: Uses query-conditioned activation to identify relevant sub-regions of the graph rather than retrieving isolated text chunks.
• Composition: Converts the retrieved subgraph into a structured summary for the answer generator.
• Configurable Levers: The system includes specific components (L1: Entity-first activation/planning; L3: Typed answer layer) designed to improve cross-space reasoning and answer precision.

3. Key Contributions

1. EngramaBench: A novel benchmark explicitly targeting cross-space reasoning as a first-class metric, alongside adversarial abstention and emergent synthesis, to better evaluate structured memory.
2. Controlled Comparison: A rigorous evaluation showing that while full-context prompting wins globally, structured graph memory (Engrama) outperforms it specifically on cross-space reasoning at a fraction of the cost.
3. Ablation Analysis: A detailed study revealing a "systems-level tension" where components that optimize for cross-space integration trade off against the global composite score, explaining why specialized memory architectures may underperform in broad evaluations despite excelling in specific reasoning tasks.

4. Results

Performance Metrics (Composite Scores)

• GPT-4o Full-Context: 0.6186 (Highest global score).
• Engrama (Full): 0.5367 (Second highest).
• Mem0: 0.4809 (Lowest).

Key Findings

• Cross-Space Advantage: Engrama is the only system to outperform full-context prompting on cross-space reasoning (0.6532 vs. 0.6291). This metric is considered the most diagnostic for structured long-term memory.
• Cost Efficiency: Engrama achieves ~87% of GPT-4o's global performance at roughly 20% of the query-time cost ($0.67 vs. $3.33 for 150 queries). Mem0 is cheaper ($0.36) but significantly weaker in factual precision.
• Failure Modes:
  • Temporal Reasoning: Remains difficult for all systems (GPT-4o scores only 0.3902), suggesting this is a fundamental LLM limitation.
  • Adversarial Abstention: Mem0 achieved perfect abstention (1.0), while Engrama scored 0.80. Graph-based multi-hop activation can sometimes surface structurally plausible but factually irrelevant evidence for fabricated queries.

Ablation Insights

• Removing Engrama's specialized levers (L1 and L3) improved the global composite score but degraded cross-space performance.
• This reveals a design tension: mechanisms that specialize in connecting disparate memory spaces do not automatically maximize aggregate utility across all task types.

5. Significance and Conclusion

The paper demonstrates that memory organization fundamentally changes system behavior.

• Structured vs. Flat: Flat vector retrieval (Mem0) is insufficient for complex reasoning; the structure of the memory (graph vs. flat) is as important as the retrieval mechanism.
• Specialization vs. Generalization: Structured memory systems like Engrama are not yet "drop-in" replacements for full-context prompting for general tasks. However, they are sharper tools for specific, high-value reasoning demands (cross-space integration) that transcript inclusion handles only incidentally.
• Future Direction: The authors argue the next frontier is "selective structure"—systems that can dynamically apply structural organization when needed for complex reasoning while remaining efficient for simple recall.

Limitations: The benchmark uses synthetic data rather than organic logs, and the "Emergent Insight" metric relies on token-level F1 rather than human judgment. Additionally, Engrama is a proprietary system, and some implementation details (heuristics, weighting formulas) are omitted for confidentiality.