Technical Summary: Don't Ask the LLM to Track Freshness

1. The Problem: Conflict Resolution in Evolving Memory

Large Language Model (LLM) based memory systems are increasingly tasked with maintaining facts that evolve over time (e.g., user preferences, policy updates, corrected records). A critical failure mode in these systems is conflict resolution: when a memory contains multiple contradictory values for the same fact, determining which value to return.

The standard convention in modern memory systems (Mem0, Graphiti, HippoRAG-v2, etc.) is that the "most recent" fact should win. However, evaluations on the MemoryAgentBench (MAB) FactConsolidation task reveal a stark underperformance across the field:

• The Task: Agents are given a corpus of numbered facts where counterfactual (newer) versions have higher serial numbers. Agents are explicitly told "newer facts have larger serial numbers."
• The Failure: Despite this explicit rule, every published system substantially underperforms.
  • HippoRAG-v2 (best published RAG): 54.0% accuracy on single-hop (FC-SH).
  • Zep/Graphiti (temporal knowledge graph designed for this): 7.0% on FC-SH.
  • Multi-hop (FC-MH): All 22 evaluated systems score ≤7%.
• The Hypothesis: The bottleneck is not storage architecture (graphs, hippocampal models, agentic loops) but the assembly step: existing baselines rely on the LLM to perform conflict resolution (judging freshness) within a free-text generation step, rather than using explicit, version-aware aggregation.

2. Methodology: A Deterministic Recipe

The authors propose a pipeline that decouples semantic retrieval from freshness logic, moving the latter out of the LLM and into deterministic code.

Core Primitives

1. Fact-Level Retrieval: Instead of chunking text arbitrarily (e.g., 512 or 4096 tokens), the system indexes individual facts. This preserves the version serial number as a distinct key, preventing the LLM from having to reason about freshness within a single noisy chunk.
2. Structured Candidate Extraction: An LLM is prompted to extract all semantically matching candidates from the retrieved set (Top-K=10) into a structured JSON format. The LLM is explicitly instructed not to pick a winner or compare serials, but to list every matching fact with its associated serial number and entity.
3. Deterministic Aggregation: A Python function (max(serial)) selects the candidate with the highest serial number. This operation is exact, fast, and immune to the LLM's tendency to override explicit rules with training-data priors.

Pipelines Evaluated

• SH-Conflict (Single-Hop): Retrieves facts, extracts candidates, and applies max(serial).
• CAR (Chain-Aware Resolution) for Multi-Hop: Decomposes complex multi-hop questions (e.g., "X of Y of Z") into atomic hops using a Self-Ask style prompt. Each hop is resolved independently using the SH-Conflict pipeline before passing the result to the next hop.

3. Key Contributions

1. Matched-Setup Pipeline Comparison: The authors conducted a controlled experiment holding the backbone (gpt-4o-mini), retrieval (BM25), chunking, and dataset constant, varying only the resolution logic.
   • Replacing the LLM-judgment pipeline with the Candidate Extraction + Python max(serial) pipeline yielded a +10.8 percentage point (pp) average gain on FC-SH.
   • The gap widened significantly with context length: +8 pp at 6K context, rising to +21 pp at 262K context.
2. New State-of-the-Art (SOTA) on MAB:
   • FC-SH: 78.0% (gpt-4o-mini) and 94.8% (gpt-4o), beating the best published system (HippoRAG-v2 at 54%) by +24 pp (pooled) and +28 pp at 262K.
   • FC-MH: 30.2% (gpt-4o-mini) and 51.5% (gpt-4o), beating the best published result (7%) by +23 pp (pooled) and +20 pp at 262K.
3. Mechanistic Evidence of LLM Failure: The study identifies two specific failure modes in LLM-judgment baselines that the deterministic approach eliminates:
   • Prior-Override: LLMs ignore the "newer wins" rule when the newer fact contradicts a strong training-data prior (e.g., "ice hockey" vs. "pesäpallo" for Finland).
   • Serial-Comparison Drift: As the candidate pool grows (longer context), LLMs lose track of which serial number is largest. The deterministic max() function remains exact regardless of pool size.
4. Cross-Benchmark Generalization: On LongMemEval (conversational memory with real timestamps), the mechanism ports from max(serial) to max(timestamp). However, it ties LLM judgment overall (57.8% vs 64.4%), indicating that deterministic aggregation is the correct primitive for current-value conflicts but must be composed with question-type-aware handlers for Yes/No, historical, or aggregation questions.

4. Results and Analysis

• Context Length Robustness: The LLM-judgment baseline degrades sharply at 262K context (dropping from 75% at 64K to 61% at 262K). The deterministic pipeline remains stable (71–82% across all lengths for gpt-4o-mini; 92–99% for gpt-4o).
• Retrieval Ceiling: A union analysis shows that 88.5% of questions are solvable by at least one pipeline, suggesting retrieval (BM25 Top-10) is not the primary bottleneck; the failure lies in the post-retrieval assembly.
• Complementarity: The pipelines are not strictly ordered. The deterministic pipeline wins on 21.3% of questions where the LLM fails (due to prior override or drift), while the LLM wins on 10.5% of questions (often where strict extraction over-rejects valid candidates).
• Cost Efficiency: The gpt-4o-mini deterministic pipeline costs ~$0.0001 per query, offering a 78% accuracy baseline that is significantly cheaper and more robust than complex graph-based systems.

5. Significance and Claims

The paper argues for a corrective shift in the memory-framework subfield:

• Assembly vs. Storage: The bottleneck for conflict resolution is assembly (how facts are identified and aggregated post-retrieval), not storage architecture (graphs, embeddings, or agentic loops).
• Simplicity Over Complexity: Elaborate temporal knowledge graphs (like Zep/Graphiti) designed specifically for temporal memory scored 7.0% on this task. The authors claim that for data with explicit version markers, the right primitive is structured candidate extraction followed by deterministic aggregation, not LLM-mediated judgment.
• Design Pattern: Future memory systems should narrow the LLM's role to tasks it performs reliably (semantic identification and candidate extraction) and delegate comparisons over structured metadata (serials, timestamps) to deterministic code.

The authors remain modest about the scope, noting that the recipe is specifically for current-value conflict resolution with explicit version markers. They acknowledge that multi-hop resolution remains difficult (30.2% is an improvement but not "solved") and that the approach requires composition with other handlers for non-freshness question types (Yes/No, historical queries).