Technical Summary: RECON – Benchmarking Agent Memory for Compositional Reasoning over Long Contexts

Problem Statement

Large Language Model (LLM) based agents are increasingly deployed in high-stakes workflows (coding, clinical, legal, financial) where they must retain, access, and reason over information accumulated across long contexts and multiple interactions. Current memory architectures (e.g., Mem0, Zep, MemGPT) and retrieval-augmented generation (RAG) systems primarily model memory as a state machine, tracking the current values of facts. However, in real-world scenarios, facts do not merely accumulate; they interact, contradict, and invalidate one another through explicit dependency structures.

Existing benchmarks (e.g., RULER, LongBench, LoCoMo) evaluate whether agents can retrieve scattered facts or detect if a fact has changed. They fail to evaluate the consequences of such changes: whether an agent can trace which downstream conclusions are affected by an invalidation, which conclusions survive via independent support, and how alternative timelines would unfold. There is a critical gap in evaluating compositional reasoning over evolving, long-context narratives where the validity of a conclusion depends on a complex provenance graph rather than a static fact list.

Methodology: The RECON Benchmark

The authors introduce RECON (Reasoning over Extended Contexts with Obfuscated Narratives), a benchmark designed to evaluate agents on six memory-intensive tasks over long contexts (50k–100k tokens).

1. Deterministic Generation Pipeline

To ensure ground-truth reliability, RECON employs a fully deterministic generation pipeline where LLMs are restricted to surface realization (narration) only and never influence causal structure, provenance, or answer keys.

• Blueprint Synthesis: A seeded production engine generates a case blueprint using typed terminals (actors, evidence) and production rules, ensuring logical prerequisites are met before conclusions are drawn.
• Skeleton Expansion: The blueprint expands into a structured skeleton of timestamped events, causal dependencies, invalidations, source conflicts, and parallel temporal streams (e.g., surveillance, transactions).
• Provenance DAG: A global Directed Acyclic Graph (DAG) is induced from the generation contract. Nodes represent events/evidence, and edges encode causal, revisionary, and invalidating relationships. This DAG serves as the authoritative ground truth.
• Task Synthesis: Questions are algorithmically generated from the DAG. For example, Cascade Propagation questions are created by selecting an invalidation node and computing reachability to determine which conclusions collapse.
• Narrative Realization: An LLM converts the structured skeleton into natural language case files (police reports, clinical logs, financial audits) while strictly adhering to the immutable facts of the skeleton.

2. Task Categories

RECON spans 24 case files across three domains (criminal, medical, financial) and 1,604 questions, categorized into six tasks:

1. Chain Reconstruction: Locating and causally ordering 5–15 evidence hops scattered across the document.
2. Cascade Propagation: Determining which conclusions break and which survive via independent support after a specific evidence invalidation.
3. Source Conflict Resolution: Adjudicating contradictory accounts using independent corroborating evidence.
4. Counterfactual Reasoning: Determining how downstream events change under an alternative timeline (e.g., if an event occurred earlier).
5. Temporal Constraint Satisfaction: Cross-referencing parallel data streams against a specific time window.
6. Temporal Fact Retrieval: Baseline tasks for temporal ordering and state queries.

3. Evaluation Setup

The benchmark evaluates three system families plus an Oracle ceiling:

• Long-Context: Models processing the full 100k-token case file.
• RAG: Variants including dense retrieval, hybrid retrieval, and reranking.
• Memory Agents: Systems like Mem0, Mem0-Graph, Supermemory, and Hindsight.
• Oracle: Receives the structured ground-truth DAG instead of the narrative text, providing a retrieval-perfect upper bound.

Scoring involves strict accuracy metrics, with penalties for wrong commits on multiple-choice questions and abstention options. A contamination filter removes questions answerable via prior LLM knowledge.

Key Results

The evaluation reveals substantial limitations across all current architectures:

• Overall Performance: No non-Oracle system exceeds 25% Accuracy. The best non-Oracle system (Gemini-2.5-Pro) achieves only 22.4% Accuracy, while the Oracle (with perfect retrieval) reaches 54.6%. This indicates that even with perfect access to the dependency graph, reasoning remains a significant bottleneck.
• Task-Specific Performance:
  • Cascade Propagation: Memory systems excel here, with Supermemory achieving a score of 0.708, nearly double the best long-context model. This suggests memory architectures are effective at tracking state changes when the dependency structure is simple.
  • Counterfactual Reasoning: This is the hardest task. Even the Oracle reaches only 0.483, indicating that chained inference is the primary bottleneck, not retrieval.
  • Chain Reconstruction & Source Conflict: Long-context models lead these tasks, but performance remains low. RAG and memory systems struggle significantly, likely due to the loss of inter-fact edges during compression or retrieval.
• Retrieval vs. Reasoning:
  • Retrieval is necessary but insufficient. In "full-coverage" scenarios where the correct evidence is retrieved, systems still answer incorrectly roughly 80% of the time.
  • The gap between Oracle and LLM performance is largest for tasks requiring multi-hop reasoning (Chain Reconstruction, Source Conflict), suggesting that the inability to compose retrieved facts is a major failure mode.
• Token Efficiency: Retrieval-based approaches (RAG, Memory) are 8–20× more token-efficient than long-context models but often trade off performance on complex reasoning tasks.

Significance and Claims

The paper claims that RECON represents a shift in how agent memory should be evaluated:

1. From State Machine to Provenance Graph: The benchmark argues that memory should be modeled as a graph of derivation histories where invalidations propagate through dependencies, rather than a simple state machine of current fact values.
2. Compositional Reasoning is the Bottleneck: The results demonstrate that current agents fail not just at finding information, but at composing it. The large gap between Oracle (perfect retrieval) and LLM performance highlights that the ability to trace causal chains and handle invalidations is a fundamental limitation of current architectures.
3. Deterministic Ground Truth: By decoupling the generation of causal structure from LLM narration, RECON provides a rigorous, reproducible benchmark free from the hallucinations often found in synthetic datasets generated entirely by LLMs.

The authors conclude that both retrieval and reasoning remain open challenges. Even with perfect context, agents struggle to maintain coherent, evolving understandings of long documents where facts interact and contradict. The release of the benchmark, generator, and evaluation harness aims to support future work in developing agents capable of robust, compositional reasoning over long contexts.