Technical Summary: EMemBench

Problem Definition

Current evaluation practices for long-term memory in Large Language Models (LLMs) and Vision-Language Models (VLMs) are largely static. Existing benchmarks typically assess models by providing pre-defined, fixed conversation histories or logs and asking questions about them. This approach fails to treat memory as an interactive and individualized competence. In cognitive psychology, episodic memory is defined as the recording of personal experiences, which is inherently individualized. Static benchmarks abstract away the memory formation process, failing to evaluate how an agent forms structured records of its own decisions and observations in a dynamic environment. Furthermore, many existing benchmarks are text-centric, lack spatial reasoning capabilities, and rely heavily on manual annotation, limiting scalability.

Methodology

The authors introduce EMemBench, a programmatic benchmark generator designed to evaluate long-term memory through interactive games. Unlike fixed datasets, EMemBench generates questions dynamically from each agent's own interaction trajectory.

Core Framework

1. Interactive Environments: The benchmark operates on two environments:
   • Jericho: A suite of human-authored interactive fiction games (text-only).
   • Crafter: An open-world survival game with visual observations (64x64 RGB frames + HUD).
2. Three-Phase Evaluation Pipeline:
   • Phase 1 (Interaction): An agent plays an episode, producing a trajectory $\tau = \{(o_t, a_t, r_t)\}$ of observations, actions, and rewards. The system logs both agent-observable signals and underlying game states (e.g., global maps, coordinates).
   • Phase 2 (Generation): A programmatic generator $G$ converts the episode trace and game state into a balanced set of Questions ($Q$) with deterministic ground truths ($Y$). The generator uses a library of templates covering seven reasoning categories:
     • Single-hop and Multi-hop recall
     • Induction (pattern discovery)
     • Temporal reasoning
     • Spatial reasoning
     • Logical reasoning
     • Adversarial (counterfactual) questions
   • Phase 3 (Evaluation): The same agent that played the episode answers the generated questions using only the memory formed during interaction (plus the current context), without access to the full raw log unless specified in the "open-book" setting.

Key Design Controls

To ensure fairness and verifiability, the authors implement several controls:

• Individualization: Questions are derived from the specific agent's trajectory, ensuring the evaluation is experience-conditioned.
• Verifiable Ground Truth: Answers are computed programmatically from underlying game signals (e.g., exact coordinates, inventory states), eliminating the need for human post-editing.
• Query-Horizon Control: To prevent length-based confounds (where longer trajectories inherently make recall harder), the benchmark can restrict evidence selection and answer computation to a fixed window of interaction steps (e.g., steps 1–50).
• Balanced Coverage: The generation process enforces a balanced distribution of question types across reasoning abilities.

Key Contributions

1. A Programmatic Benchmark Generator: Shifting the artifact from a static dataset to a reproducible pipeline that generates individualized, verifiable QA pairs from agent trajectories.
2. Multi-Modal and Multi-Domain Scope: Covering both text-only (Jericho) and visual (Crafter) settings, explicitly including spatial and induction reasoning which are often absent in prior work.
3. Rigorous Evaluation Controls: Introducing mechanisms like query-horizon control and adversarial negative generation to isolate specific memory capabilities and prevent overfitting to specific trajectory lengths or priors.
4. Human Baseline: Conducting a human study to establish upper and lower bounds for performance, confirming that many questions require trajectory-grounded memory rather than general priors.

Experimental Results

The authors evaluated strong foundation models (GPT-5.1, Qwen2.5/3, InternVL) and memory agents (Mem0, LangMem, A-MEM) across 15 text games and multiple visual seeds.

• Overall Performance: Results are far from saturated. Even the strongest settings achieve only ~51.9% accuracy on text games and ~43.8% on visual games.
• Persistent Bottlenecks: Induction and Spatial reasoning remain the most difficult abilities, particularly in visual settings where agents struggle to bind transient observations to step-indexed episodes and aggregate them into a coherent map.
• Memory Agents vs. In-Context:
  • Text Games: Persistent memory modules (especially A-MEM) yield clear gains over in-context prompting for open backbones (e.g., Qwen3-32B improved from 44.9% to 51.9% overall accuracy).
  • Visual Games: Improvements are less consistent. While some reasoning capabilities improve, the gains are smaller or negative compared to text, suggesting that visually grounded episodic memory remains an open challenge.
• Human Comparison: Human performance in "closed-book" (memory-only) settings is significantly lower than "open-book" (context-access) settings, particularly for Multi-hop, Induction, and Spatial questions. This gap validates the benchmark's difficulty and indicates that current agents struggle to match human episodic recall without external aids.
• Stability: Results are stable across different random seeds (standard deviation < 0.05), indicating that rankings are not driven by specific environment seeds.

Significance and Claims

The paper claims that EMemBench provides a reliable testbed for developing robust memory-augmented agents by moving beyond static long-context QA. Its significance lies in:

1. Diagnosing Specific Deficits: It reveals that while models are improving at simple recall, they fail at complex, environment-grounded reasoning (induction, spatial) required for true episodic memory.
2. Highlighting the Visual Gap: It demonstrates that current VLM agents lag significantly behind text-only agents in maintaining and retrieving fine-grained, spatially grounded evidence.
3. Methodological Shift: It advocates for an evaluation paradigm where memory is tested as an interactive, individualized process rather than a passive retrieval task over a fixed document.

The authors modestly note that while the benchmark reduces test-set leakage through dynamic generation, contamination via pattern memorization is not entirely eliminated. They also acknowledge that the current environment coverage is limited due to the high cost of validating game-specific reasoning templates, but they plan to expand this in future releases.