Technical Summary: LongMemEval-V2

Problem Statement

Long-term memory is essential for large language model (LLM) agents operating in specialized, customized web environments where success depends on recalling interface affordances, state dynamics, workflows, and recurring failure modes. However, existing memory benchmarks primarily focus on user chat histories, short traces, or downstream task success rates. These benchmarks fail to directly evaluate whether memory systems can effectively internalize environment-specific experience accumulated over long horizons. Specifically, there is a gap in evaluating if an agent can transition from an "inexperienced" user to an "experienced colleague" who possesses deep, reusable knowledge of a specific environment's quirks and procedures.

Methodology

Benchmark Construction: LongMemEval-V2 (LME-V2)

The authors introduce LME-V2, a benchmark designed to evaluate whether memory systems help agents acquire the experience needed to become knowledgeable colleagues in customized environments.

• Data Source: Trajectories are collected from WebArena (OneStopShop, CMS, Reddit) and WorkArena/WorkArena++ (ServiceNow), using the AgentLab2 library. The dataset includes 1,540 trajectories (599 from WebArena, 941 from WorkArena) with an average of 28.1 states per trajectory.
• Question Curation: 451 manually curated questions cover five core memory abilities:
  1. Static State Recall: Remembering landmarks, layouts, and subtle differences.
  2. Dynamic State Tracking: Understanding how the environment changes given states and actions (world modeling).
  3. Workflow Knowledge: Knowing steps for common tasks.
  4. Environment Gotchas: Awareness of recurring issues and failure modes.
  5. Premise Awareness: Recognizing assumptions valid in other environments but wrong in the current one.
• Context Scale: The benchmark features "haystacks" of history trajectories. LME-V2-Small uses a shared 100-trajectory context (25M tokens), while LME-V2-Medium uses question-specific 500-trajectory contexts (115M tokens).
• Evaluation Formulation: The benchmark uses a context gathering formulation. A memory system receives a sequence of trajectories, inserts them, and then answers a query by returning a compact, multimodal evidence context. A fixed reader LLM (Qwen3.5-9B) then answers the question based on this truncated context.

Proposed Baselines: AgentRunbook

To address the challenge of organizing noisy agent trajectories, the authors propose two memory methods:

1. AgentRunbook-R (RAG-based): An efficient retrieval-augmented generation pipeline. It maintains three separate knowledge pools updated at insertion time:
   • Raw State Slices: Local UI observations and actions.
   • State Transition Events: Descriptions of how actions change the environment.
   • Procedure/Hint Notes: High-level reusable workflows and environment-specific pitfalls.
     At query time, an LLM controller generates specific queries for each pool, retrieves top-k results, and assembles a multimodal context.
2. AgentRunbook-C (Coding Agent-based): A method that treats memory management as a file management problem. Trajectories are stored directly as files. At query time, an off-the-shelf coding agent (Codex) is invoked within a sandbox containing:
   • Workflow documents instructing the agent on evidence collection.
   • Manifest artifacts summarizing the memory layout.
   • Helper scripts for efficient trajectory inspection.
     The agent assembles a compact evidence set, which is then passed to the reader.

Key Results

Experiments were conducted on both LME-V2-Small and LME-V2-Medium tiers.

• Baseline Performance: Simple RAG methods retrieving state slices achieved only 40% accuracy. The off-the-shelf Codex coding agent achieved a competitive 69.3% accuracy but suffered from high latency (182 seconds per query).
• AgentRunbook-R: Improved performance to 57.8% average accuracy with significantly lower latency (~26 seconds) compared to the coding agent approach. Ablation studies showed that the raw state pool is critical for static questions, while event and note pools are vital for dynamic and workflow questions.
• AgentRunbook-C: Achieved the best overall performance with 72.5% average accuracy on the medium tier, outperforming the vanilla Codex baseline (69.3%) and the strongest RAG baseline (48.5%). Crucially, AgentRunbook-C was 32% faster than the vanilla Codex agent (139.9s vs 185.8s on medium tier) due to its lightweight scaffolding (workflow instructions, manifests, and helper scripts) which reduced unnecessary exploration.
• Accuracy-Latency Frontier: AgentRunbook-C advances the Pareto frontier, offering a superior trade-off between accuracy and query latency compared to existing methods.

Significance and Claims

The paper claims that LME-V2 establishes a new standard for agent memory evaluation by shifting the focus from simple retrieval to experience accumulation.

• Experience as Memory: The benchmark posits that a high-quality memory system should enable an agent to act as an "experienced colleague" who internalizes environment-specific dynamics, not just facts.
• Context Depth: LME-V2 pushes the context depth of memory benchmarks to over 100 million tokens, revealing that standard RAG approaches struggle with the noise and complexity of raw agent trajectories.
• Coding Agents as Controllers: The results suggest that general coding agents, when augmented with specific workflow guidance and file-system tools, serve as highly effective memory controllers, outperforming traditional RAG pipelines in both accuracy and efficiency for this specific domain.
• Future Direction: While AgentRunbook-C advances the state-of-the-art, the authors modestly note that substantial room for improvement remains, particularly in further reducing latency costs while maintaining high accuracy.

The work provides a concrete testbed for developing memory modules that make long-running agents more reliable, adaptive, and useful in real-world, specialized environments.