Technical Summary: Mem2ActBench

Problem Statement

Current Large Language Model (LLM) agents are increasingly deployed for complex, tool-based tasks requiring long-term memory. However, existing benchmarks (e.g., MSC, LoCoMo) primarily evaluate an agent's ability to passively retrieve isolated facts in response to explicit questions (e.g., "What is the user's budget?"). These benchmarks fail to assess the more critical capability of actively applying memory to execute tasks. In realistic scenarios, users provide underspecified instructions (e.g., "Book a flight") and expect agents to implicitly retrieve and ground relevant historical constraints (e.g., budget, preferences) into tool parameters without explicit restatement. There is a lack of benchmarks that evaluate whether agents can infer task-critical constraints from dispersed, long-term interaction histories and ground them into executable tool calls.

Methodology

To address this gap, the authors introduce Mem2ActBench, a benchmark constructed via a three-stage automated pipeline designed to simulate persistent, interruption-heavy assistant usage.

1. Heterogeneous Data Integration

The pipeline synthesizes long-term interaction histories by interleaving task-oriented tool-use data with conversational noise:

• Task Data: Multi-step tool-use trajectories are reconstructed from ToolACE and BFCL datasets into natural multi-turn dialogues.
• Conversational Noise: High-quality ranked responses from OASST1 are reconstructed into full threads to simulate realistic, non-task-related interruptions.
• Scale: This process generates 2,029 dialogue sessions with an average of 13 turns per session.

2. Memory Evolution Chain Construction

To transform raw dialogues into a coherent ground-truth memory, the authors implement a conflict-resolution mechanism:

• Fact Extraction: An LLM extracts structured facts as triples (attribute, fact, source_id).
• Clustering: Facts are clustered by semantic topic using BERTopic and HDBSCAN.
• Conflict Resolution: A modified topological sorting algorithm (based on Kahn's algorithm) constructs a global evolution sequence. It resolves temporal conflicts by ordering facts chronologically, preserving valid updates (e.g., refinement from "sports" to "basketball"), and discarding contradictory or redundant facts. When cycles (contradictions) are detected, the algorithm deterministically removes the node with the highest out-degree to restore a valid ordering.

3. Reverse Query Generation

The core innovation is a reverse-generation approach to ensure genuine memory dependence:

• Ground Truth Construction: A fully specified tool invocation is constructed where all parameters are strictly grounded in the memory chain.
• Underspecification: The system reverse-engineers a user query that omits these key parameter values, relying on anaphoric expressions (e.g., "book that flight").
• Leakage Control: A discriminator LLM attempts to reconstruct the correct tool invocation using only the query and API documentation (without memory). Samples are retained only if the discriminator fails, guaranteeing that the task is unsolvable without retrieving the specific long-term memory.
• Human Verification: Expert annotators verify fact extraction, conflict resolution, and memory dependency, confirming that 91.3% of the 400 generated tasks cannot be solved without long-term memory access.

Key Contributions

1. Principled Benchmark Design: Mem2ActBench is the first benchmark specifically targeting inference-driven long-term memory utilization in tool-augmented agents, moving beyond simple fact retrieval to evaluate the grounding of underspecified requests.
2. Dataset Release: The release of 400 memory-dependent tool-use tasks derived from 2,029 long-context sessions, with human validation ensuring high reliability and strict memory dependence.
3. Comprehensive Evaluation: A systematic evaluation of seven representative memory frameworks (including RAG, Generative Agents, MemGPT, A-Mem, etc.) reveals persistent bottlenecks in current systems.

Experimental Results

The authors evaluated seven memory frameworks using Qwen2.5 models (7B, 32B, 72B) on the benchmark. Key findings include:

• Parameter Grounding Gap: While Tool Selection Accuracy (TSA) remains high (~87–97%), parameter grounding (F1 score) is significantly lower, ranging from ~20% to ~36%. Scaling model size improves performance but yields diminishing returns, suggesting the bottleneck is not just reasoning capacity but memory application.
• Retrieval vs. Reasoning: An "Oracle Retrieval" experiment (providing ground-truth memory) boosted F1 scores from ~30% to ~54%, indicating that the primary bottleneck is evidence hitting/retrieval quality rather than the model's ability to reason once the correct memory is available.
• Positional Bias ("Lost-in-the-Middle"): Performance drops sharply when supporting evidence lies in the middle of the conversation history (25–50% position), confirming the "lost-in-the-middle" phenomenon affects tool-use tasks.
• Failure Modes: Analysis reveals that as memory frameworks improve, failures shift from "Retrieval Miss" to "Retrieved-but-Unused," indicating that even when memory is found, agents struggle to utilize it for parameter grounding. Additionally, models frequently hallucinate default values when parameters are not explicitly specified.

Significance and Claims

The paper claims that Mem2ActBench exposes a critical limitation in current autonomous agents: the inability to proactively leverage long-term memory to execute tasks under underspecified instructions. The results highlight that current systems are inadequate at actively applying memory for parameter grounding, particularly when evidence is buried in mid-context histories.

The authors conclude that future research must focus on enhancing the active utilization of memory and improving mechanisms for locating sparse, critical evidence under implicit queries, rather than solely relying on scaling model parameters or passive retrieval. The benchmark serves as a necessary tool to drive the development of agents capable of true long-term task execution.