Technical Summary: MOMENTO

Problem Statement

While recent advances in agentic AI have demonstrated strong capabilities in reasoning, tool use, and multi-step planning within isolated single-session settings, existing benchmarks fail to reflect real-world deployments. In practice, users interact with agents across days and weeks, bringing evolving goals, accumulated history, and prior decisions that must be integrated to fulfill personalized objectives. Current evaluation frameworks predominantly ignore these cross-session dependencies, treating agents as stateless entities or evaluating them only on fact recall rather than on whether retrieved context drives better task decisions. This creates a significant gap between current agent capabilities and the requirements of realistic, long-horizon human-agent interaction where memory must support consequential, tool-mediated actions.

Methodology

To address this gap, the authors introduce MOMENTO, a benchmark and evaluation framework designed for persistent agentic task completion in multi-session, multimodal service environments.

Framework Architecture

The system operates through a role-playing simulation involving two agents:

1. User Agent ($\pi_u$): Simulates human behavior by generating natural conversational responses conditioned on a user policy ($\Phi_u$). This policy defines personas, long-term goals, and preferences. The agent retrieves context from a shared memory module to maintain temporal coherence across sessions.
2. Assistant Agent ($\pi_a$): Responsible for responding to user inputs, retrieving relevant multi-session summaries, invoking external tools, and generating final responses. The agent must perform temporal reasoning to resolve dependencies between past sessions and current requests.

Agentic Memory

A core component of MOMENTO is the Agentic Memory module ($M$), operating within a bounded 128k-token context. It employs a hybrid approach:

• Long-term Recall: Retrieves relevant prior sessions from a database using LLM-generated SQL queries. This allows for explicit reasoning over cross-session dependencies, combining semantic matching with temporal constraints and structural conditions.
• Short-term Compression: Manages in-context dialogue history by retaining the most recent 80% of the context verbatim for fine-grained detail, while condensing older prefixes into compact summaries to reduce overhead.

Benchmark Construction

MOMENTO consists of 161 task instances generated via human role-playing to simulate realistic interaction histories. The tasks span scenarios such as personal assistance, planning, scheduling, and recommendation, often involving food-related entities from the WorldCuisines dataset. The dataset incorporates evolving preferences, implicit references, contradictory updates, and temporally distributed information to increase realism. The environment provides 26 tools covering reservation management, search, ordering, membership management, and text-to-SQL access.

Evaluation Metrics

Performance is measured using three primary metrics:

1. DB STATE: Checks if the final database state matches the user's goal.
2. OUTPUT: Measures if the assistant's response satisfies the user request.
3. TOOL RECALL: Evaluates whether the correct tools are invoked in the correct order.
   A task is considered successful only if all three metrics are satisfied. The study also reports PASS@K and PASS^K to measure consistency across multiple runs.

Key Contributions

• First Persistent Multi-Session Benchmark: MOMENTO is the first framework to evaluate persistent agentic task completion where cross-session context is a first-class input to decision-making, moving beyond single-session or simple fact-recall evaluations.
• Realistic Simulation: The benchmark utilizes a large-scale dataset of temporally separated interactions featuring evolving user goals and latent memory dependencies, generated through role-play-based user simulation.
• Public Release: The dataset and source code are publicly released under CC-BY-SA and Apache 2.0 licenses, respectively, to facilitate further research.

Results and Analysis

The authors evaluated several state-of-the-art open-weight visual language models (VLMs), including QWEN3.6 and GEMMA4 variants.

• Performance: GEMMA4-31B-IT exhibited the strongest overall performance, securing the highest PASS@3 and PASS^3 scores and demonstrating superior consistency across repeated trials. While GPT-5.4-MINI showed marginal leads in specific metrics like DB STATE, it did not surpass GEMMA4-31B-IT in overall task success.
• Instability: A substantial gap between PASS@3 and PASS^3 across all models highlights a pervasive instability in multi-step task execution.
• Error Analysis: The dominant failure mode is TOOL RECALL, specifically the misestimation of user state. Agents tend to treat prior session history as a reliable proxy for the current context, forming implicit assumptions without verification. Consequently, agents frequently omit necessary verification or retrieval calls (e.g., checking membership status or current order states) before executing actions. This failure is structurally enabled by the multi-session setting, where history creates a false sense of state continuity.

Significance

The paper claims that MOMENTO reveals a substantial gap between current agent capabilities and the demands of realistic long-horizon human-agent interaction. The primary finding is that current agents fail not due to a lack of tool availability, but due to a failure to re-validate stale information from past sessions. By treating prior history as static truth rather than information requiring re-validation, agents miss critical steps in the reasoning chain. The authors posit that MOMENTO provides a necessary foundation for future research into persistent, memory-informed agentic systems that can accurately navigate temporal dependencies and evolving user goals.

Limitations

The authors acknowledge that, like previous benchmarks, MOMENTO relies on LLM-based user simulators. While this enables scalable data collection, simulated users may not perfectly exhibit realistic human behavior. Future work is suggested to explore stronger constrained simulation methods, role-specific fine-tuning, or real-human interactions to further bridge the behavioral gap.