Technical Summary: VitaBench 2.0

Problem Statement

While Large Language Models (LLMs) have evolved into interactive agents capable of reasoning and tool use, existing benchmarks primarily evaluate these capabilities in settings where user intent is explicitly stated and context is fully provided. In real-world scenarios, however, effective human-agent collaboration relies on understanding user intent that is often implicit, fragmented, and evolves over time. Current benchmarks largely overlook the challenges of inferring and leveraging user preferences from such fragmented interactions, as well as the need for agents to proactively acquire missing information when user preferences are conditional. This gap leaves a significant disconnect between the capabilities of current agents and the requirements for realistic, long-term personalized assistance.

Methodology

To address this gap, the authors introduce VitaBench 2.0, a benchmark designed to evaluate personalized and proactive agent behavior in long-term user interactions.

Benchmark Construction

• Task Formulation: Tasks are organized as temporally ordered sequences for individual users, spanning multiple domains (Delivery, In-store Consumption, and Online Travel Agency). Each task is modeled as a Partially Observable Markov Decision Process (POMDP) where the agent must interact with domain-specific tools and an executable environment.
• User Modeling: The benchmark features 56 curated users with over 2,000 fine-grained preferences. User profiles are constructed based on real-world demographic and socioeconomic statistics. Preferences are not explicitly provided; instead, they are embedded into fragmented interaction histories comprising both multi-turn dialogues and behavioral logs (browsing, ordering, searching).
• Preference Dynamics: To simulate real-world evolution, the benchmark introduces preference drift events between tasks, including the addition, deletion, and modification of preferences.
• Proactiveness: A subset of tasks involves conditional preferences where the correct decision depends on context not observable in the current query (e.g., time of day, companion). Successful completion requires the agent to recognize missing information and proactively query the user or environment.
• Memory Interface: VitaBench 2.0 provides an extensible memory interface with UPDATE and RETRIEVE operations. This allows for controlled comparison across different memory architectures. The authors implement two representative mechanisms:
  • Agentic Memory: The agent actively controls memory content, deciding what to retain, update, or discard via a structured summary.
  • RAG Memory: Interaction records are stored in a vector database, and retrieval is based on similarity to the task query without explicit agent control over retention.

Experimental Setup

The authors evaluated a diverse set of frontier proprietary and open-source LLMs, including models from OpenAI, DeepSeek, Anthropic, Google, and others. Both "thinking" (reasoning-enhanced) and "non-thinking" configurations were tested. Evaluation metrics included Avg@4 (mean performance), Pass@4 (probability of at least one success in four trials), and Pass^4 (probability of all four trials succeeding).

Key Contributions

1. VitaBench 2.0 Benchmark: A novel benchmark specifically targeting the evaluation of personalization and proactiveness in long-term agent interactions, moving beyond static, single-turn, or explicitly instructed tasks.
2. Extensible Memory Interface: A unified framework enabling systematic analysis of how different memory mechanisms (Agentic vs. RAG) impact long-term user modeling and decision-making.
3. Comprehensive Evaluation: Extensive testing across a wide range of state-of-the-art models, revealing specific failure modes and capability bottlenecks in real-world personalized decision-making.
4. Analysis of Failure Modes: Detailed identification of why current agents struggle, including the inability to distinguish signal from noise in interaction histories, the failure to update preferences dynamically, and the lack of proactiveness in handling conditional intents.

Results

The evaluation reveals that real-world personalization remains highly challenging even for state-of-the-art models:

• Performance Gap: Even under the "Full Context" setting (where full interaction history is accessible), top models achieved an Avg@4 of only around 0.5 and a Pass^4 of roughly 0.3. This indicates a substantial gap between current capabilities and practical requirements.
• Memory Limitations: While memory is essential for long-term modeling, most models experienced performance degradation when relying on memory mechanisms compared to full context. Both Agentic and RAG memory approaches struggled to consistently translate stored information into improved performance, suggesting that current agents cannot effectively utilize memory for preference inference.
• Reasoning vs. Personalization: Enabling "thinking" modes did not consistently lead to better performance on VitaBench 2.0. This suggests that personalization requires capabilities beyond general reasoning, such as robust preference extraction and handling noisy observations.
• Proactiveness Deficit: Agents consistently performed worse on proactive tasks (requiring clarification of missing info) compared to standard personalization tasks. Models often proceeded with incomplete information rather than initiating clarification.
• Preference Utilization: Even when provided with ground-truth user preferences, models struggled to effectively leverage them for decision-making, indicating that the bottleneck is not just extraction but also the reasoning and prioritization of preference information.
• Failure Patterns: Analysis showed that as tool-use capabilities improve, personalization becomes the dominant bottleneck. Errors are primarily driven by failures to capture and utilize user preferences rather than tool invocation errors.

Significance

The paper claims that VitaBench 2.0 highlights a critical gap between current LLM agents and realistic personalized assistants. By demonstrating that current models struggle to infer, utilize, and update user preferences in fragmented, long-term settings, the benchmark provides a necessary testbed for future research. The authors position VitaBench 2.0 as a resource to advance the development of memory systems, personalization techniques, and proactive agent behaviors, moving the field beyond reasoning and tool orchestration toward truly user-aligned, long-term collaboration.