Technical Summary: APeB – Benchmarking Personalization Ability of Large Language Model Agents

1. Problem Statement

Large Language Model (LLM)-powered agents face significant challenges in personalization when users issue raw, underspecified queries. In real-world scenarios, user objectives are rarely explicit; preferences are implicit and must be inferred from noisy, heterogeneous interaction histories (e.g., videos, livestreams, product views). Furthermore, correctness is user-centric rather than globally optimal, requiring agents to discriminate between closely competing alternatives based on latent intent.

Existing benchmarks fail to adequately test this capability because they typically rely on:

• User-refined queries rather than vague, early-stage intents.
• Simplified or sanitized histories that lack the noise and diversity of real-world behavior.
• Easily separable candidates that do not require fine-grained preference discrimination.

Consequently, current evaluations often conflate semantic matching with true personalization, failing to assess whether agentic workflows (e.g., ReAct) can effectively infer intent and align actions with specific user preferences.

2. Methodology: The APeB Benchmark

The authors introduce APeB (Agent Personalized Benchmark), a testbed designed to evaluate agentic personalization through the Personalized Product Search (PPS) task. APeB is constructed from real-world action logs of a leading content-commerce platform, capturing the end-to-end flow from content consumption to product purchase.

2.1 Data Construction

• Source: 5,648 non-trivial shopping sessions collected from a 30-day retained-session window, with up to 60 days of pre-session history.
• Session Definition: A "search-to-order" session is defined as a time-bounded window starting with a search query and ending with a purchase ($\Delta \le 30$ minutes).
• Non-Trivial Filtering: Sessions are filtered to ensure they represent high-effort shopping regimes, requiring:
  1. Iterative Query Refinement: At least two distinct search queries ($|Q_{search}| \ge 2$).
  2. Extensive Browsing: Substantial exploration of products and media before purchase ($|P_{view}| + |V_{view}| \ge \tau_v$).
  3. Intent-Order Alignment: The final purchase must be semantically consistent with the user's expressed intent and explored candidates.

2.2 Benchmark Components

For each retained session, APeB constructs three core components:

1. Heterogeneous Histories ($H_u$): Long-term interaction logs containing diverse signals, including product interactions (views, likes, carts) and media interactions (videos, livestreams with OCR/ASR content).
2. Query Spectrum:
   • Intent Query ($q_{intent}$): The earliest query semantically aligned with the purchase, representing a vague, early-stage goal (e.g., "campus OOTD").
   • Refined Query ($q_{refined}$): The last query before purchase, representing a well-specified intent (e.g., "oversized long-sleeve navy blue").
3. Hard Candidate Set ($C_{hard}$): A set of products the user viewed but did not buy, plus the final purchased item. These candidates are semantically close, requiring fine-grained attribute discrimination rather than coarse semantic matching.

2.3 Evaluation Protocol

Models are evaluated on their ability to rank the purchased item within the hard candidate set given an input of $(q, H_u, C_{hard})$.

• Metrics: Hit@1 and Hit@5.
• Rubric Analysis: Intermediate reasoning steps (Query Inference, Preference Quality, Recommendation Quality, Agent Reasoning) are scored using LLM-as-a-Judge to diagnose specific failure modes.

3. Key Contributions

1. APeB Benchmark: The first benchmark to jointly evaluate agents on noisy heterogeneous histories, vague-to-refined search trajectories, and closely competing candidates. It bridges the gap between recommendation systems (next-item prediction) and agentic reasoning (multi-step planning).
2. Empirical Analysis of LLM Agents: A comprehensive evaluation of state-of-the-art LLMs (including GPT-4/5, Qwen3, DeepSeek-R1, Gemini) and agentic workflows (ReAct, Deerflow) reveals that while models handle explicit queries well, they struggle significantly with early-stage intent discovery and preference recovery.
3. Diagnosis of Failure Modes: Rubric analysis identifies that performance gaps are primarily driven by ineffective history utilization. Models fail to extract relevant preferences from noisy histories to guide planning, and multi-step reasoning (ReAct) often amplifies these errors under vague intents.
4. VQRA Pipeline: The authors propose VQRA (Vague Query Refinement Agent), a simple pipeline that uses a shared backbone to rewrite intent queries based on selected history context before evaluation. This approach yields consistent gains, demonstrating that explicit history-utilization modules are critical for personalized agents.

4. Experimental Results

• Performance Gap: In settings dominated by semantic alignment (refined queries or random candidates), LLMs substantially outperform the specialized PPS baseline (UniSAR). However, in tasks requiring user modeling and intent refinement (intent queries with hard candidates), LLMs fall to the level of zero-shot UniSAR, with limited gains from ReAct workflows.
• History Utilization: LLMs struggle to benefit from history under intent queries. While adding history improves performance for refined queries, it offers marginal gains for intent queries, suggesting current LLM workflows do not naturally extract recoverable preferences from noisy, underspecified contexts.
• Candidate Difficulty: LLMs outperform baselines on random candidates (coarse semantic matching) but fail to distinguish fine-grained attributes among hard candidates (user-viewed items). This indicates that current models lack true personalization capabilities when semantic cues are insufficient.
• VQRA Efficacy: Implementing VQRA improves Hit@1 across various backbones, confirming that explicit history-grounded query refinement helps models infer latent intent and select appropriate candidates.

5. Significance and Claims

The paper claims that APeB serves as a necessary testbed for evaluating agentic systems under realistic personalization constraints. Its primary significance lies in exposing the limitations of current LLM agents: they are proficient at executing tasks with clear goals but struggle with the inference of latent intent and the recovery of preferences from noisy histories.

The authors conclude that the gap in personalization ability is not merely a lack of model scale but a structural deficiency in history-utilization modules. The success of the simple VQRA pipeline suggests that future personalized agents require dedicated mechanisms to explicitly ground planning and action in user-specific historical signals, rather than relying solely on implicit context within the prompt.

Limitations Acknowledged:

• The benchmark is derived from a single platform, potentially limiting transferability to other e-commerce ecosystems.
• Evaluation is static and offline, not reflecting real-time inventory or multi-turn dynamic interactions.
• Observed purchases serve as proxies for preference but may be confounded by ranking exposure and stock availability.
• Rubric analysis relies on LLM-as-a-Judge, which may introduce systematic biases.