Technical Summary: PERSONATRAIL: Benchmarking Personalized Web Agents through Browsing Trails

Problem Statement

Recent advances in Large Language Models (LLMs) have enabled web agents to autonomously execute complex navigation tasks. However, current benchmarks fail to evaluate agents in realistic, user-centric scenarios where instructions are underspecified. In practice, users expect agents to infer missing context—such as personal preferences or references to past interactions—from raw browsing histories rather than receiving fully explicit prompts. Existing benchmarks either restrict tasks to explicit instructions, abstract interaction history into simplified forms (e.g., transaction records or LLM-synthesized high-level events), or fail to capture the fine-grained, low-level interaction trajectories (clicks, searches, page transitions) where user preferences and episodic contexts are actually embedded. Consequently, there is a lack of a standardized framework to evaluate an agent's ability to perform preference inference (deducing latent habits) and episodic grounding (recalling specific past sessions) in a managed open-web environment.

Methodology

1. The PERSONATRAIL Benchmark

The authors introduce PERSONATRAIL, a benchmark designed to evaluate personalized web agents using fine-grained, live-web browsing trajectories. The benchmark is constructed through a pipeline involving:

• Website Curation: Selection of 317 websites across 23 domains from the Tranco list and Similarweb, filtered for English content and accessibility.
• User Profile & Assignment: Generation of 200 diverse user personas (age, job, hobbies) and assignment of relevant subdomains and specific websites based on these profiles.
• Query Template Discovery: An LLM-based agent explores assigned websites to discover parameterized query templates containing preference variables ($V_p$) and target variables ($V_t$).
• Trajectory Rollout: Agents execute queries on live websites, generating fine-grained interaction logs (timestamps, URLs, elements, actions). Preferences are expressed through site-native interactions (e.g., filtering by genre), creating consistent behavioral patterns across multiple sessions.
• Task Definition: The benchmark evaluates two complementary tasks:
  • Preference Inference: The agent must infer latent preferences (e.g., "Find a movie in the genre I usually watch") from recurring patterns in the history.
  • Episodic Grounding: The agent must retrieve and re-navigate to a specific past session referenced by temporal or event-based cues (e.g., "Show me the recipe I searched for last Tuesday").
• Validity Maintenance: A managed open-web paradigm is used, featuring a two-stage verification pipeline (automatic replay and LLM verification) to ensure tasks remain executable despite website changes.

2. PACMEM Framework

To address the limitations of existing memory-based agents that discard episode-level details and latent preferences, the authors propose Preference-Aware Contextual Memory (PACMEM). This framework structures raw browsing histories into a two-level hierarchy:

• Factual Memory: Captures episodic records of past actions. Raw logs are segmented into atomic, goal-oriented segments, then grouped into coherent episodes based on semantic similarity. Each episode is structured into a memory entry containing a title, timestamps, URLs, and interaction sequences.
• Preference Memory: Encodes recurring behavioral patterns. Entries from the factual memory are clustered based on similarity, and an LLM generates natural language descriptions of underlying patterns (e.g., "Always uses IMDb for movies," "Prefers Mystery genre").
• Retrieval: At inference time, a coarse-to-fine retrieval pipeline selects task-relevant entries from both memory pools. It uses embedding-based similarity for initial candidate selection, followed by LLM-based reranking based on semantic relevance, temporal alignment, and utility.

Key Contributions

1. PERSONATRAIL Benchmark: A novel benchmark for personalized web agents featuring fine-grained, live-web browsing trajectories as user history. It uniquely supports both preference inference and episodic grounding tasks across single-hop and multi-hop scenarios.
2. PACMEM Framework: A simple yet effective reference method that transforms raw browser-level histories into structured factual and preference memories. This approach establishes a strong baseline for personalized web navigation by preserving both episodic details and preference signals.
3. Empirical Validation: Extensive experiments demonstrating that distilling raw trajectories into structured memory significantly outperforms existing memory-augmented baselines.

Experimental Results

Experiments were conducted on PERSONATRAIL using diverse backbone models (including Claude Haiku 4.5, GPT-5.4-mini, Gemini-3-Flash, and open-source models like Qwen3.6-27B and Llama-4-Scout).

• Performance Gap: Baselines without retrieval ("No Retrieval") achieved near-zero Task Success Rates (TSR) on personalization tasks, highlighting the necessity of history. While strong backbones achieved high Intent Scores (IS) without retrieval, they failed to apply personal constraints.
• PACMEM Superiority: PACMEM consistently outperformed memory-augmented baselines (AWM and ReasoningBank) across all backbones and task complexities.
  • On Preference Inference, PACMEM improved the TSR for Qwen3.6-27B on single-hop queries to 53.61%, surpassing ReasoningBank (46.45%).
  • On Episodic Grounding, PACMEM achieved a TSR of 39.03% (Qwen3.6-27B, multi-hop), significantly outperforming AWM (17.84%).
• Ablation Studies: Removing either factual or preference memory degraded performance. Factual memory was critical for episodic grounding (TSR dropped by ~22-33% when removed), while preference memory was essential for preference inference.
• Complexity Analysis: As the volume of user history and task complexity (number of steps) increased, baseline performance degraded sharply. PACMEM maintained its advantage, suggesting that structured memory distillation remains effective under complexity.
• Error Analysis: The primary failure mode for PACMEM was "Navigation Failure" (62-96% of failures), where the agent received correct memory but failed to apply it during navigation. This indicates that while memory retrieval is improving, the ability to execute complex navigation based on retrieved context remains a challenge.

Significance and Claims

The paper claims that PERSONATRAIL provides a practical foundation for developing web agents capable of assisting individual users in real-world environments. By grounding evaluation in fine-grained browsing trajectories rather than abstracted summaries, the benchmark reveals that current agents struggle to infer and apply user preferences even on static, solvable tasks.

The introduction of PACMEM demonstrates that transforming raw histories into structured memory representations—specifically separating episodic details from generalized preferences—is a necessary step for effective personalization. The authors assert that this work bridges the gap between functional task automation and the extraction of latent user preferences from long, noisy, cross-session trajectories.

The paper modestly acknowledges limitations, noting that the benchmark currently assumes stable preferences and solvable tasks, excluding scenarios like preference drift or unanswerable requests. Additionally, while the browsing trajectories are executed on live websites, the underlying user profiles and intents are synthetically generated to ensure privacy and scalability, though the authors argue this construction offers a safe and effective way to build user histories at scale.