AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment

1. Problem Statement

Despite the rapid evolution of Large Language Models (LLMs) into potential lifelong AI assistants, LLM personalization remains hindered by a lack of robust, gold-standard evaluation benchmarks. Existing benchmarks suffer from two critical limitations:

• Neglect of Information Governance: Many benchmarks (e.g., LaMP, PersonalLLM) focus solely on the final output alignment with user preferences, ignoring the intermediate lifecycle of personalized information management (extraction, updating, and retrieval).
• Synthetic Data Bias: Existing memory-aware benchmarks rely heavily on synthetic dialogues generated by LLMs. These suffer from a distributional gap compared to real-world interactions, characterized by:
  • Homogeneity: Lack of diverse conversational styles.
  • Over-explicitness: Synthetic data often contains explicit preference statements, failing to capture the implicit signals and nuanced expressions found in real human-LLM interactions.

2. Methodology: The AlpsBench Framework

The authors introduce AlpsBench, a comprehensive benchmark derived from real-world human-LLM dialogues. The framework is built on a four-step curation pipeline and evaluates personalization across four core tasks.

A. Data Construction Pipeline

1. Data Collection: Utilized the WildChat dataset (90,000+ users), filtering for long-term interaction sequences (6–249 turns) to ensure rich contextual dependencies.
2. Memory Extraction & Filtering: Used the DeepSeek-v3.2 reasoning model to automatically extract structured memories from raw dialogues. A filtering strategy was applied to prioritize users with implicit memory signals and remove semantic redundancy, resulting in 2,500 high-quality samples.
3. Human Annotation: Senior practitioners with domain expertise verified and corrected the extracted memories to establish a high-quality ground truth.
4. Task Construction: Generated evaluation queries and ground-truth answers for the four tasks using GPT-5.2, followed by rigorous human quality control to prevent information leakage.

B. The Four Core Evaluation Tasks

AlpsBench defines a systematic lifecycle for personalization:

1. Personalized Information Extraction (Task 1): Evaluates the ability to distill raw dialogue into structured memory (Explicit/Implicit, Label, Value, Confidence).
   • Metrics: F1 score (exact match) and LLM-as-a-judge (semantic similarity).
2. Personalized Information Update (Task 2): Assesses the capacity to track user dynamics. Given historical and new dialogue, the model must output updated memories and classify the action: Retention, Addition, or Modification.
   • Metrics: Accuracy against ground-truth actions.
3. Personalized Information Retrieval (Task 3): Measures the ability to select relevant memories from a candidate set containing one positive sample and multiple negative distractors.
   • Metrics: Recall and Precision.
4. Personalized Information Utilization (Task 4): Examines the alignment of the final response with user preferences. It breaks down into five dimensions:
   • Persona Awareness (PA): Recalling explicit attributes (e.g., education).
   • Preference Following (PF): Aligning with implicit tastes/habits.
   • Virtual-Reality Awareness (VRA): Distinguishing real facts from role-play/fiction.
   • Constraint Following (CF): Adhering to user-imposed rules.
   • Emotional Intelligence (EI): Providing emotionally appropriate responses.
   • Metrics: Binary scores (0/1) for PA, PF, VRA, CF; 1–5 scale for EI.

3. Key Contributions

• AlpsBench Dataset: A dynamic benchmark comprising 2,500 real-world dialogues paired with human-verified structured memories, capturing both explicit and implicit personalization signals.
• Comprehensive Framework: The first benchmark to evaluate the entire lifecycle of memory management (Extraction $\to$ Update $\to$ Retrieval $\to$ Utilization) rather than just the final output.
• Multi-Dimensional Evaluation: Introduces a granular assessment of utilization capabilities, including critical but often overlooked dimensions like Virtual-Reality Awareness and Emotional Intelligence.
• Public Leaderboard: A released platform to facilitate continuous community benchmarking.

4. Experimental Results & Key Findings

The authors evaluated frontier general-purpose LLMs (e.g., GPT-5.2, Gemini-3 Flash, DeepSeek Reasoner) and memory-centric systems (e.g., MemoryOS, Mem0).

• Extraction Limitations: Models struggle significantly with implicit user traits. Extraction recall for indirect expressions is much lower than for direct ones. Memory-oriented systems sometimes underperform their backbone models due to "storage-policy bias" (storing everything rather than filtering for relevance).
• Update Ceiling: Even the strongest models hit a performance ceiling in memory updating (GPT-5.2 scored ~81.5). There is a weak correlation between a model's ability to add new memories and its ability to modify conflicting ones, suggesting distinct underlying mechanisms.
• Retrieval Sensitivity: While models perform well in low-noise environments, retrieval accuracy declines sharply as the number of distractor memories increases (e.g., dropping significantly at 1,000 distractors). Classical retrieval methods (BM25, embeddings) fail catastrophically in high-noise personalized scenarios compared to LLM-based reasoning.
• Utilization & Bias:
  • No single model dominates all utilization dimensions.
  • Memory systems introduce bias: While memory-augmented systems improve Persona Awareness and Preference Following, they often degrade Virtual-Reality Awareness and Emotional Intelligence. Current systems prioritize memory capacity over effective filtering and emotional regulation, leading to responses that are factually aligned but emotionally flat or contextually confused.

5. Significance

AlpsBench addresses a critical gap in the AI community by shifting the focus from synthetic, static evaluations to real-world, dynamic personalization.

• Research Direction: It highlights that current LLMs are not yet ready for "lifelong" personalization due to failures in handling implicit signals, managing memory conflicts, and maintaining emotional resonance.
• System Design: The findings suggest that future personalization systems must move beyond simple "plug-and-play" retrieval modules. They require logic-aware retrieval layers and architectures that prioritize memory accuracy, robust filtering, and emotional intelligence over mere storage capacity.
• Standardization: By providing a standardized, human-verified benchmark, AlpsBench enables the community to rigorously compare and improve the next generation of personalized AI assistants.