Towards Personalized Deep Research: Benchmarks and Evaluations

This paper introduces PDR-Bench, the first benchmark for evaluating personalization in Deep Research Agents, along with the PQR Evaluation Framework to assess performance across personalized, open-ended research tasks.

The Gist

Imagine you have a super-smart research assistant who can read thousands of books, browse the entire internet, and write a detailed report for you in minutes. This is what Deep Research Agents (DRAs) are today. They are like brilliant librarians who can find any fact instantly.

However, there's a problem. Right now, these assistants are like generic tour guides. If you ask them, "Plan a trip to Japan," they will give you the same standard itinerary they give to everyone else. They don't know if you are a budget backpacker, a luxury traveler with a family, or a foodie who hates spicy food. They miss the "personal" part.

This paper, titled "Towards Personalized Deep Research," is like a new rulebook and a new test designed to fix that. Here is the breakdown in simple terms:

1. The Problem: The "One-Size-Fits-All" Trap

Currently, we test these AI assistants on how well they find facts (like a trivia quiz). But in the real world, a good answer depends on who is asking.

• The Old Way: Testing if the AI knows the capital of France.
• The New Need: Testing if the AI knows that you are a student on a tight budget who wants to visit Paris for a weekend, not a CEO looking for a 5-star business trip.

2. The Solution: PDR-Bench (The "Personalized Exam")

The authors created a new test called PDR-Bench. Think of this as a giant role-playing game for AI.

• The Characters: They built 25 real, detailed user profiles (like a 20-year-old student, a 34-year-old busy dad with a dog, a psychology grad student). These aren't fake; they are based on real people's habits, budgets, and dreams.
• The Missions: They created 50 complex research tasks (like "Plan a PhD application," "Design a marathon training plan," or "Invest in stocks").
• The Mix: They paired every task with every user profile, creating 250 unique scenarios.
  • Analogy: Imagine a chef (the AI) who has to cook a meal. The old test just asked, "Is the food cooked?" The new test asks, "Is this meal cooked perfectly for this specific customer who is gluten-free, hates cilantro, and is in a rush?"

3. The Scoring System: The "PQR" Framework

How do you grade a personalized report? The authors invented a three-part scorecard called PQR:

• P = Personalization (The "Is this for me?" Score):
  Does the report sound like it was written just for you? Did it consider your budget, your knowledge level, and your goals?

  • Metaphor: If you ask for a "simple explanation," and the AI gives you a PhD thesis, you get a low P-score. If it gives you a friendly, easy-to-read guide, you get a high P-score.

• Q = Quality (The "Is it well-written?" Score):
  Is the report logical, deep, and easy to read? Does it make sense?

  • Metaphor: Even if a report is perfectly personalized, if it's messy and confusing, it's a bad report. This checks the craftsmanship.

• R = Reliability (The "Is it true?" Score):
  Did the AI make things up? Did it check its sources?

  • Metaphor: This is the fact-checker. It ensures the AI isn't just hallucinating (making up) facts to sound smart.

4. What They Found (The Results)

The authors tested many different AI systems (from big tech companies like Google and OpenAI to open-source projects) using this new exam.

• The Surprise: The "Open-Source" agents (often built by communities) were surprisingly good at Personalization. They listened to the user's specific needs better than the big corporate agents.
• The Trade-off: The big commercial agents were great at Facts (Reliability) but sometimes felt a bit robotic and generic (lower Personalization).
• The "Context" Problem: When the AI had to guess who the user was just by reading a chat history (without a clear profile), it struggled. It's like trying to guess someone's favorite movie just by hearing them talk about the weather. It needs a clear "User Profile" to do its best work.

5. Why This Matters

This paper is a wake-up call for the AI industry. It says: "Stop just making AI that is smart. Start making AI that is helpful."

By creating this benchmark, the authors are giving developers a map. They are saying, "Here is exactly how to measure if your AI is truly personal." This will push the next generation of AI assistants to become less like search engines and more like trusted personal advisors who actually know you, your life, and your needs.

In short: This paper built a new gym where AI agents can train to stop being generic robots and start becoming the perfect, personalized research partners we all wish we had.

Technical Summary

Here is a detailed technical summary of the paper "Towards Personalized Deep Research: Benchmarks and Evaluations":

1. Problem Statement

Deep Research Agents (DRAs) have emerged as powerful AI systems capable of autonomously conducting complex investigations, retrieving information, and generating comprehensive reports. However, current evaluation methodologies suffer from two critical limitations:

• Lack of Personalization: Existing benchmarks (e.g., GAIA, BrowseComp, DeepResearch Bench) focus almost exclusively on factual accuracy and task completion in a generic context. They fail to assess whether an agent can adapt its reasoning, information filtering, and reporting style to specific user needs, preferences, and backgrounds.
• Inadequate Scope: Existing personalization benchmarks (e.g., LaMP, PersonaGym) are confined to narrow domains like dialogue or recommendation systems and do not address the complexity of multi-step, tool-augmented deep research workflows.

This gap prevents the development of AI assistants that can truly act as personalized partners in real-world decision-making scenarios (e.g., investment planning, career development, health management).

2. Methodology

The authors propose a comprehensive solution comprising a new benchmark and a novel evaluation framework.

A. Personalized Deep Research Bench (PDR-Bench)

PDR-Bench is the first benchmark designed specifically to evaluate personalization in DRAs. Its construction involves three main stages:

1. Task Formulation: 50 diverse deep research tasks were designed across 10 domains (e.g., Education, Career, Health, Finance). These tasks were validated by domain experts and a committee to ensure complexity, clarity, and relevance to personalized scenarios.
2. Authentic User Profile Construction: Instead of synthetic personas, the authors collected data from 25 real volunteers.
   • Structured Explicit Personas: Volunteers mapped their real demographics, occupation, family status, and preferences into a structured schema.
   • Dynamic Personalized Context: Annotators simulated daily interactions (logs, conversations with an assistant) to capture evolving interests and implicit preferences.
3. Query Generation: The 50 tasks were paired with the 25 user profiles using a committee-guided alignment protocol to ensure relevance. This resulted in 250 unique user-task pairs, creating a realistic testbed where the "correct" answer depends heavily on the specific user context.

B. PQR Evaluation Framework

To assess system performance, the authors introduced the PQR Framework, which evaluates reports along three orthogonal dimensions:

1. Personalization Alignment (P): Measures how well the report aligns with the user. It is dynamic and LLM-driven, consisting of four sub-dimensions:
   • Goal Alignment: Does the report address the user's explicit/implicit goals?
   • Content Alignment: Is the depth and topic suitable for the user's knowledge level?
   • Presentation Fit: Does the style and structure match the user's preferences?
   • Actionability & Practicality: Is the advice actionable for this specific user?
   • Mechanism: An LLM meta-evaluator dynamically assigns weights to these dimensions and generates specific sub-criteria for each user-task pair before scoring.
2. Content Quality (Q): Evaluates the intrinsic quality of the report regardless of the user, focusing on Depth & Insight, Logical Coherence, and Clarity & Readability.
3. Factual Reliability (R): Assesses the truthfulness of the report through an automated grounding process. It extracts factual claims, verifies them against source URLs using a reader API, and calculates Factual Accuracy (percentage of supported claims) and Citation Coverage (percentage of claims with citations).

3. Key Contributions

• First Personalized DRA Benchmark: Introduced PDR-Bench, featuring 250 realistic queries combining 50 complex tasks with 25 authentic, multi-modal user profiles.
• Novel Evaluation Framework: Proposed the PQR framework, which moves beyond static metrics to a dynamic, user-centered assessment of personalization, quality, and reliability.
• Comprehensive Empirical Analysis: Conducted extensive experiments across a wide spectrum of systems, including commercial DRAs (e.g., Gemini, O3, Perplexity), open-source agents (e.g., OAgents, DeerFlow), and LLMs with search tools.
• Human Consistency Validation: Validated the LLM-based judges against human experts, demonstrating high agreement (using GPT-5 as the primary judge).

4. Experimental Results

The evaluation of various systems on PDR-Bench revealed several key insights:

• Open-Source vs. Commercial: Open-source agents (specifically OAgents) demonstrated superior Personalization Alignment (scoring 6.64 vs. 6.58 for the best commercial agent), effectively tailoring content to user goals. However, commercial agents (e.g., Gemini-2.5-Pro) excelled in Factual Reliability and consistent content quality.
• Limitations of Search-Only LLMs: LLMs equipped with search tools (e.g., GPT-4.1 w/Search) generally underperformed compared to dedicated DRAs, particularly in citation coverage and personalization depth.
• Impact of Information Availability:
  • Providing explicit User Personas yielded the highest personalization scores.
  • Providing only Context (conversational logs) improved performance over "Task Only" baselines but still lagged behind explicit personas, indicating agents struggle to infer deep preferences from unstructured data.
• Memory Systems: Experiments with memory systems (Mem0, Memory OS, O-Mem) showed that while they can improve personalization over a "No Memory" baseline, they still fail to synthesize context into explicit personas as effectively as direct persona input.

5. Significance

This work establishes a rigorous foundation for the next generation of AI research assistants. By shifting the evaluation focus from "what is the answer?" to "is this the right answer for this user?", the paper:

• Highlights the critical need for agents to integrate user profiles into their reasoning loops, not just as a prompt injection but as a core component of the research workflow.
• Identifies the current trade-off between personalization capabilities (stronger in open-source) and factual robustness (stronger in commercial systems).
• Provides a scalable, reproducible benchmark (PDR-Bench) and a standardized evaluation protocol (PQR) that will guide future research in building truly personalized, reliable, and high-quality AI agents.