💬 NLP

Language Models Don't Know What You Want: Evaluating Personalization in Deep Research Needs Real Users

This paper introduces MyScholarQA, a personalized Deep Research tool that outperforms baselines on synthetic benchmarks but reveals critical, undetectable limitations through real-user interviews, arguing that genuine progress in personalization requires evaluation with actual users rather than relying solely on LLM judges.

Original authors: Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik

Published 2026-03-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Smart" Librarian Who Doesn't Know You

Imagine you walk into a massive, high-tech library (the internet's entire collection of scientific papers) and ask a super-smart librarian (an AI) to find you the best books on a specific topic.

In the past, the librarian would just grab the most popular books or the ones with the most stars. But you, a researcher, have a specific style. Maybe you hate long introductions, maybe you only care about experiments, or maybe you prefer visual diagrams over text. You want the librarian to know you so they can curate the perfect list just for your brain.

This paper introduces a new system called MyScholarQA. It's a "Deep Research" tool that tries to be that personal librarian. It does three things:

Reads your past work to build a "profile" of who you are.
Asks you what you want by suggesting specific ways to tweak the search (like "skip the basics" or "focus on math").
Writes a custom report based on your profile and your tweaks.

The Problem: The "Fake" Test vs. The Real Test

The researchers wanted to see if their new "Personal Librarian" was actually good. So, they tried two different ways to test it.

1. The Simulation Test (The "Robot Judge")

First, they did what most computer scientists do: they created a fake test.

The Setup: They made up fake researchers and fake papers. They used other AIs (called "LLM Judges") to grade the reports.
The Result: The AI Judges gave the new system a gold star! They said, "Great job! It cited the right papers and followed the rules."
The Analogy: Imagine a cooking contest where the judges are robots. The robot tastes a soup and says, "Perfect! It has salt and pepper." But the robot can't tell if the soup tastes good to a human, or if it's too spicy for a specific person's palate. The robot is just checking boxes, not tasting the food.

2. The Real User Test (The "Taste Test")

Then, the researchers did something rare: they asked 21 real human researchers to use the system.

The Setup: Real people used the tool, edited the profiles, and rated the reports.
The Result: The humans were much pickier. They found nine specific ways the system failed that the robot judges completely missed.
The Analogy: When a real human tasted the soup, they said, "It's too salty for me," or "I don't like the texture," or "You forgot to mention the main ingredient I asked for." The robot never noticed these things because it was just looking for "salt" and "pepper."

The Nine "Hidden" Flaws

The paper found that while the AI thought it was doing great, it was actually making subtle mistakes that only humans would catch. Here are a few examples:

The "Over-Confident" Profile: The AI would say, "You are an expert in Quantum Physics," when the user was actually just a beginner. It was flattering but wrong.
The "Off-Topic" Suggestion: The AI suggested actions that were technically correct but totally irrelevant to what the user actually wanted to know.
The "Vague" Report: The AI wrote a report that was too high-level. The user wanted deep details, but the AI gave a summary.

The Big Lesson: The "Robot Judges" (other AIs) were terrible at predicting what real humans wanted. They got the score wrong 100% of the time compared to real human feedback.

The Takeaway: Stop Simulating, Start Talking

The authors argue that the field of Artificial Intelligence is relying too much on simulations. We are building AI tools and testing them with other AIs, thinking, "If the AI likes it, humans will too."

This paper says: No, that doesn't work.

The Metaphor: You wouldn't design a new car and only test it in a video game. You need to drive it on real roads with real drivers to see if the seats are comfortable or if the brakes feel right.
The Conclusion: To make AI that truly helps people, we need to stop just using "Robot Judges" and start talking to Real Users. We need to watch how they use the tool, listen to their complaints, and watch them get frustrated.

Summary in One Sentence

Just because an AI thinks it's doing a great job at personalizing a report doesn't mean it actually knows what you want; you can't test a personal assistant with a robot, you have to test it with a real person.

1. Problem Statement

Deep Research (DR) systems, which synthesize scientific literature to answer complex queries, are becoming essential for researchers. However, current DR tools lack personalization; they treat all users as generic entities, failing to adapt reports to a researcher's specific background, expertise level, or ongoing work.

The paper identifies a critical gap in the evaluation of personalized DR systems:

Over-reliance on Offline Metrics: The NLP community predominantly evaluates personalization using synthetic datasets and Large Language Model (LLM) judges.
The "Simulation Gap": The authors hypothesize that these offline methods fail to capture nuanced user needs and specific failure modes that only real human users can identify. They argue that "easy-to-use" LLM judges are insufficient proxies for real user satisfaction in complex personalization tasks.

2. Methodology

The authors introduce MyScholarQA (MYSQA), the first open-source personalized DR system, and evaluate it through a multi-stage methodology combining offline benchmarks and online user studies.

A. The System: MyScholarQA (MYSQA)

MYSQA operates on an adaptive hypermedia framework with three distinct steps:

Profile Inference: The system infers a persistent user profile ( $P$ ) from a set of papers the user selects. Using an LLM, it generates sentence-level inferences across five dimensions: Knowledge, Research Style, Writing Style, Audience, and Positions. Users can edit or disable these inferences.
Action Proposal: Upon receiving a query ( $q$ ), MYSQA proposes a list of personalized actions ( $A$ ) tailored to the user's profile. These actions modify the retrieval, organization, and generation of the report (e.g., "Focus on scientific QA papers," "Skip basic definitions"). Users can toggle or edit these actions.
Report Synthesis: A multi-LLM pipeline generates a multi-section report that answers the query while executing the selected actions. The system highlights specific text spans where personalization occurred to ensure transparency.

B. Evaluation Strategy

The study employs a "Formative $\to$ Offline $\to$ Online" pipeline:

Formative Study: A small-scale study (5 users) validated the UI and workflow, confirming that users found paper-based profiling intuitive but desired more transparency.
Offline Evaluation (Synthetic):
- Dataset: Created a synthetic benchmark pairing 572 DR queries with simulated user paper sets (derived from CS-PaperSum).
- Metrics: Used LLM judges (Gemini-2.5 Flash, GPT-4.1, etc.) to evaluate profile accuracy, action coherence, and report quality (citation precision, action adherence).
Online Evaluation (Real Users):
- Study Design: Conducted 90-minute interviews with 21 active DR users (CS researchers). Participants built profiles, selected actions, and rated reports while "thinking aloud."
- LLM Judge Simulation: The authors attempted to replicate the online findings offline by training LLM judges to predict user satisfaction on the specific error types found in the interviews.

3. Key Contributions

MYSQA System: The first open-source, personalized Deep Research tool with a live demo, featuring a three-step workflow (Profile $\to$ Action $\to$ Report) that allows user control and transparency.
Synthetic Benchmark: A new dataset and metric suite for offline evaluation of personalized DR, showing that MYSQA outperforms baselines (including commercial tools like OpenAI DR and Perplexity) in citation accuracy and action adherence.
Discovery of "Invisible" Errors: The study identified nine nuanced failure modes in personalized DR that offline metrics and LLM judges completely missed. These include:
- Profile Errors: Overclaiming expertise, misidentifying domain conventions, and making false contrasts.
- Action Errors: Actions that are too narrow (over-constraining) or off-topic.
- Report Errors: Content that is too vague (uninformative) or ignores implicit user requirements.
Failure of LLM Judges: The authors demonstrated that state-of-the-art LLM judges cannot predict user satisfaction for these nuanced errors. Their accuracy in predicting user satisfaction was no better than a majority-class baseline, proving that synthetic evaluation is insufficient for personalization.
Design Lessons: Four key lessons for future personalized DR design:
- Balance user control with effort (users are willing to invest effort upfront for better long-term utility).
- Make personalization easy to digest (structured profiles/actions are better than long prompt engineering).
- Go beyond papers (incorporate code, formulas, and other modalities).
- Use mixed evaluation designs (offline for baseline quality, online for user value).

4. Results

Offline Performance: MYSQA achieved superior scores compared to baselines (SQA, STORM, OpenAI DR) in 3 out of 5 report quality metrics and consistently beat its base system (SQA) in action adherence.
Online Satisfaction: Users were satisfied with 73% of MYSQA's outputs (profiles, actions, and reports).
The Critical Finding: Despite high offline scores, the online study revealed that 27% of outputs contained significant personalization flaws. Crucially, when LLM judges were tasked with predicting these specific failures (e.g., "Is this profile inference too broad?"), they failed to outperform random guessing.
- Example: An LLM judge might rate a profile as "accurate" because it cites papers, while a human user rejects it because the inference is too generic or misrepresents their specific research stance.

5. Significance and Conclusion

The paper delivers a paradigm-shifting argument for the NLP community: Personalization cannot be solved or validated solely through synthetic data and LLM judges.

Methodological Shift: It challenges the prevailing trend of "LLM-as-a-judge" for personalization tasks, showing that these models lack the contextual understanding to detect subtle mismatches between a system's output and a user's actual identity and needs.
Human-Centric Design: The authors advocate for a "real user" pillar in personalization research. They argue that while synthetic benchmarks are useful for checking basic functionality (e.g., citation accuracy), real progress requires real users to uncover the qualitative nuances that define true helpfulness.
Future Directions: The paper calls for longitudinal studies and mixed-method evaluations to guide the development of DR systems that are not just factually correct, but genuinely aligned with individual researchers' workflows and identities.

In summary, the paper proves that while Deep Research tools can synthesize information well, they currently "don't know what you want" in a deep, personalized sense, and the only way to fix this is to listen to real users rather than relying on simulated proxies.