Technical Summary: Beyond Expert Users – Agents Should Help Users Construct Preferences, Not Just Elicit Them

1. Problem Statement

Current agentic recommender systems (e.g., ChatGPT Shopping Research, Amazon Rufus) and the benchmarks used to evaluate them operate under a critical, unrealistic assumption: that users are experts with well-formed, static preferences that can be fully specified via clarifying questions. In reality, non-expert users often lack the domain knowledge to define preferences for all relevant features. They may be unable to answer questions about specific attributes (e.g., "lug depth" for hiking boots or "viscose vs. lyocell" for fabrics) because they do not yet understand what those features mean or how they impact the user experience.

The paper argues that when agents rely solely on clarifying questions, they fail to help users construct the preferences necessary to evaluate recommendations. Consequently, even if an agent successfully finds the ideal item, the user may be unable to identify it as the best choice due to an underdeveloped preference state. This leads to a failure in the "team" performance of the human-agent system, not just the agent's retrieval capability.

2. Methodology

2.1 The COPREF Model

The authors introduce COPREF (Co-constructed Preferences), a formal model of user-agent interaction where preferences are not retrieved but constructed through dialogue.

• Preference State ($S_t$): A subset of features $F$ over which the user has formed a preference at turn $t$. This state evolves from an incomplete initial state ($S_1$) based on agent actions.
• SEC Framework: Drawing from Information Economics (Search-Experience-Credence), the model partitions features into three types, each requiring a different agent dialog action to unlock:
  1. Search Features ($F_s$): Users know their preference (e.g., price, size). Unlocked via clarifying questions.
  2. Experience Features ($F_e$): Users need to see concrete examples to decide (e.g., "is this pattern too busy?"). Unlocked via example options.
  3. Credence Features ($F_c$): Users need domain knowledge to understand the feature (e.g., "what is a yoke panel?"). Unlocked via technical explanations.
• Mechanism: A feature is added to the user's preference state only if the agent performs the specific dialog action corresponding to the feature's SEC type.

2.2 The COSHOP Benchmark

To evaluate agents under this model, the authors introduce COSHOP, an interactive benchmark for agentic recommender systems across three domains:

• Datasets: H&M (fashion), MovieLens (films), and Goodreads (books).
• User Simulation: Unlike typical simulators that prompt a persona once, COSHOP uses a simulator (based on gpt-oss-120b) explicitly anchored to the current preference state $S_t$. The simulator's SEC split (which features are search/experience/credence) is personalized based on the user's historical purchase/rating data.
• Interaction Protocol:
  1. Conversation: Agent and user converse for up to 5 turns (budget: 20 questions, 10 unique items shown).
  2. Research: Agent searches the catalog (budget: 250 retrievals).
  3. Report: Agent generates $k=5$ recommendations.
  4. Selection: The user selects one item based on their final preference state $S_{T+1}$.
• Metrics:
  • State Completeness: Fraction of the user's full preference set recovered.
  • Agent Recall: Whether the target item $x^*$ appears in the top-$k$ recommendations.
  • Team Accuracy: Whether the user successfully selects $x^*$ from the report. This is the primary metric, bounded by both recall and the richness of the user's preference state.

2.3 Human Study Validation

A study with $n=25$ participants validated COPREF's core claims:

• Non-expert users possess a mix of search, experience, and credence features (no user had only search features).
• Appropriate dialog actions (examples for experience, explanations for credence) successfully "unlock" preferences that were previously unformulated.
• SEC classifications vary by individual; a feature is not universally credence or experience but depends on the user's specific domain knowledge.

3. Key Contributions

1. COPREF Model: A formalization of preference construction that challenges the "expert user" assumption, distinguishing between features that require questions, examples, or explanations to be defined.
2. COSHOP Benchmark: An open-sourced interactive benchmark evaluating agents on their ability to co-construct preferences with non-expert users, measuring "team accuracy" rather than just retrieval.
3. Empirical Analysis: A comprehensive evaluation of five frontier language models (GPT and Claude families) demonstrating that current agents fail to expand user knowledge effectively.

4. Results

Evaluations of five frontier models (including gpt-5.2, claude-sonnet-4.6, and gpt-oss-120b) on COSHOP yielded the following findings:

• Performance Drop: While models score highly (e.g., 94.3% for claude-sonnet-4.6) when given fully specified preferences upfront, their Team Accuracy on COSHOP drops significantly (max 56% across all models and domains).
• Failure Modes:
  1. Underdeveloped Preference States: Agents fail to perform the necessary dialog actions (examples, explanations) to unlock experience and credence features. Consequently, users cannot distinguish the target item from distractors even when it is present in the report.
  2. Poor Report Quality: Agents often omit or misrepresent item features in their final reports, preventing users from evaluating options correctly.
• Interaction Strategies:
  • Models tend to rely heavily on clarifying questions (often redundant or confirmation-based) rather than providing examples or explanations.
  • No model proactively provided technical explanations for credence features.
  • Larger models (e.g., gpt-5.2) exhibited an "execution bias," ending conversations prematurely to move to the research phase, leaving the task underspecified.
  • Smaller models exhausted the conversation budget but failed to introduce new features, often looping on known attributes.
• Interventions: Providing agents with user history did not significantly improve performance. However, overriding agent item descriptions with complete catalog data (simulating perfect report quality) substantially improved team accuracy, confirming that report quality and preference state development are the primary bottlenecks.

5. Significance and Claims

The paper claims that the primary limitation of current agentic recommender systems is not their ability to search or retrieve items, but their inability to teach users what they need to know to make a decision.

• Shift in Paradigm: The authors argue that the field must move from viewing agents as "preference elicitors" (retrieving static preferences) to "preference constructors" (actively building user knowledge).
• Team Accuracy: Success should be measured by the joint capability of the human-agent team. An agent that finds the perfect item but fails to help the user understand why it is perfect has failed its task.
• Future Directions: The paper suggests that future work should focus on training agents to recognize when a user lacks domain knowledge and to proactively deploy examples and explanations, rather than relying solely on clarifying questions. It also calls for extending the COPREF model beyond recommendation settings to other task-oriented domains.

The authors remain modest, noting that their model simplifies preference construction (assuming single-step unlocking and monotonic state growth) and that even under these favorable conditions, current frontier models struggle to adapt.