ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?

Technical Summary: ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?

1. Problem Statement

As Large Language Models (LLMs) evolve into autonomous shopping agents (e.g., ChatGPT DeepResearch), there is a critical need for robust evaluation frameworks. Existing e-commerce benchmarks suffer from several limitations:

• Closed-World Assumptions: Most benchmarks use static, bounded product catalogs, failing to reflect the noisy, heterogeneous, and constantly changing nature of the real-world web.
• Lack of Implicit Reasoning: Real users often express needs through goal-oriented intents (e.g., "suitable for elderly use") rather than explicit technical constraints, requiring models to infer latent requirements.
• Absence of Safety Evaluation: Current benchmarks rarely assess whether an agent can recognize and avoid safety-critical risks (e.g., recommending a metal pan for use in a microwave).
• Insufficient Justification Testing: Most evaluations focus on what was recommended rather than why, ignoring the necessity of factual, evidence-grounded reasoning in consumer decision-making.

2. Methodology

The authors introduce ShoppingComp, an end-to-end open-world benchmark designed to evaluate shopping agents across three core tasks: Product Retrieval, Expert-level Report Generation, and Safety-Critical Decision Making.

A. Data Collection & Construction (Human-in-the-Loop):
The benchmark consists of 145 instances and 558 scenarios, curated by 35 domain experts through a four-step pipeline:

1. Question Collection: Three types of queries are used: Synthesized User Questions (attribute-based), Expert Questions (complex, real-world scenarios), and Safety Questions (risk-prone scenarios).
2. Rubric Generation: Experts and LLMs generate detailed rubrics that decompose complex intents into atomic, verifiable requirements.
3. Candidate Product Construction: A diverse pool of products is gathered via web agents, expert curation, and similarity-based retrieval, each linked to verifiable evidence (URLs, manuals, etc.).
4. Verification & Filtering: Human annotators verify product-rubric matches and filter out trivial or redundant instances to ensure high difficulty.

B. Unified Evaluation Framework (LLM-as-a-Judge):
The authors utilize specialized LLM-based verifiers (grounded in Google Search) to assess four dimensions:

• Answer Match (F1): Measures the precision and recall of retrieved products against ground truth.
• Score of Products (SoP): Evaluates the proportion of recommended products that satisfy all rubric-defined constraints.
• Rationale Validity (RV): Assesses whether the report's justifications are factually accurate and logically sound.
• Safety Rubric Pass Rate: Measures the ability to recognize and respond to "safety traps."

3. Key Contributions

• A Realistic Benchmark: A high-complexity, expert-curated dataset that moves beyond closed-world catalogs to open-world web environments.
• Multi-Dimensional Evaluation: A framework that evaluates not just retrieval accuracy, but also reasoning faithfulness, scenario coverage, and safety awareness.
• Empirical Characterization of the "Trust Threshold": The paper identifies specific capability gaps that must be bridged before AI agents can be proactively trusted with real-world financial and safety decisions.

4. Results and Analysis

The experimental results reveal a significant performance gap between state-of-the-art (SOTA) LLMs and human experts:

• Retrieval Gap: Even top-tier models like GPT-5.2 achieve only 17.76% Answer Match F1, compared to 30.02% for human experts.
• Behavioral Divergence: Humans employ a "constraint-first" strategy (high precision, lower recall), whereas LLMs exhibit a "semantic expansion" strategy (high recall, low precision), often recommending products that satisfy some but not all constraints.
• Reasoning vs. Browsing: "DeepResearch" systems (agentic workflows) excel at Scenario Coverage (finding many relevant contexts) but suffer in Rationale Validity (the quality of the synthesized reasoning), suggesting that more browsing does not automatically lead to better reasoning.
• The Safety Crisis: Safety performance is alarmingly low. The strongest models achieve safety pass rates below 40%, whereas experts score 77.08%. This indicates a high risk of models providing dangerous recommendations in real-world deployment.

5. Significance

ShoppingComp serves as a rigorous "stress test" for the next generation of AI agents. By exposing the failure modes in implicit requirement inference, information grounding, and risk awareness, the paper provides a roadmap for future research. It shifts the focus of LLM development from mere "information retrieval" to "reliable, safe, and expert-level decision support," which is essential for the commercial viability of autonomous shopping assistants.