Technical Summary: RecoAtlas

Problem Statement

Large Language Model (LLM) recommendation agents are increasingly tasked with generating structured reports containing sets of items accompanied by natural-language justifications, rather than simple ranked lists. Existing evaluation paradigms fail to adequately capture the complexity of this setting. Current benchmarks often reduce the problem to reranking small, pre-selected candidate sets containing a single ground-truth item, which evaluates local re-ranking but not the ability to search large catalogs or construct coherent multi-item sets. Furthermore, evaluations relying on "LLM-as-a-Judge" primarily measure semantic plausibility—assessing whether a report is coherent and well-justified—without verifying if the recommended items align with actual user behavior or utility. Consequently, a report can be semantically plausible yet recommend products users rarely purchase or combine. There is a lack of open, behavior-grounded benchmarks that evaluate agents on set-level utility, complementarity, and diversity while distinguishing between semantic coherence and behavioral effectiveness.

Methodology

The authors introduce RecoAtlas (Recommendation Agent Tool-Level Assessment for Shopping), a benchmark and toolkit designed to evaluate shopping agents under behavior-grounded, set-level objectives. The framework operates within a controlled tool environment where agents must search a large product catalog, invoke specific tools, and construct a fixed-size shopping report (e.g., $K=20$ items) with justifications.

Task Families

RecoAtlas defines two primary task types, both requiring query-item relevance but differing in item-item structure:

1. Comparative Shopping: The user seeks alternatives for a broad need. The goal is to provide relevant items while avoiding near-duplicates (emphasizing non-redundancy/diversity).
2. Bundle Shopping: The user seeks a coherent setup or kit (e.g., "guitarist starter pack"). The goal is to provide items that play complementary roles and work together (emphasizing complementarity).

Evaluation Metrics

RecoAtlas employs three complementary, separate evaluation views to avoid collapsing report quality into a single scalar:

1. Exact Recovery (SetHit@K): Measures the fraction of held-out ground-truth items (derived from user interaction data) recovered in the final report. For bundle tasks, this measures recovery of a target set rather than a single item.
2. Learned Reward Models: Three behavior-grounded proxies trained on user interaction data (purchases and co-purchases) provide component-level scores:
   • Relevance: A query-item dual encoder trained with multi-negative BPR loss.
   • Complementarity: An item-item dual encoder trained on co-purchase data using contrastive InfoNCE loss to identify items frequently purchased together.
   • Diversity (Non-redundancy): A score based on cosine similarity within subcategories, penalizing near-duplicates.
3. LLM Judges: Separate assessments of relevance, complementarity, diversity, and explanation quality. These are kept distinct from behavior-grounded metrics to highlight the potential disconnect between semantic plausibility and behavioral utility.

Tool Suite

A central feature is the provision of recommendation-specific tools that agents can call (up to 10 calls per episode):

• Search Tools: Retrieve items based on relevance (semantic vs. behavior-aligned versions).
• Complementarity Tools: Expand item sets with behaviorally compatible items.
• Substitutability Tools: Prune near-duplicates based on subcategory and similarity thresholds.
  The benchmark includes faulty tool variants that inject controlled errors (e.g., returning low-quality candidates or failing to prune duplicates) to diagnose agent robustness.

Key Contributions

1. A Large-Catalog, Set-Level Benchmark: Unlike prior work that relies on shortlist reranking, RecoAtlas evaluates agents that must search a large catalog without a provided shortlist, construct fixed-size multi-item reports, and recover sets of ground-truth items.
2. Behavior-Grounded Evaluator and Tool Suite: The framework combines exact recovery, learned reward models, and LLM judging. It uniquely provides a suite of semantic, behavior-aligned, and faulty tools to test agent reasoning, signal utilization, and tool-use policies.
3. Diagnostic Empirical Study: The authors conduct ablation studies across model families, task types, and tool qualities to demonstrate that RecoAtlas exhibits properties of a meaningful benchmark for agentic systems.

Experimental Results

Experiments were conducted on three Amazon product categories (Musical Instruments, Electronics, Video Games) using various proprietary and open-weight LLM backbones (e.g., GPT-4.1, Qwen3, Nemotron).

• Semantic Plausibility vs. Utility: A significant misalignment was observed between LLM judges and utility metrics. Agents using semantic tools often achieved higher LLM-judge scores but significantly lower SetHit@20 (behavioral utility) compared to agents using behavior-aligned tools. This confirms that semantic plausibility is not a reliable proxy for behavioral utility.
• Tool Necessity: In-context LLMs without tools performed poorly (SetHit@20 $\approx$ 0.01–0.03). Adding search tools improved comparative shopping performance but was insufficient for bundle shopping. Full utility required the combination of search, complementarity, and substitutability tools, with bundle performance heavily dependent on the complementarity tool.
• Scaling Laws: Performance scaled with model capacity and test-time compute. Larger models (e.g., 32B parameters) with reasoning enabled showed substantial gains over smaller models and models without reasoning, particularly in bundle tasks.
• Task-Specific Capabilities: While comparative shopping was dominated by relevance retrieval, bundle shopping required distinct agentic capabilities to construct coherent sets. Open-weight models (e.g., Nemotron 49B, Mistral Small 24B) often outperformed proprietary models on bundle tasks by more aggressively utilizing complementarity signals.
• Robustness: Agents exhibited a trade-off between performance and robustness. Aggressive tool use improved performance with clean tools but made agents more sensitive to faulty tools, whereas conservative agents retained a higher fraction of performance under noise.

Significance and Claims

The paper claims that RecoAtlas provides a foundation for developing and evaluating shopping assistants that optimize not just for plausible recommendations, but for coherent, behaviorally grounded recommendation sets.

The authors position RecoAtlas as a reproducible offline testbed that complements, rather than replaces, online A/B testing. By separating semantic coherence from behavioral utility and introducing controlled tool environments, RecoAtlas enables the diagnosis of whether performance gains stem from stronger reasoning, better signals, or more effective tool-use policies. The framework is designed to be extensible, allowing for the generation of new tasks and the application of optimization strategies like reinforcement learning or supervised fine-tuning, while maintaining held-out splits for final evaluation.