Technical Summary: RetailBench

Problem Statement

While Large Language Model (LLM) agents have demonstrated proficiency in short-horizon, well-scoped tasks (e.g., code editing, information retrieval), their ability to sustain coherent decision-making in dynamic, long-horizon environments remains uncertain. Existing benchmarks often isolate individual subproblems or lack the coupled, closed-loop dynamics of real-world business operations. Consequently, the community lacks an open, data-grounded benchmark to evaluate whether tool-using LLM agents can maintain persistent objective alignment and stable behavioral consistency over extended periods, particularly when local errors compound into delayed consequences that reshape future state distributions.

Methodology: RetailBench

The authors introduce RetailBench, a data-grounded simulation benchmark designed to evaluate tool-using LLM agents in the context of single-store supermarket operations. The environment is modeled as a Partially Observable Markov Decision Process (POMDP), $M = (S, A, O, T, \Omega, R)$, where the agent interacts with a simulator through a specific set of tools.

Environment Construction

• Data Grounding: The simulator is anchored in real-world retail data from the Dominick's dataset (University of Chicago Booth School of Business) to define the product universe (96 products across 20 categories), historical prices, sales trajectories, and procurement costs.
• Dynamic Feedback: Review texts are sampled from the Amazon Reviews 2023 corpus, and news events are adapted from financial news articles. These introduce corpus-grounded dynamics where hidden metadata drives demand and supply-side effects.
• Modules: The environment couples seven modules: Inventory, Suppliers & Orders, Shelf, Customer Demand, Reviews & Returns, News Events, and Finance & Records.
• State and Actions:
  • State ($S$): Includes product attributes, inventory (with aging), shelf assortment, supplier candidates, demand factors, external news, and financial accounting.
  • Actions ($A$): Agents select from three tool groups:
    1. Read-only ($A_{read}$): Querying funds, inventory, prices, sales history, supplier quotes, and reviews.
    2. Auxiliary ($A_{other}$): Constrained analysis, memory management, and code execution (sandboxed).
    3. State-changing ($A_{act}$): Updating prices, placing replenishment orders, managing shelf assortment, writing notes, and terminating the day.
• Simulation Horizon: The benchmark supports thousand-day-scale simulations. The evaluation protocol uses a 180-day horizon.

Evaluation Protocol

• Models: Seven contemporary LLMs were evaluated using three prompting-based frameworks: ReAct, Reflection, and Plan-and-Act.
• Selection Rule: Due to the variance in long-horizon performance, the authors employ a "survival-first" selection rule: for each model, the run with the longest survival horizon is selected, with final net worth and total sales as tie-breakers.
• Baseline: A privileged oracle policy is included as a non-agent reference. This policy has direct access to structured simulator fields (e.g., hidden supplier quality scores) and follows manually specified, quality-aware operating rules.

Metrics

Performance is measured across five categories:

1. Operational Viability: Final Net Worth and Survival Days.
2. Sales Effectiveness: Total Sales, Daily Sold Products, and Return Ratio.
3. Inventory Control: Stockout Days and Expired Ratio.
4. Tool-Use Behavior: Direct and Total Tool Calls per Day.
5. Token Use: Average daily token consumption.

Key Contributions

1. A Long-Horizon Retail Environment: RetailBench provides a controlled, data-grounded testbed that couples pricing, inventory, supplier selection, customer feedback, external events, and financial constraints over extended episodes.
2. Evaluation of Current Systems: The benchmark reveals that current agent frameworks and state-of-the-art LLMs struggle to achieve stable, high-performing operational policies over long horizons.
3. Systematic Failure Analysis: The authors identify three recurring failure modes through diagnostic metrics:
   • Incomplete Evidence Acquisition: Agents often act without querying critical information (e.g., supplier quality proxies, inventory aging).
   • Surface-Level Decision Making: Agents rely on immediately visible signals (e.g., price) while underweighting consequential but less visible factors (e.g., supplier quality, return risks).
   • Lack of Consistent Long-Horizon Policy: Agents fail to maintain attention on important products and do not consistently revisit the delayed consequences of prior actions.

Results

• Survival: Under the survival-first protocol, survival ranged from 58 to 180 days. Only DeepSeek-V4-Pro and GPT-5.5 completed the full 180-day horizon. Most other models terminated between 58 and 73 days.
• Performance Gap: Even the strongest LLM runs fell substantially short of the oracle policy.
  • Net Worth: The oracle policy achieved 131,510.42, while the best LLM (GPT-5.5) achieved 24,350.98.
  • Sales: The oracle policy achieved 267,998 units, while the best LLM (DeepSeek-V4-Pro) achieved 164,417.
• Behavioral Diagnostics:
  • Evidence Acquisition: GPT-5.5 showed the highest evidence completeness (0.9689), while others like Grok-4.3 showed near-zero completeness.
  • Decision Quality: LLMs exhibited a strong bias toward "PriceFirst" (choosing the cheapest supplier) over "QualityFirst." The oracle policy prioritized quality, whereas LLMs chose the cheapest supplier in 55.6% of orders compared to only 21.5% for the best quality choice.
  • Temporal Follow-up: LLMs missed 32.86% to 88.18% of high-demand opportunities and failed to consistently follow up on delayed consequences (e.g., stockouts, returns) within a seven-day window.

Significance and Claims

The paper positions RetailBench not merely as a performance benchmark but as a diagnostic testbed for studying reliable autonomy in economically grounded, long-horizon decision-making. The authors claim that the results demonstrate a critical gap between current LLM capabilities and the requirements for sustained operational continuity.

The significance lies in the identification that improving long-horizon LLM agents requires more than just increased tool usage; it necessitates advancements in:

1. Evidence Selection: The ability to identify and acquire all necessary decision-critical information.
2. Evidence-to-Action Reasoning: The capacity to translate acquired evidence into decisions that account for long-term outcomes (e.g., balancing cost vs. quality).
3. Stateful Policy Maintenance: The ability to maintain persistent attention on critical entities and revisit delayed consequences to align local tool calls with long-term business outcomes.

The authors conclude that before LLM agents can reliably operate in complex long-horizon environments, they must overcome these specific bottlenecks in evidence acquisition, reasoning depth, and policy consistency.