🤖 AI

ShopGym: An Integrated Framework for Realistic Simulation and Scalable Benchmarking of E-Commerce Web Agents

This paper introduces ShopGym, an integrated framework that bridges the gap between realistic and controllable e-commerce agent evaluation by converting live storefronts into reproducible sandbox environments (ShopArena) and synthesizing grounded benchmark tasks (ShopGuru), demonstrating that synthetic shops preserve key structural properties and correlate with agent performance on real-world sites.

Original authors: Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chinmay Savadikar, Mingyu Zhao, Yuanzheng Zhu, Han Li, Shuang Xie, Alberto Castelo, Tianfu Wu, Lingyun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to shop online. To do this effectively, you need a training ground. The paper "ShopGym" argues that the current training grounds are broken, and it proposes a new, better solution.

Here is the breakdown of the problem and their solution, using simple analogies.

The Problem: The "Live Store" vs. The "Toy Store" Dilemma

Currently, researchers have two bad options for testing shopping robots:

The "Live Store" (Too Chaotic): Imagine trying to teach a robot to shop in a real, bustling supermarket. The problem is that the store changes every day. The shelves get rearranged, the prices change, the layout shifts, and sometimes the store is running a secret test for some customers but not others.
- The Result: If the robot fails today, you don't know if it's because the robot is bad or because the store changed its layout. You can't repeat the test exactly the same way tomorrow. It's like trying to grade a student's math test while the teacher keeps changing the numbers on the board.
The "Toy Store" (Too Simple): The alternative is a fake store built by hand, like a Lego set. It's very stable; the shelves never move.
- The Problem: It's too simple. It only has a few types of boxes and a tiny selection of toys. Real shopping involves thousands of products, complex filters, and weird website layouts. A robot that masters the Lego store might still get lost in a real supermarket.

The Core Bottleneck: We need a training ground that feels like a real store (so the robot learns real skills) but acts like a toy store (so we can test it fairly and repeatedly).

The Solution: ShopGym

The authors built ShopGym, a framework that acts like a "Digital Twin Factory." It has two main parts: ShopArena and ShopGuru.

1. ShopArena: The "Digital Twin Factory"

Think of this as a machine that takes a photo of a real, messy store and prints out a perfect, clean, but identical-looking replica.

How it works:
- The Explorer: A team of AI agents visits a real website (the "Seed Shop"). They don't buy anything; they just take notes. They map out the navigation, count the products, and note how the filters work. Crucially, they strip away all the real names (like "Nike" or "Coca-Cola") and replace them with fake names (like "Brand X" or "Drink Y").
- The Architect: This team writes a "blueprint" (a specification) based on those notes.
- The Builder: Another team of AI agents reads the blueprint and builds a brand-new, self-contained website from scratch. It looks and behaves exactly like the real store, but it's made of synthetic data.
The Magic: Because it's built from a blueprint, you can reset it to zero instantly. You can run the same test 1,000 times, and the store will look exactly the same every time. But because it was built from a real store's structure, it's just as complex as the real thing.

2. ShopGuru: The "Exam Creator"

Once the fake store is built, you need to give the robot a test. ShopGuru is the teacher that writes the exam questions.

Short-Horizon Tasks (The Pop Quiz): These are simple, single-step questions like, "Find the red shirt" or "Check the shipping policy." These test basic skills.
Long-Horizon Tasks (The Final Project): These are complex stories. For example: "You are a person with three dogs. Find dog food, then find cat litter, then check if you qualify for free shipping, and finally put both in your cart."
The Safety Check: ShopGuru doesn't just guess. It checks its own questions against the fake store's database to make sure the answers actually exist. It won't ask the robot to find a "purple toaster" if the store doesn't sell toasters.

How They Tested It

To prove their idea works, they did two things:

Structural Check: They compared the "Digital Twin" stores to the real stores using a graph (like a map of connections). They found the fake stores had the same number of pages, the same depth of menus, and the same complexity as the real ones.
Behavioral Check: They ran top-tier AI robots on both the real stores and the fake "Twin" stores.
- The Result: The robots performed almost identically on both. If a robot was good at shopping on the real store, it was good on the fake one. If it struggled on the real one, it struggled on the fake one.

The Bottom Line

ShopGym solves the "Realism vs. Control" trade-off.

It gives researchers a stable, resettable, and fair environment (like a video game level).
But because it is built from real data, it preserves the complexity and difficulty of real-world shopping.

This means we can finally train and test shopping robots in a way that is scientific, reproducible, and actually useful for the real world, without the chaos of live websites changing under our feet.

Technical Summary: ShopGym

Problem Statement

The evaluation of e-commerce web agents faces a fundamental methodological bottleneck: a tradeoff between realism and experimental control. Existing approaches fall into two categories, each with significant limitations:

Live Storefronts: While offering high realism with natural layouts and interaction patterns, they are non-stationary. Catalogs, designs, and A/B test variants change over time, making training and evaluation irreproducible and difficult to compare fairly across different runs or agents.
Hand-Built Sandboxes: These provide control and reproducibility but suffer from a lack of diversity. Manually constructed environments cover only a narrow range of layouts, catalog structures, and policies, failing to capture the complexity of real-world e-commerce.

The field lacks a scalable methodology to construct evaluation settings that are simultaneously realistic, diverse, controllable, inspectable, and reproducible.

Methodology: ShopGym Framework

To address this gap, the authors introduce ShopGym, an integrated framework comprising two synergistic components: ShopArena (simulation environment generation) and ShopGuru (grounded task generation).

1. ShopArena: Scalable Simulation Environment Generation

ShopArena transforms live seed storefronts into self-contained, anonymized sandbox shops through a two-phase pipeline:

Exploration Phase (Live to Specification):
- The system autonomously browses one or more live seed storefronts.
- It extracts structural and behavioral signals (navigation, affordances, page organization, catalog statistics) while strictly anonymizing brand names, product details, and URLs.
- These signals are synthesized into a human-readable, intermediate Design Specification ( $M$ ). This specification acts as the sole interface between exploration and generation, allowing for multi-seed composition (merging structural diversity from multiple sources) and manual editing without re-exploring the source.
Generation Phase (Specification to Sandbox):
- A staged code synthesis pipeline converts the specification $M$ into a runnable sandbox shop.
- Catalog Synthesis: Generates synthetic product data (names, descriptions, images) based on the design manual and statistics from $M$ .
- Source-Code Synthesis: Uses an Execution–Verification Loop driven by multiple agents. An Execution Agent edits the codebase, while a separate Verification Agent (combining rule-based checks and multimodal visual inspection) provides feedback. This loop iterates until the shop is functional, ensuring stability and preventing context-window degradation common in long-horizon code generation.
- The result is a synthetic shop that preserves the structural, behavioral, and statistical properties of the seed storefronts but uses synthetic brand identity and imagery.

2. ShopGuru: Grounded Task Generation

ShopGuru synthesizes benchmark tasks directly grounded in the generated sandbox shops' data (catalog, navigation graph, policies).

Short-Horizon Tasks: Generated via deterministic, rule-based scripts to evaluate primitive skills:
- Product Discovery: Exact and substitute search.
- Filter-Selection: Browsing collections and applying specific filters (ensuring feasibility by checking product availability).
- Information Seeking: Locating policy pages (shipping, returns).
Long-Horizon Tasks: Generated by an LLM (e.g., GPT-5, Gemini 3 Pro) to simulate complex human shopping journeys (e.g., filter $\to$ $\to$ sort $\to$ $\to$ inspect $\to$ $\to$ policy detour $\to$ $\to$ add to cart).
- Polish Loop: To mitigate LLM hallucinations (e.g., requesting non-existent variants), a validator-driven loop iteratively refines the generated tasks. The validator checks for grounding mismatches (unknown products, infeasible filters) and feeds actionable errors back to the LLM for regeneration.

Key Contributions

Methodological Identification: The paper identifies the realism-control tradeoff as a core bottleneck and argues for scalable construction of realistic, controllable environments as the missing capability.
ShopArena Pipeline: A novel pipeline that converts live storefronts into anonymized, self-contained sandbox shops via specification synthesis and staged code generation, enabling diverse and reproducible environments.
ShopGuru Task Generator: A grounded task-generation system producing both short-horizon skill evaluations and long-horizon E2E shopping journeys, validated by a rule-based and LLM-driven polish loop.
Validation: Empirical evidence showing that synthetic shops preserve meaningful evaluation signals from live storefronts.

Results and Validation

The authors validated ShopGym using 224 generated tasks across six sandbox shops (three fully synthetic, three "twin" shops using real product data but synthetic branding).

Structural Validity: Comparisons between synthetic and real shops showed comparable observation and interaction complexity (accessibility tree depth, unique interactive elements). Synthetic shops exhibited slightly fewer state transitions due to the intentional exclusion of external links and marketing noise, but maintained similar coverage of distinct UI states.
Behavioral Validation: Agent performance (using models like GPT-5 and Gemini 3) on synthetic "twin" shops was positively correlated with performance on the corresponding live storefronts. This indicates that the synthetic environments retain the difficulty and behavioral signals necessary for evaluation.
Task Complexity: Frontier models achieved low success rates on the generated long-horizon tasks (e.g., ~48% for GPT-5-mini, ~62% for GPT-5 on BrowserGym harness), demonstrating that the tasks are sufficiently complex to challenge current agents.

Significance and Claims

The paper claims that ShopGym provides a practical middle ground between live-web realism and sandbox control. By producing evaluation artifacts that are self-contained, inspectable, resettable, and stable, ShopGym offers a scalable foundation for more reproducible and interpretable evaluation of e-commerce web agents.

The authors modestly note that ShopGym does not completely replace live-web evaluation, as it removes sources of real-world non-stationarity (e.g., inventory churn, personalization, latency). However, it serves as a stronger methodology for standardized benchmarking, allowing researchers to isolate agent capabilities from environmental noise. The framework is designed to support responsible research use, with safeguards against unauthorized automation and data redistribution.