🤖 AI

SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

SimGym is a framework that leverages traffic-grounded vision-language model agents operating in a live browser to rapidly and accurately simulate e-commerce A/B tests, achieving 77% directional alignment with real-world outcomes while reducing experimental cycles from weeks to under an hour without exposing actual users to unproven variants.

Original authors: Han Li, Vibhor Malik, Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie, Ailin Fan, Keat Yang Koay, Yuanzheng Zhu, Meysam Feghhi, Ronie Uliana, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Franci

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Han Li, Vibhor Malik, Zahra Zanjani Foumani, Alberto Castelo, Shuang Xie, Ailin Fan, Keat Yang Koay, Yuanzheng Zhu, Meysam Feghhi, Ronie Uliana, Zhaoyu Zhang, Angelo Ocana Martins, Mingyu Zhao, Francis Pelland, Jonathan Faerman, Nikolas LeBlanc, Aaron Glazer, Andrew McNamara, Zhong Wu, Lingyun Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a store owner who wants to repaint your shop. You're torn between two colors: "Ocean Blue" and "Sunset Orange." In the real world, to see which color sells more, you'd have to paint half your store blue and half orange, wait weeks to see which side gets more customers, and hope you don't accidentally scare away your best shoppers while you wait.

SimGym is like a high-tech, crystal-ball simulation lab that lets you test these changes before you paint a single wall. Instead of using real customers, it uses a team of AI "digital shoppers" to walk through your store, look at the new designs, and tell you which one they prefer.

Here is how the paper explains this system, broken down into simple parts:

1. The "Digital Shoppers" (Traffic-Grounded Personas)

Most AI simulations just use generic robots that all act the same. SimGym is different. It builds its shoppers by studying the actual history of your store's real visitors.

The Analogy: Imagine you have a library of thousands of receipts and shopping lists from real people who visited your store. SimGym reads these and creates 600 unique "digital twins."
Some twins are "bargain hunters" who only buy on sale. Others are "luxury seekers" who want the best quality. Some are "quick browsers" who just want to grab one thing and leave.
By creating a crowd that looks exactly like your real customer base, the simulation ensures the results aren't just random guesses; they reflect how your specific people actually behave.

2. The "Live Browser" Agent (The Eyes and Hands)

These digital shoppers don't just read text on a screen; they actually "see" the store.

The Analogy: Think of a human shopper. They don't just read a list of items; they look at the bright red "Sale" sign, notice the layout of the shelves, and see how the pictures look.
SimGym's agents use Vision-Language Models (VLMs). This means they have "eyes" (they take screenshots of the webpage) and "brains" (they read the code behind the page).
They can navigate a live website just like a human, clicking buttons, scrolling, and adding items to a cart. If the design changes (like making a "Buy Now" button bigger or changing the background color), the agent notices it visually, not just as a code change.

3. The "Memory" and "Safety Rules"

To make the simulation realistic, the agents need to remember what they've done and know when to stop.

Memory: Just like a real shopper remembers, "I already looked at those shoes in the back," the agent keeps a log of its journey. This stops it from walking in circles or getting confused.
Guardrails: The system has safety rules to prevent the AI from getting stuck in an infinite loop or crashing the browser. If an agent gets confused, it can recover or stop gracefully, just like a human might give up if a website is too broken.

4. The "Magic Mirror" (Evaluation)

How do we know the AI isn't just hallucinating? The researchers tested SimGym against 50 real stores that had already done real A/B tests (where they actually split traffic between two designs).

The Result: The AI's predictions were surprisingly accurate.
- Directional Alignment: In 77% of cases, the AI correctly predicted which design would win (e.g., "Blue will sell more than Orange").
- Correlation: There was a strong statistical link between what the AI predicted and what actually happened with real humans.
Speed: While a real test takes weeks, SimGym ran these simulations in under an hour per store.

Why This Matters (According to the Paper)

The paper argues that this system solves a major problem: Risk.
Currently, store owners are afraid to try bold new designs because if they fail, they waste weeks of sales and annoy real customers. SimGym acts as a safe sandbox. You can test wild, risky ideas with your "digital shoppers" first. If the simulation says, "This new design will hurt sales," you don't have to risk your real customers. If it says, "This will boost sales," you can move forward with confidence.

In short: SimGym is a framework that turns your store's past data into a team of realistic AI shoppers, lets them test new website designs in a live browser, and tells you the results in an hour instead of weeks, all without bothering a single real human.

Technical Summary: SimGym – A Framework for A/B Test Simulation in E-Commerce

1. Problem Statement

A/B testing remains the gold standard for evaluating e-commerce storefront modifications, such as UI and theme changes. However, traditional testing incurs significant costs: it diverts real traffic to potentially suboptimal variants, requires weeks to achieve statistical significance, and risks degrading the user experience before unsuccessful treatments are detected. These limitations often force merchants to adopt risk-averse strategies, favoring incremental changes over bold redesigns, which constrains innovation.

While recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated the ability of agents to navigate complex interfaces and exhibit realistic user behaviors, existing work lacks an end-to-end framework for synthetic A/B testing in real-world e-commerce settings. Prior research has either addressed persona generation, browser interaction, or evaluation in isolation, or relied on static datasets and offline benchmarks that fail to capture the visual and dynamic nature of live storefronts. Crucially, no existing approach has rigorously validated whether simulated agents can predict actual shifts in conversion metrics (e.g., add-to-cart rates) resulting from real interface interventions.

2. Methodology: The SimGym Framework

SimGym is a modular framework designed to simulate A/B tests using VLM-powered browser agents operating in a live browser environment. The framework consists of three core components:

2.1 Traffic-Grounded Persona Generation Pipeline

To ensure synthetic buyers reflect the heterogeneity of real users, SimGym derives buyer archetypes and intents directly from merchant-specific production clickstream data, rather than relying on hand-crafted or generic personas. The pipeline operates in six stages:

Session Clustering: Sessions are represented as feature vectors (engagement, exploration depth, funnel progression, economic value) and clustered using k-means++ to identify recurring purchase patterns.
Product Preference Extraction: Using a VLM (GPT-5), the system extracts structured product preferences (categories and specific items) for each cluster.
Buyer Intent Generation: A structured shopping intent is created for each agent, comprising a product target (sampled from cluster preferences) and a purchase-decision guide.
Behavior Aggregation: Cross-session summaries (funnel outcomes, cart values, interaction histories) are computed for each cluster.
Buyer Archetype Construction: Multi-dimensional profiles are constructed along five continuous dimensions: Price Sensitivity, Exploration Depth, Premium Focus, Performance Focus, and Ethics Focus. These are derived from aggregated statistics and keyword analysis.
Prompt Composition: The final agent persona pairs a shopping intent with a buyer archetype, creating a profile that reflects observed product interests and behavioral traits.

2.2 Multimodal Live-Browser Agent Architecture

Unlike offline simulations, SimGym agents interact directly with live storefronts. The architecture employs an observe-plan-act loop:

Multimodal Web Perception: At each decision step, the VLM receives two modalities:
- Textual Structure: A DOM-derived accessibility tree providing semantic hierarchy and unique reference IDs for action targeting.
- Visual Perception: A page screenshot capturing layout, imagery, typography, and visual emphasis.
Action Planning: The agent acts as a high-level planner, generating a schema-constrained response containing reasoning, a termination decision, and a proposed action (e.g., click, scroll) based on the current state, session memory, and persona constraints.
Memory Management: An episodic session memory accumulates the full browsing trajectory (navigation context, observations, reasoning, actions, outcomes). This allows the agent to maintain temporal coherence, avoid redundant behaviors, and track visited pages.
Guardrails: To ensure reliability, the system enforces loop protection, step/time budgets, model retry logic, and error propagation to handle unexpected page states or execution failures gracefully.

2.3 Evaluation Protocol

The protocol validates whether simulated agents predict the direction and magnitude of behavioral shifts observed in real human traffic.

Ground Truth Construction: A dataset of 50 paired control/treatment storefront variants is curated. Pairs are selected based on sufficient traffic and substantive visual/layout differences, while filtering out confounding factors like overlapping promotions or pricing changes.
Metrics:
- Directional Validity: Measured by Alignment Rate (percentage of shops where the sign of the simulated $\Delta$ A2C matches the observed human $\Delta$ A2C).
- Magnitude Validity: Measured by the Pearson Correlation Coefficient between simulated and observed A2C shifts across shops.

3. Key Contributions

End-to-End Framework: SimGym is the first unified framework integrating traffic-grounded persona generation, multimodal live-browser interaction, and rigorous predictive validation for synthetic A/B testing.
Traffic-Grounded Synthesis: The pipeline transforms raw clickstream data into diverse, shop-specific buyer archetypes and intents, ensuring the simulated population mirrors empirical customer distributions without manual tuning.
Multimodal Live Interaction: The agent architecture combines visual and structural web perception with episodic memory, enabling coherent shopping sessions across heterogeneous, dynamically changing interfaces.
Empirical Validation: The framework is validated against real-world A/B tests of visually driven theme changes, demonstrating strong alignment between simulated and observed outcomes.

4. Experimental Results

The framework was evaluated on 50 real-world storefronts across 16 countries and 11 product categories, using 600 agents per shop.

Overall Performance: Using Gemini 3 Flash with visual integration, SimGym achieved a 77% directional alignment with observed human add-to-cart (A2C) shifts and a Pearson correlation of 0.55.
Multimodal Necessity: Removing visual input (text-only Gemini 3 Flash) reduced alignment to 70% and correlation to 0.49, confirming that visual features (layout, imagery) provide critical behavioral signals not captured by DOM structure alone.
Persona Importance: Ablation studies showed that the Full Persona (archetype + intent) is essential. Removing the buyer archetype (Shopping Intent Only) dropped alignment to 44% and correlation to near zero. Using only a product target (Product Only) resulted in chance-level alignment (51%).
Memory Necessity: Removing episodic session memory caused directional alignment to drop to 42% and correlation to 0.00, as agents became stuck in navigation loops and failed to track browsing context.
Efficiency: SimGym reduces experimental cycles from weeks to under an hour (approx. 5.3 minutes per shop with 600 agents using Gemini 3 Flash).
Model Robustness: While proprietary models performed best, open-source models (GPT-OSS) still achieved 59% alignment, suggesting the framework's robustness across different model capabilities.

5. Significance and Claims

The paper claims that SimGym provides a rigorous basis for conducting synthetic A/B testing on real e-commerce use cases. By achieving strong directional alignment (77%) and significant correlation (0.55) with real human behavioral shifts, the framework demonstrates that synthetic agents can effectively predict the impact of interface changes.

The primary significance lies in enabling rapid experimentation without exposing real buyers to candidate variants. This capability allows merchants to test bold redesigns and visually driven theme changes with lower risk, potentially unlocking innovation that is currently stifled by the costs and risks of traditional A/B testing. The authors position SimGym as a potential "inner loop" for automated UI exploration, where candidate changes can be proposed, screened, and prioritized in simulation before deployment.

The authors remain modest regarding generalization, noting that the study is currently limited to a single major e-commerce platform and that the impact of post-training alignment on human traces remains an open area for future validation.