MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments

MarketGen is a scalable simulation platform featuring an agent-based procedural content generation framework that automatically creates realistic, multi-modal supermarket environments and establishes a new benchmark for evaluating embodied agents on complex long-horizon tasks like checkout unloading and in-aisle item collection.

The Gist

Imagine trying to teach a robot to work in a supermarket. It sounds simple, right? But in the world of robotics, it's like trying to teach a toddler to navigate a crowded, chaotic city without ever having left their bedroom.

Most existing robot training programs are like playgrounds: they have small tables, a few toys, and very simple rules. They are great for learning how to pick up a block or pour a cup of water. But a real supermarket is a massive, complex city with thousands of items, narrow aisles, and specific rules about where things go.

The paper introduces MarketGen, a new "virtual city builder" designed specifically to train robots for this big, messy, real-world job.

Here is how it works, broken down into simple concepts:

1. The "Infinite Lego" Factory (The Asset Library)

Imagine you have a box of Lego bricks. In the past, you might only have 50 bricks to build a house. MarketGen gives you a warehouse with 1,100+ different types of bricks.

• The Goods: It has digital models of over 1,000 real-world items: milk cartons, bananas, soda cans, and snack bags.
• The Shelves: Instead of just having one fixed shelf, MarketGen has "smart" shelves. You can tell the computer, "I want a shelf that is 6 feet tall with 4 shelves," and it instantly builds a custom version of that shelf. It's like having a 3D printer that can make any furniture layout you can imagine.

2. The "Architect & The Builder" Team (Auto-Generation)

In the old days, scientists had to manually build every single virtual supermarket, which took weeks of hard work. MarketGen automates this with a two-person team:

• The Architect (The AI Agent): You give the Architect a description like, "Build a modern, bright fruit store with wooden shelves," or even show it a picture of a store you like. The Architect uses its brain (a Large Language Model) to figure out the layout: Where does the dairy go? Where is the checkout? How wide are the aisles? It ensures the store makes logical sense.
• The Builder (The PCG System): Once the Architect draws the blueprints, the Builder (a procedural system) instantly constructs the 3D world, filling the shelves with the correct products and placing the checkout counters.

The Analogy: Think of it like ordering a house on a video game. You don't place every brick by hand; you just say, "I want a Victorian style with a big garden," and the game generates the whole house instantly. MarketGen does this for supermarkets.

3. The "Driving Test" (The Benchmark)

To see if the robots are ready for the real world, MarketGen gives them two tough tests, just like a driving license exam:

• Test A: The Cashier (Checkout Unloading): The robot sits at a counter. A customer hands them a basket full of messy, overlapping items. The robot has to pick them up one by one without knocking anything over. This tests their "hand-eye coordination" and patience.
• Test B: The Shopper (In-Aisle Collection): The robot has a mobile base (wheels). It needs to drive through the store, find a specific item (like "the red soda can") hidden among hundreds of similar cans, and grab it. This tests their ability to navigate a maze and find things in a crowd.

4. The "Virtual Reality" Bridge (Sim-to-Real)

The biggest fear in robotics is the "Uncanny Valley" of physics: A robot learns in a video game, but when it goes to the real world, it fails because the real world feels different (friction, lighting, weight).

The researchers tested MarketGen by having a robot learn to grab objects in the simulation and then immediately trying to grab the exact same objects in the real world.

• The Result: The robot performed almost exactly the same in both worlds.
• The Metaphor: It's like practicing piano on a digital keyboard that feels exactly like a real Steinway. When you switch to the real instrument, your fingers already know exactly what to do. MarketGen bridges that gap.

Why Does This Matter?

Currently, robots are great at playing with toys on a table but terrible at working in a busy store. MarketGen is the training ground that finally lets robots practice in a realistic, endless variety of supermarket environments.

It's not just about making robots faster; it's about giving them the experience they need to one day work alongside us, stocking shelves, scanning items, and helping us shop, without needing a human to hold their hand every step of the way.

Technical Summary

Here is a detailed technical summary of the paper "MarketGen: A Scalable Simulation Platform with Auto-Generated Embodied Supermarket Environments."

1. Problem Statement

The development of embodied AI agents for complex commercial environments (e.g., supermarkets, hospitals, factories) is currently hindered by a significant gap in existing robotics datasets and benchmarks.

• Limitations of Current Platforms: Most existing simulators (e.g., ProcTHOR, RoboCasa) focus on household or tabletop settings with short-horizon tasks.
• Scalability Issues: Creating commercial-scale environments manually is time-consuming and labor-intensive.
• Generation Flaws: Existing automated generation methods (LLM-based or PCG-based) often fail in commercial contexts due to:
  • Inability to ensure controllable scene consistency.
  • Lack of robust 3D spatial understanding (hallucinations in LLM-only approaches).
  • Restricted input modalities that cannot capture the complexity of commercial layouts.
• Resource Scarcity: There is a lack of diverse, high-fidelity 3D assets specifically for supermarket goods and facilities.

2. Methodology

MarketGen addresses these challenges through a scalable simulation platform built on NVIDIA's Omniverse Isaac Sim and Unreal Engine, featuring two core components: an Automatic Scene Generation System and a Comprehensive 3D Asset Library.

A. Automatic Scene Generation Pipeline

The system employs a hybrid Agent-based Procedural Content Generation (PCG) framework to generate structured, realistic supermarket layouts from multi-modal inputs (text descriptions and reference images).

1. Agent System (High-Level Planning):

   • Spatial Agent: Parses input to define functional zones (e.g., produce, dairy, checkout) using Binary Space Partitioning (BSP) algorithms to ensure architectural constraints are met.
   • Semantic Agent: Specifies asset properties for each zone, including style, parametric attributes (layer count, spacing), and visual characteristics.
   • Reflection & Refiner: An iterative module where the LLM reviews the generated 2D layout (via visual prompting) to correct spatial errors and refine the plan before 3D construction.

2. PCG Workflow (3D Construction):

   • Asset Retrieval: Matches semantic zone definitions against a database to select appropriate assets.
   • Asset Adjustment: Dynamically configures parameterized facilities (e.g., shelving units) by adjusting vertical tiers, spacing, and modular combinations.
   • Asset Placement: Executes framework creation, shelf placement, and an automated Product Filling procedure to populate shelves with contextually appropriate goods.

B. 3D Asset Library

To support diverse scene generation, the authors constructed a massive library:

• Goods: Over 1,100 high-fidelity 3D models of supermarket products across 10 categories (beverages, produce, dairy, etc.), annotated with physical properties (mass, friction) and semantic data.
• Facilities: Over 100 distinct facility assets (shelves, refrigerators, counters). Crucially, key assets like shelving are parametrically decomposed into minimal units (boards, panels, supports), allowing for exponential variation in layout configurations.
• Materials: A diverse library of Physically Based Rendering (PBR) materials to support sim-to-real transfer.

C. Benchmark for Supermarket Agents

The authors propose a new benchmark featuring two distinct daily tasks:

1. Checkout Unloading (Static/Tabletop): A long-horizon task where a stationary agent (cashier) unloads items from a shopping basket. This tests manipulation in cluttered, occluded environments.
2. In-Aisle Item Collection (Mobile): A complex mobile manipulation task where an agent navigates aisles, locates specific items among similar products, and grasps them. This tests navigation, long-range perception, and multi-object grasping.

The benchmark includes a Modular Manipulation System utilizing:

• Affordance Generator: Uses SAM (Segment Anything Model) and Set-of-Mark (SoM) prompting for target selection, and AnyGrasp for pose generation.
• Action Executor: Uses cuRobo for collision-free trajectory planning.
• Long-Horizon Planner: Uses LLMs/VLMs for task decomposition and failure recovery.

3. Key Contributions

1. Scalable Simulation Platform: The first automated system for generating diverse, structured supermarket environments using a hybrid Agent-PCG approach that supports multi-modal inputs.
2. Extensive Asset Library: A repository of 1,100+ supermarket goods and 100+ parameterized facilities, enabling high-fidelity and varied scene generation.
3. Comprehensive Benchmark: A novel evaluation suite for supermarket agents covering both static (checkout) and mobile (aisle collection) manipulation tasks, including a framework for zero-shot modular policy evaluation.

4. Experimental Results

• Scene Generation Quality: Qualitative results (Fig. 5) show that MarketGen generates scenes with fidelity and logical coherence comparable to handcrafted environments (e.g., GRUtopia). The hybrid pipeline ensures realistic spatial arrangements and visual diversity.
• Benchmark Performance:
  • Experiments with various Multi-modal Large Language Models (MLLMs) on the benchmark revealed significant challenges.
  • Success Rates: Even the best-performing model (Gemini-2.5-Pro) achieved an overall Success Rate (SR) of only 13.87%.
  • Analysis: The low performance highlights the difficulty of long-horizon tasks in high-clutter commercial settings. Current modular methods struggle with occlusion, precise object localization, and collision avoidance in narrow aisles.
• Sim-to-Real Transfer:
  • A real-world experiment compared grasp success rates between simulation and reality for four object types (bottles, cans, boxes, bags).
  • Correlation: The results showed a strong correlation (e.g., Bottles: 55% Sim vs. 50% Real; Cans: 45% vs. 45%). This validates that MarketGen's physics, asset properties, and rendering effectively bridge the sim-to-real gap.

5. Significance

• Bridging the Commercial Gap: MarketGen fills a critical void by providing a dedicated platform for commercial robotics, moving beyond the limitations of household-centric simulators.
• Accelerating Research: By automating scene generation, it removes the bottleneck of manual environment design, allowing researchers to scale experiments and test agents in diverse, realistic commercial scenarios.
• Benchmarking Standard: It establishes a rigorous standard for evaluating embodied agents in complex, long-horizon commercial tasks, revealing that current state-of-the-art modular approaches are insufficient for these environments and highlighting the need for end-to-end learning advancements.
• Sim-to-Real Viability: The successful transfer of affordance prediction models to the real world demonstrates the platform's utility for training and validating robots for actual deployment in supermarkets.