🤖 AI

RoboBenchMart: Benchmarking Robots in Retail Environment

This paper introduces RoboBenchMart, an open-source simulated benchmark for retail dark-store environments that evaluates state-of-the-art vision-language-action models on complex manipulation tasks, revealing their current limitations in generalizing beyond household settings.

Original authors: Konstantin Soshin, Alexander Krapukhin, Andrei Spiridonov, Gregorii Bukhtuev, Andrey Kuznetsov, Vlad Shakhuro, Denis Shepelev

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Konstantin Soshin, Alexander Krapukhin, Andrei Spiridonov, Gregorii Bukhtuev, Andrey Kuznetsov, Vlad Shakhuro, Denis Shepelev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you've built a super-smart robot chef. You've trained it in a perfect, clean kitchen where it learned to pick up a single apple and put it in a bowl. It's a whiz at that specific task. But now, you want to send this robot to a real grocery store to restock shelves.

The store is a different beast entirely. It's crowded, the aisles are narrow, the shelves are packed tight with thousands of different items, and the lighting is weird. The paper "RoboBenchMart" argues that just because our robot is great at the "kitchen test," it doesn't mean it can handle the "grocery store test."

Here is a simple breakdown of what the researchers did and what they found:

1. The Problem: The "Kitchen" vs. The "Store"

Most robot tests today are like playing a video game in a small, empty room. The robots learn to pick up blocks or move cups on a table. But a real grocery store is like a chaotic maze.

The Challenge: In a store, items are stacked high, packed closely together, and look very similar (like 50 different boxes of cereal). The robot has to navigate narrow aisles without knocking things over.
The Gap: The researchers wanted to know: If a robot is a master chef in a clean kitchen, can it suddenly become a master stocker in a messy supermarket?

2. The Solution: Building a "Digital Supermarket"

To test this without buying a real store and hiring a robot, the team built RoboBenchMart. Think of this as a highly realistic video game simulator specifically for grocery stores.

They created three main tools to build this world:

The Store Architect: A program that automatically designs different store layouts. It's like a game engine that randomly places shelves, fridges, and boxes so the robot never sees the exact same store twice.
The Product Packer: A system that puts thousands of virtual grocery items on the shelves. It even simulates how a shelf looks when it's full, half-empty, or when items are knocked over.
The Robot Teacher: A system that generates "perfect" robot movements (trajectories) to show the robot how to do tasks like picking up a soda can or opening a fridge door.

3. The Test: Can the "Generalist" Robots Pass?

The researchers took four of the most advanced, "generalist" AI robots (models that are supposed to be good at anything) and tried to teach them to work in this digital supermarket. They gave the robots a small amount of training data (about 2,900 examples) and then tested them.

The Results were humbling:

The "Kitchen" Robots Struggled: Even the smartest robots, which had been trained on massive amounts of data, failed miserably in the store.
Simple Tasks Failed: Robots couldn't reliably pick up a single item from a crowded shelf without knocking others over.
Complex Tasks Were Impossible: When asked to do a multi-step job (like "Open the fridge, grab a milk, and close the door"), the robots failed 100% of the time.
New Environments = Total Failure: When the researchers changed the store layout or the items on the shelves (something the robot hadn't seen before), the robots got completely confused and stopped working.

4. Where Did They Go Wrong?

The researchers analyzed why the robots failed. It wasn't just one thing; it was a chain reaction of errors:

Getting Lost: The robot couldn't figure out which specific item the human asked for (e.g., "Get the blue Fanta," not the red one).
The "Grasp" Problem: Even if the robot found the right item, it often missed the grip or dropped it immediately.
The "Door" Problem: Opening a sliding fridge door requires a specific, delicate motion that the robots couldn't master.

The Bottom Line

The paper concludes that while our robots are getting better at simple tasks in simple rooms, they are not yet ready for the messy, complex reality of a grocery store.

Think of it like this: We have taught robots to play chess on a perfect board. But the real world is like playing chess while the board is shaking, the pieces are all different shapes, and the rules change every time you walk into a new room. RoboBenchMart is the new "exam" that proves we still have a long way to go before robots can truly work in our stores.

The good news? The researchers released all their tools, code, and the "digital store" to the public, so other scientists can use this benchmark to help build better robots for the future.

Technical Summary: RoboBenchMart: Benchmarking Robots in Retail Environment

Problem Statement

Current robotic manipulation benchmarks predominantly focus on tabletop or household scenarios (e.g., ALFRED, RLBench, RoboCasa), where tasks often reduce to pick-and-place operations in relatively unconstrained spaces. While these environments have driven significant progress in Vision-Language-Action (VLA) models, it remains unclear whether generalist policies that excel in these settings can generalize to domains with distinct geometry, semantics, and workflows.

The retail sector, specifically "dark stores" (small distribution centers fulfilling online orders), presents a critical gap. These environments feature dense object clutter, multi-level shelving, mixed fixtures (shelves and refrigerators), and a vast variety of visually similar products. Existing benchmarks fail to capture the specific challenges of retail, such as navigating narrow aisles, performing reliable collision avoidance in cluttered spaces, and manipulating items at varying heights and depths. Furthermore, testing policies directly in real-world retail settings is expensive and logistically difficult, necessitating high-fidelity simulation benchmarks that accurately reflect these complexities.

Methodology

1. RoboBenchMart Suite

The authors introduce RoboBenchMart, an open-source simulated benchmark suite built on the ManiSkill3 framework, utilizing the Fetch mobile manipulator (differential-drive base, 7-DOF arm, prismatic torso). The suite comprises three core components:

Store Plan Generator: A procedural pipeline for generating diverse and realistic store layouts.
- Fixture Arrangement: Inspired by procedural street modeling, the system uses a tensor field approach. It computes a basis tensor field based on initial random fixture placements (pallets, freezers) and aggregates them to determine optimal shelving orientations (horizontal or vertical) while ensuring collision-free, navigable aisles.
- Product Arrangement: Leveraging the scene synthesizer package, products are placed on a regular grid with pose perturbations to simulate natural variability. The system supports vertical stacking and uses a Poisson process to simulate shelf depletion (empty spaces) over time.
- Assets: The suite includes 370 product assets across 21 categories, along with shelving and refrigerator models. To ensure rendering performance in scenes with hundreds of objects, the authors developed an automatic mesh simplification pipeline (using QuadriFlow, Marching Cubes, and shape-specific approximations) to reduce triangle counts while minimizing geometry quality loss (measured by Chamfer distance).
Store Trajectories Sampler: A pipeline for generating demonstration trajectories using motion planning.
- The system heuristically defines sequences of anchor poses (initial, intermediate, final) for task stages.
- Trajectories are generated using screw motion for non-mobile segments (with collision checking) and RRT-Connect for obstacle avoidance. Mobile base repositioning uses task-specific heuristic planners.
- Only successful, full-task trajectories are retained. The dataset includes 2,976 demonstrations (248 per task/item/fixture triplet) for the Fetch robot.
Store Robotics Benchmark: A standardized evaluation framework.
- Tasks: Includes atomic tasks (Pick to basket, Pick from floor, From board to board, Open/Close fridge) and composite tasks (Pick N items, Pick from fridge).
- Evaluation Scenarios: The benchmark tests generalization across:
  1. In-Domain: Randomized robot positions only.
  2. Unseen Scenes: Randomized positions, textures, and store layouts.
  3. Unseen Scenes & Items: Unseen scenes combined with Out-of-Distribution (OOD) items drawn from other tasks.
- Baselines: Four state-of-the-art generalist VLA models were evaluated: Octo, SmolVLA, $\pi_0$ , and $\pi_0.5$ . All models were fine-tuned via imitation learning on the generated trajectories.

2. Experimental Setup

Training: Models were fine-tuned exclusively on atomic tasks. Composite tasks were evaluated by decomposing them into sequences of atomic instructions.
Data Constraints: To model realistic adaptation, training was limited to 2–3 objects per task, with strict separation between training and testing object sets for specific tasks.
Metrics: Success rates were measured over 100 trials per task/item/fixture triplet. Failure modes were annotated based on the earliest primary failure in the pipeline (e.g., task grounding, navigation, grasping, placement).

Key Results

Performance on Generalist Models

The evaluation reveals that current state-of-the-art generalist VLAs struggle significantly in the retail domain, even after fine-tuning:

SmolVLA: Performed poorly across all scenarios, achieving non-zero success only on the simplest "open/close fridge" tasks.
Octo & $\pi_0$ : Achieved moderate success in the In-Domain setting but degraded significantly in Unseen Scenes and failed completely in Unseen Scenes & Items.
$\pi_0.5$ : Performed the best among the evaluated models, showing non-zero success in the Unseen Scenes & Items scenario, though reliability remained low.
Composite Tasks: All models achieved 0% success on composite tasks (e.g., picking 3 items), indicating a failure to execute multi-step instructions or generalize across task stages.

Failure Analysis

Failure annotation (Table 3) identified two primary bottlenecks:

Grasping Failures: Models like Octo, $\pi_0$ , and $\pi_0.5$ predominantly failed at the grasping stage, struggling with precise shelf-level interaction.
Grounding Errors: A significant portion of failures occurred during "target" selection (identifying the correct product) and "pregrasp" positioning. SmolVLA failed most often during pregrasp positioning.
Distribution Shift Sensitivity: As the evaluation distribution shifted from training (e.g., from In-Domain to Unseen Scenes & Items), failure rates increased drastically, often exceeding 90% for most models.

Significance and Claims

The paper claims that RoboBenchMart is the first open benchmark dedicated to evaluating robotic policies specifically in retail environments. Its significance lies in:

Exposing a Domain Gap: The results demonstrate that high performance in household/tabletop benchmarks does not guarantee transferability to retail settings. Current generalist models lack the robustness required for the geometry, clutter, and semantic complexity of dark stores.
Identifying Limitations: The study highlights that existing pretrained models are insufficient for effective adaptation in retail without targeted pretraining on retail-specific data. The models exhibit fragility to minor scene changes and poor generalization from limited demonstrations to novel object-task combinations.
Enabling Future Research: By releasing the procedural store layout generator, trajectory generation pipeline, evaluation tools, and baseline models, the authors aim to facilitate the development of more robust, scalable, and task-aware robotic systems for the retail sector.

The authors maintain a modest tone, acknowledging that the benchmark currently supports only parallel-jaw grippers, rigid-body packages, and excludes suction or dexterous hands. They position RoboBenchMart as a foundational step toward addressing the underexplored challenges of retail automation.