From Pixels to Shelf: An Integrated Robotic System for Autonomous Supermarket Stocking with a Mobile Manipulator

Technical Summary: From Pixels to Shelf

Problem Statement
Autonomous stocking in retail environments, particularly supermarkets, faces significant hurdles due to dynamic human interactions, constrained spaces, and the geometric diversity of products. While Autonomous Mobile Robots (AMRs) have succeeded in structured warehouses, their adoption in retail remains limited by high costs, technical complexity, and the difficulty of "last-meter" navigation and manipulation in shared human-robot spaces. Existing solutions often rely on high-end hardware, lack full manipulation capabilities, or focus on submodules (e.g., inventory scanning) rather than complete end-to-end stocking pipelines.

Methodology
The authors propose an integrated, cost-effective robotic system designed for commercial deployment using off-the-shelf hardware and a scalable ROS2-based software architecture.

• Hardware Platform: The system utilizes the Hello Robot Stretch 3, a mobile manipulator with a differential drive base, a telescopic arm, and a 3-DoF wrist. It is powered by an Intel NUC for on-board planning and an external workstation (Intel i9, NVIDIA RTX 4090) for heavy perception tasks.
• Planning Architecture: High-level task management is handled via a Behavior Tree (BT) framework. The BT manages reactive decision-making, including docking, charging, and task execution (picking and placing). Tasks are generated via a web application and transmitted to the robot via MQTT.
• Navigation and Localization:
  • Global Navigation: Uses the Nav2 framework with a Model Predictive Path Integral (MPPI) controller and a Spatio-Temporal Voxel Layer (STVL) to detect obstacles invisible to LiDAR (e.g., elevated objects).
  • "Last-Meter" Navigation: To overcome LiDAR minimum range limitations near shelves, the system employs ArUco markers placed in shelf label holders. A 14-state Kalman Filter (KF) fuses visual marker data for robust pose estimation.
  • Control: A two-step Model Predictive Control (MPC) framework coordinates the base and camera head. The first step optimizes base motion to reach the shelf, while the second optimizes camera head motion to maintain the marker in the field of view and ensure smooth approach trajectories.
• Perception Pipeline:
  • Object Detection: Utilizes fine-tuned YOLO11n-cls models for grocery product identification. For products without specific models, the system attempts to use GPT-4o with reference images, though this is noted as less efficient.
  • Segmentation and Tracking: Employs SAM2 (Segment Anything Model 2) for scene segmentation and video-based object tracking to isolate target products.
  • Manipulation: The picking process involves generating point clouds from RGB-D data, computing bounding boxes based on known product dimensions, and using a velocity-based arm controller. The placing process queries a planogram to determine shelf positions, uses the KF for precise localization, and employs an Inverse Kinematics (IK) controller for placement, with contact detection to ensure gentle release.

Key Contributions

1. System Integration: The paper presents a fully integrated, deployable ROS2 pipeline combining off-the-shelf hardware with advanced perception (YOLO, SAM2), planning (BTs), and control (MPC, KF) specifically tailored for retail stocking.
2. Cost-Effective "Last-Meter" Solution: The authors demonstrate a robust navigation and manipulation strategy using low-cost ArUco markers and a two-step MPC, achieving high reliability without requiring expensive, high-end sensor suites.
3. Comprehensive Validation: The work includes extensive laboratory experiments replicating overnight stocking scenarios, providing a quantitative benchmark against teleoperation and human workers, and decomposing performance gaps into hardware and software limitations.
4. Open Source: Key modules, including the simulator and ROS2 packages, are released as open source to facilitate further research.

Experimental Results
The system was evaluated in a mock supermarket environment over 70 stocking tasks involving 724 individual pick-and-place operations.

• Success Rates: The system achieved an object-level success rate of over 98% and a task-level success rate of over 87%. Most failures occurred during the picking stage due to classification errors or network latency, while placement was highly reliable.
• Perception Performance: Fine-tuned YOLO models significantly outperformed GPT-4o in both accuracy (97.5% vs. 84.7%) and inference time (0.2s vs. 6.9s).
• Performance vs. Humans: The autonomous system was approximately 10 times slower than human workers. However, it was significantly faster and more cost-effective than teleoperation.
• Cost Analysis: While the autonomous system is currently less efficient than human labor, it offers economic advantages through overnight operation without incremental labor costs. The authors estimate that ~80% of the performance gap relative to humans is attributable to hardware constraints (e.g., gripper size, speed), while ~20% is due to software limitations.

Significance and Claims
The paper claims to demonstrate the practical viability of autonomous stocking in retail environments using affordable, commercially available hardware. It highlights that while current autonomous systems are not yet ready to fully replace human workers due to speed and cost-effectiveness gaps, the integration of modular perception and control algorithms has achieved high reliability (98% success) in complex, unstructured tasks. The work serves as a baseline for future improvements, identifying hardware capabilities as the primary bottleneck for achieving human-level efficiency in retail automation. The authors explicitly state that widespread commercial deployment requires further progress in both hardware speed and software optimization.