🤖 AI

Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems

This paper proposes OLSF-TRS, a generalized and scalable framework that integrates structured combinatorial optimization with multi-agent reinforcement learning to optimize sequential order-tote-robot decisions in tote-handling robotic systems, achieving near-optimal performance on small scales and significantly outperforming state-of-the-art heuristics in large-scale scenarios by reducing tote movements and enhancing operational efficiency.

Original authors: Jiaxin Liu, Peng Yang, Yuping Li, Xinyue Xie

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiaxin Liu, Peng Yang, Yuping Li, Xinyue Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-tech warehouse as a giant, chaotic kitchen during the dinner rush. In this kitchen:

Orders are the dinner tickets coming in from customers.
Totes are the plastic bins holding the ingredients (like tomatoes or cheese).
Robots are the busy chefs running around the kitchen to grab those bins and bring them to the cooking stations.

For a long time, managing this kitchen was like trying to conduct an orchestra where every musician only knew their own sheet music. If the "Order Chef" shouted for a tomato, the "Bin Runner" might grab the wrong bin, and the "Robot Chef" might take a path that caused a traffic jam. Existing computer programs tried to fix this, but they were like custom-made tools: a wrench that worked perfectly for one specific robot but broke on another. They couldn't adapt if the kitchen layout changed or if the robots were different.

The Paper's Big Idea: The "Universal Kitchen Conductor"

The authors of this paper propose a new system called OLSF-TRS. Think of this as a "Universal Kitchen Conductor" that can work in any type of automated kitchen, whether it's a flat kitchen with wheeled carts (2D robots) or a multi-story kitchen with climbing robots (3D robots).

Instead of giving specific instructions for every single situation, this system learns the logic of the kitchen. It uses two main tricks to make decisions:

The "Grouping" Trick (Bisimulation Quotienting):
Imagine you have 50,000 bins in the kitchen. Checking every single one is slow. This system realizes that a bin of "Tomatoes" in the back corner is functionally the same as a bin of "Tomatoes" in the front if they both take the same amount of time to reach the chef. It groups these identical situations together, shrinking a massive, confusing map into a tiny, easy-to-read one. This lets the computer think much faster.
The "Team Huddle" Trick (Multi-Agent Learning):
The system doesn't just tell one robot what to do. It creates three types of "agents" (Order, Bin, and Robot) that act like teammates.
- The Order Agent decides which orders should be cooked together (batching).
- The Bin Agent picks the best bins to grab.
- The Robot Agent plans the fastest route.
Crucially, they don't work in isolation. They use a "Central Coach" (a neural network) that watches the whole kitchen. If the Robot Agent is about to get stuck in traffic, the Coach tells the Order Agent to wait a second before sending a new order. This prevents the "traffic jams" that happen when everyone acts selfishly.

How It Was Tested

The researchers tested this "Universal Conductor" in two ways:

The Small Kitchen Test: They used small, simple kitchens where they could calculate the perfect solution by hand (using a super-precise math solver). The new system got within 3.5% of that perfect score, and it made decisions in the blink of an eye (milliseconds).
The Massive Kitchen Test: They then threw it into huge, chaotic kitchens with hundreds of orders and dozens of robots. Here, the old methods (like simple rules or other AI) started to fail. They got confused by the chaos and wasted time moving bins back and forth.
- The new system reduced the number of times bins had to be moved by over 30% compared to the best existing rule-based methods.
- In the most extreme tests, it reduced bin movements by 812% (meaning the old methods were moving bins over 8 times more than necessary!).

Why It Matters

The paper claims this system is a game-changer because it is flexible and efficient.

Flexible: You don't need to rebuild the software if you buy new robots or change the warehouse layout. The "Universal Conductor" adapts to the new setup automatically.
Efficient: By reducing the number of times robots have to run back and forth, the warehouse saves energy, the robots wear out less, and more orders get shipped faster.

In short, the paper presents a smart, adaptable "brain" for automated warehouses that stops robots from tripping over each other and ensures they are always doing the most helpful thing, no matter how big or complex the warehouse gets.

Technical Summary: Omni-scale Learning-based Sequential Decision Framework for Order Fulfillment of Tote-handling Robotic Systems (OLSF-TRS)

1. Problem Statement

The rapid expansion of e-commerce and flexible, small-batch manufacturing has driven a shift from pallet-based to tote-based intralogistics. Tote-handling robotic systems (e.g., Hairobotics, Exotec, AirRob) have become dominant in automated fulfillment centers. However, order fulfillment in these systems involves a complex, sequential decision-making process involving three interdependent layers: order assignment (allocating orders to workstations), tote matching (selecting and sequencing totes), and robot scheduling (assigning tasks and routing robots).

Existing literature faces two primary limitations:

System Specificity: Most decision mechanisms are tailored to specific robotic architectures (e.g., 2D multi-tote vs. 3D rack-climbing), making them difficult to generalize or transfer to different system types without substantial re-engineering.
Scale and Coupling Limitations: Conventional rule-based heuristics fail to capture dynamic cross-layer dependencies under varying conditions. Exact optimization methods are computationally intractable for large-scale, stochastic instances. While recent learning-based approaches exist, they often address isolated subproblems rather than the full sequential decision pipeline, failing to coordinate the "order-tote-robot" loop effectively.

The core challenge is a multi-stage joint combinatorial optimization problem where each stage is NP-hard, and the stages are strongly coupled. Decisions at the order level influence tote demand and congestion, which in turn affect robot routing, creating a feedback loop that fragmented approaches cannot resolve.

2. Methodology: OLSF-TRS

The authors propose OLSF-TRS, a generalized, scalable sequential decision framework that integrates structured neural combinatorial optimization with multi-agent reinforcement learning (MARL). The framework abstracts heterogeneous robotic systems into a unified, data-driven paradigm.

2.1. State Space Abstraction via Bisimulation Quotienting (BQ-MDP)

To address the exponential state space growth in large warehouses, the framework employs bisimulation quotienting.

Mechanism: Original states are grouped into abstract states if they are bisimulation equivalent—meaning they yield identical transition distributions and expected rewards for all actions.
Application: For example, two robots at mirrored positions relative to a workstation or two totes containing the same SKU are treated as equivalent. This collapses vast storage configurations (e.g., 50,000+ locations) into compact, behaviorally equivalent classes, preserving the optimal policy structure while ensuring computational tractability.

2.2. Neural Combinatorial Optimization (BQ-NCO)

The framework utilizes BQ-NCO to solve the underlying combinatorial subproblems (order batching, tote sequencing, task allocation).

Architecture: A shared Transformer encoder processes unified feature-action representations for orders, totes, and robots.
Training: Policies are pretrained via imitation learning using expert trajectories generated by exact solvers (e.g., Gurobi) on small-scale instances.
Soft Coupling: To prevent error propagation in the sequential pipeline, a soft coupling mechanism aligns the predicted action vectors of one stage with the ground-truth inputs of the subsequent stage, stabilizing end-to-end performance.

2.3. Multi-Agent Coordination via MAPPO

For large-scale, high-concurrency scenarios, the framework employs Multi-Agent Proximal Policy Optimization (MAPPO) with a centralized-training, decentralized-execution (CTDE) paradigm.

Agents: Three cooperative agents (Order, Tote, Robot) operate with distinct roles but share a global objective.
Centralized Critic: A recurrent neural network (RNN) critic evaluates joint states and actions based on a unified operational objective: minimizing the total number of tote movements ( $Z_{Final}$ ), which includes retrievals and returns.
Reward Shaping: Agents receive a global reward derived from the reduction in $Z_{Final}$ , supplemented by shaped local rewards that incorporate intrinsic objectives from BQ-NCO pretraining and inter-agent coordination terms (quantifying how one agent's action impacts another).

3. Key Contributions

Generalized Sequential Decision Framework: OLSF-TRS reframes order fulfillment as a structured sequence of interdependent decisions across order, tote, and robot levels. By abstracting heterogeneous architectures into a unified representation, it supports diverse tote-handling platforms without platform-specific redesign.
Dependency-Aware Modular Paradigm: Unlike isolated layer optimization or monolithic end-to-end models, OLSF-TRS explicitly preserves cross-stage dependencies by linking subproblems into a coherent pipeline. This allows for modular training while maintaining system-level consistency.
Scalable Algorithmic Realization: The integration of BQ-MDP for state abstraction, BQ-NCO for structured combinatorial decisions, and MAPPO for cooperative control enables the framework to operate effectively across the full spectrum of deployment sizes—from small instances solvable by exact methods to large-scale stochastic scenarios with hundreds of orders and dozens of robots.

4. Experimental Results

The framework was evaluated on 18 structured instance groups covering small-scale (S-1 to S-9) and large-scale (L-1 to L-15) scenarios, tested on two distinct system types: 2D Multi-Tote Handling (Hairobotics) and 3D Rack-Climbing (Exotec).

4.1. Small-Scale Performance

Optimality: OLSF-TRS achieved near-optimal performance with average optimality gaps below 3.5% compared to Gurobi exact solutions.
Efficiency: Inference times were in the millisecond range (38–59 ms), confirming computational efficiency suitable for real-time control.

4.2. Large-Scale Performance

In high-concurrency, resource-constrained scenarios (up to 600 SKUs, 350 orders, 70 robots):

Tote Movement Reduction: OLSF-TRS consistently outperformed heuristic baselines. Compared to state-of-the-art rule-based approaches (R3, G3), it reduced total tote movements by up to 43.4% (Exotec) and 42.1% (Hairobotics). Compared to the Collaborative SKU-Group Heuristic (C-SGH), reductions reached 14.9%.
Scalability: While heuristic baselines exhibited superlinear growth in tote movements as system load increased, OLSF-TRS maintained a near-linear growth profile.
Robustness: Under resource scarcity (low robot-to-order ratios), decentralized methods (IPPO) and static heuristics suffered from credit assignment issues and congestion, whereas OLSF-TRS maintained stable performance due to the centralized critic's ability to resolve global coordination conflicts.
Runtime: While simulation runtime was higher than lightweight heuristics, this was dominated by the high-fidelity environment simulation. Policy inference remained below 15 ms per decision step, ensuring real-time responsiveness in deployment.

5. Significance and Claims

The paper claims that OLSF-TRS provides a unified, transferable decision-making framework that bridges the gap between combinatorial optimization models and system-level execution in dynamic environments.

Operational Impact: The reduction in tote movements translates directly to tangible industrial benefits: lower energy consumption, reduced mechanical wear, and enhanced throughput stability.
Methodological Advance: The study demonstrates that high-quality system-level performance does not require solving the full joint optimization problem explicitly. Instead, decomposing the problem into structurally meaningful subproblems (via BQ-NCO) and resolving global coupling via centralized coordination (via MAPPO) is a more effective strategy.
Practical Applicability: The framework addresses the "scale-limited" and "system-specific" gaps in current literature, offering a solution that can adapt to diverse system architectures and operational scales without re-engineering.

The authors note limitations, including the current focus on homogeneous robot fleets and the abstraction of low-level motion planning (handled implicitly via congestion-aware travel times). Future work is suggested for heterogeneous fleets, explicit path planning, and integration with real-time sensory feedback.