🤖 AI

HiRAD: A Flexible Large-Scale AGV Routing System

This paper introduces HiRAD, a hierarchical reinforcement learning framework that overcomes the scalability and real-time limitations of classical and existing RL-based AGV routing systems by employing continuous-space representations, a decoupled control strategy, and an asynchronous event-driven pipeline to significantly reduce makespan and inference latency.

Original authors: Yunjie Huang, Ruizhong Wu, Mengxuan Zhang, Frodo Kin Sun Chan, Yan Nei Law, Lei Li

Published 2026-09-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yunjie Huang, Ruizhong Wu, Mengxuan Zhang, Frodo Kin Sun Chan, Yan Nei Law, Lei Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast, humming warehouses that power modern e-commerce, thousands of small, autonomous vehicles zip across the floor, ferrying packages from storage shelves to packing stations. These machines, known as automated guided vehicles, are the silent workhorses of global logistics, capable of moving billions of items daily. However, getting them to move together without crashing is a problem that has long stumped engineers. Traditional methods treat the warehouse floor like a giant chessboard, where vehicles jump from square to square in rigid, instant steps. While this simplifies the math, it ignores the reality of physics: real vehicles cannot stop instantly, nor can they turn on a dime. They need space to speed up, slow down, and steer. When software ignores these physical limits, the vehicles often collide in the real world, or the computer takes so long to calculate a safe path that the vehicles sit idle, waiting for instructions. This gap between the idealized digital map and the messy, continuous motion of real machines has made it difficult to deploy large fleets of these robots efficiently.

A team of researchers has now bridged this gap with a new system called HiRAD, designed to guide massive groups of these vehicles through complex warehouses in real time. Instead of forcing the robots to pretend they are jumping between grid squares, the researchers built a model that respects the continuous flow of motion. They realized that to make decisions fast enough for hundreds of vehicles moving simultaneously, the system needed to stop trying to control every tiny detail at once. Their solution involves splitting the driving task into two distinct layers. The first layer acts as a high-level navigator, looking at a broad, simplified map to decide which general direction a vehicle should head. The second layer is a local controller that handles the actual physics, managing how the vehicle accelerates, cruises, and brakes to reach that direction smoothly. By separating the "where to go" from the "how to move," the system drastically reduces the number of calculations it must perform at any single moment.

The researchers tested this approach in simulations involving up to two thousand vehicles moving across different warehouse layouts. They found that their system could complete tasks significantly faster than previous methods, reducing the total time required to finish a batch of orders by nearly half in some scenarios. Crucially, the system maintained safety without sacrificing speed; it prevented collisions even when vehicles were moving at high speeds and navigating tight spaces. Unlike older systems that would freeze and recalculate everything if a single vehicle encountered a delay, this new framework allows the rest of the fleet to keep moving, adjusting only the specific vehicles involved in the conflict. This flexibility means the system can handle unexpected hiccups, like a vehicle slowing down or a temporary blockage, without bringing the entire operation to a halt.

To ensure the system could run fast enough for real-world use, the team introduced a method of asynchronous decision-making. In a traditional setup, every vehicle would wait for the computer to check its path against every other vehicle before moving, a process that becomes impossibly slow as the fleet grows. The new system skips these checks for vehicles that are already moving smoothly, only pausing to make complex decisions when a vehicle needs to stop or change direction. This optimization reduced the computational load so effectively that the time it takes to make a decision became shorter than the time it takes for a vehicle to physically move. The result is a system that can manage large-scale fleets with the same ease as a small one, proving that it is possible to combine the adaptability of artificial intelligence with the strict physical laws that govern real-world machinery.

The study also explored how to train the artificial intelligence that powers the high-level navigation. The researchers discovered that teaching the system on simple, empty maps was not enough; the AI needed to learn on maps filled with obstacles and varying levels of difficulty to generalize well. By gradually increasing the complexity of the training environment, the system learned to find efficient paths even in crowded, chaotic warehouses. When tested against other leading methods, the new approach consistently outperformed them in both speed and reliability. While other systems struggled to handle more than a few hundred vehicles or failed to adapt when the environment changed, this framework scaled seamlessly to thousands of agents. The findings suggest that the future of warehouse automation lies not in rigid, pre-planned routes, but in flexible, continuous systems that can react to the world as it actually moves.

Technical Summary: HiRAD: A Flexible Large-Scale AGV Routing System

1. Problem Statement

The paper addresses the challenge of routing large-scale fleets of Automated Guided Vehicles (AGVs) in real-world warehouse environments. While AGVs are critical for modern logistics, existing routing solutions face a fundamental trade-off between flexibility (adapting to dynamic changes and failures) and kinematic constraints (adhering to physical motion laws in continuous space).

Classical Multi-Agent Pathfinding (MAPF): Solvers like CBS or ODrM* suffer from exploding combinatorial complexity and super-quadratic runtime. They often rely on idealized grid or piecewise-linear motion models that fail to capture real-world kinematics, potentially leading to collisions in continuous environments.
Reinforcement Learning (RL) Approaches: Recent decentralized RL solutions offer flexibility but struggle with scalability and realism. They typically depend on discretized spatiotemporal representations, require millions of training episodes, and incur high inference latency due to full-map observations at every step. This often violates real-time industrial control constraints.
The Core Gap: There is a lack of a system that simultaneously supports large-scale fleets, continuous kinematic constraints, and real-time decision-making without sacrificing safety or efficiency.

2. Methodology: The HiRAD Framework

The authors propose HiRAD (High-Resolution and Hierarchical Routing with Asynchronous Decision Framework), a hierarchical RL framework designed to bridge the gap between discrete RL algorithms and continuous physical execution. The framework consists of three core components:

A. High-Resolution Spatiotemporal (HRS) Modeling

To reconcile the step-based nature of RL with continuous motion, HiRAD introduces a high-resolution modeling approach:

HR-Map: The spatial grid is subdivided into smaller cells (e.g., $c \times c$ ), creating a finer spatial resolution.
HR-Step: Time intervals are divided into smaller steps ( $\Delta t$ ), allowing AGVs to make decisions at a higher temporal resolution.
Velocity Integration: Unlike standard MAPF where agents move one grid per step, HRS associates each step with a scalar velocity. This allows the modeling of acceleration, constant velocity, and deceleration, effectively simulating continuous dynamics within a discrete RL framework.

B. Hierarchical Alternating Decision-Velocity Control (Alt-DVC)

To manage the complexity of the expanded action space, HiRAD decouples navigation into two layers:

Macro-RL-Routing Layer: Operates on a coarse grid map. It uses a trained single-agent policy (reused for the fleet) to output collision-free directions. It treats other AGVs as dynamic obstacles within defined "Exclusion Zones" (a 5-grid area surrounding an AGV) to proactively prevent collisions.
Micro Velocity-Control Layer: Operates on the fine-grained HR-Map. It executes safe, continuous motions by mapping the macro direction into predefined velocity modes (e.g., Move to Next Grid, Continuous Cruising, Decelerate to Stop). These modes follow a three-stage profile: Acceleration $\to$ (Max) Speed $\to$ Deceleration, ensuring AGVs stop precisely at grid centers.

Collision Resolution: The system employs priority scheduling based on velocity and obstacle density. Higher-priority AGVs proceed, while lower-priority ones treat the exclusion zones of higher-priority agents as obstacles, stopping before entry.

C. Asynchronous Decision-Velocity Control (Asy-DVC)

To address the latency bottleneck where decision-making time exceeds physical execution time, HiRAD introduces an asynchronous optimization:

Observation Pruning: Since kinematic constraints prevent instantaneous heading changes, the system skips macro-policy inference for AGVs that are already moving. Inference is only triggered when an AGV is stationary ( $v=0$ ) and needs to reorient. This reduces observation complexity from $O(n)$ to roughly $O(n/4)$ .
Map-Oriented Priority: Instead of performing pairwise collision checks between all AGVs ( $O(n^2)$ ), the system uses a grid-indexed priority structure on the coarse map. An AGV only checks the top-priority agent in its destination grid, reducing priority handling complexity from $O(n^2)$ to $O(n)$ .
Result: This optimization lowers the overall inference complexity from $O(n^2)$ to $O(n)$ , cutting per-step latency by up to 71%.

3. Key Contributions

The paper claims the following contributions:

HiRAD Framework: A novel system enabling large-scale, collision-free AGV routing in kinematic-constrained environments with real-time guarantees.
High-Resolution Spatiotemporal Model: An adaptation of step-based RL to continuous environments by integrating fine-grained spatial cells and velocity states.
Hierarchical Control (Alt-DVC): A decoupled strategy separating direction and speed decisions to reduce the search space and enforce collision avoidance via exclusion zones and priority scheduling.
Asynchronous Optimization (Asy-DVC): A method utilizing observation pruning and map-oriented priority to reduce decision-making complexity from quadratic to linear, enabling real-world deployment.
Empirical Validation: Extensive experiments demonstrating superior performance over state-of-the-art solutions.

4. Experimental Results

The authors evaluated HiRAD on random graphs and two real-world warehouse layouts (Layout-A and Layout-B) with fleet sizes ranging from 64 to 2,048 AGVs.

Scalability: Classical methods (ICBS, ODrM*) failed to provide results for fleets exceeding 64 AGVs. PRIMAL (a large-scale RL baseline) could handle up to 2,048 AGVs but suffered from high latency and reduced success rates in dense scenarios. HiRAD successfully scaled to 2,048 AGVs.
Efficiency (Makespan): HiRAD reduced the makespan (time for the last agent to reach the goal) by 45%–63% compared to baselines. On a map of size 40, HiRAD was 3–4 times faster than PRIMAL; on size 160, it was approximately 2 times faster.
Runtime: HiRAD achieved significantly lower end-to-end runtime compared to both classical and other RL-based methods, meeting real-time constraints.
Flexibility: In "lifelong" task simulations involving dynamic failures (e.g., 10% of AGVs encountering issues), HiRAD maintained performance without system-wide re-planning. In contrast, centralized planners like OHSMD required halting and re-planning the entire fleet, leading to significant delays.
Training: The use of curriculum learning (starting with small, obstacle-free maps and gradually increasing difficulty) and Invalid Action Masking (IAM) significantly improved convergence speed and success rates on large, complex maps.

5. Significance and Claims

The paper positions HiRAD as a bridge between theoretical RL-based routing and practical industrial application. The authors claim that:

Previous solutions were impractical due to either discrete assumptions (violating kinematic constraints) or centralized planning (lacking flexibility).
HiRAD is the first RL-based solution to support large-scale fleets (up to 2,048 agents) while strictly adhering to kinematic constraints and ensuring collision-free operation in continuous space.
The asynchronous optimization (Asy-DVC) is critical for real-time applicability, ensuring that decision latency remains shorter than physical execution time.
The system offers a robust, adaptable solution for lifelong multi-AGV pathfinding in dynamic warehouse environments, outperforming state-of-the-art methods in both efficiency and scalability.