🤖 AI

Learning to replenish: A hybrid deep reinforcement learning for dynamic inventory management in the pharmaceutical supply chains

This paper proposes a hybrid deep reinforcement learning algorithm, specifically an A3C DPPO model, to optimize dynamic inventory replenishment in pharmaceutical supply chains by balancing product availability and waste reduction under uncertain demand and lead times, ultimately demonstrating improved profitability and lower costs compared to existing benchmarks.

Original authors: Amandeep Kaur, Gyan Prakash

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amandeep Kaur, Gyan Prakash

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, high-stakes game of musical chairs, but instead of people and chairs, you have medicines and patients. The music is the unpredictable demand from sick people, and the chairs are the shelves in pharmacies and hospitals. If the music stops too fast (a sudden surge in demand), someone gets left standing (a patient can't get their medicine). If the music plays too long without stopping, the chairs sit empty, and the medicine sitting on them might expire and have to be thrown away.

This paper is about teaching a computer how to be the perfect music director for this game, ensuring everyone gets a chair without wasting any seats.

Here is the breakdown of the paper in simple terms:

The Problem: A Chaotic Dance

Pharmaceutical supply chains are incredibly messy.

The Players: There are manufacturers (who make the drugs), distribution centers (the big warehouses), and retailers (pharmacies and hospitals).
The Chaos: People get sick unpredictably. Sometimes a flu hits hard (high demand), sometimes it's quiet. Also, medicines have expiration dates (like milk in your fridge). If you order too much, it rots. If you order too little, people suffer.
The Old Way: Traditional methods are like using a static map in a city where traffic changes every second. They try to guess the future based on fixed rules, but they often fail when things get weird or urgent.

The Solution: A "Smart Coach" (Deep Reinforcement Learning)

The authors built a new kind of computer brain called Deep Reinforcement Learning (DRL). Think of this as a video game AI that learns by playing the game thousands of times.

Trial and Error: The AI tries different ordering strategies. If it orders too much and medicine expires, it gets a "bad score" (penalty). If it runs out of stock and a patient can't get medicine, it gets a "bad score." If it keeps just the right amount, it gets a "good score" (reward).
Continuous Learning: Unlike old computers that only choose from a fixed list of options (like "Order 10" or "Order 20"), this AI can choose any number. It's like a chef who can add exactly 1.5 grams of salt, rather than just "a pinch" or "a cup."

The Secret Sauce: The "Hybrid A3C-DPPO" Algorithm

The paper introduces a specific recipe for this AI called A3C-DPPO. Let's break down the name with an analogy:

The Team (A3C - Asynchronous Advantage Actor-Critic): Imagine a football team. Instead of one coach shouting instructions to the whole field at once, you have multiple assistant coaches running around different parts of the field simultaneously. They all practice different plays at the same time. This makes the team learn much faster because they are exploring many possibilities at once.
The Safety Net (DPPO - Distributed Proximal Policy Optimization): Sometimes, when you learn something new, you might swing too hard and make a mistake. PPO acts like a safety harness. It allows the AI to learn and change its strategy, but it gently pulls it back if the change is too drastic. This keeps the learning stable and prevents the AI from going crazy.
The Hybrid: By combining the speed of the "multiple coaches" (A3C) with the safety of the "harness" (PPO), the system learns quickly but stays reliable.

How They Tested It

The researchers didn't just guess; they put this AI through rigorous tests:

Simulated Chaos: They created computer simulations with different types of "weather" for demand:
- Normal Weather: Predictable demand (like a sunny day).
- Stormy Weather: Skewed, unpredictable demand (like a sudden storm).
- Seasonal Weather: Demand that goes up and down in cycles (like flu season).
Real-World Data: They tested the AI using real data from a hospital that supplies two pharmacies. They looked at real medicines like Tamiflu (for the flu), Metformin (for diabetes), and Spikevax (a vaccine).

The Results: The AI Wins

The results were clear:

Better Scores: The AI earned significantly higher "rewards" (profit) than the old methods.
Less Waste: It reduced the cost of throwing away expired medicine by huge margins (up to 95% in some cases compared to other AI methods).
Fewer Stockouts: It kept the shelves stocked enough to meet the needs of patients almost 100% of the time, even when demand was crazy.
Adaptability: When the demand patterns changed suddenly, the AI adjusted its strategy much faster than the old rules-based systems.

The Bottom Line

This paper shows that by using a smart, hybrid computer brain that learns by doing, pharmaceutical companies can stop the game of "musical chairs" from being so chaotic. They can ensure patients get their life-saving medicines on time while wasting far less money on expired drugs. It's a move from guessing to smart, adaptive decision-making.

Technical Summary: Learning to Replenish in Pharmaceutical Supply Chains

Problem Statement
Pharmaceutical supply chains (PSCs) face significant challenges in inventory management (IM) due to unpredictable demand patterns, variable lead times, and the finite shelf lives of medications. Traditional deterministic models and classical reinforcement learning (RL) algorithms often fail to adapt to these dynamic conditions, particularly when dealing with large state-action spaces and continuous decision variables. The core problem addressed is the optimization of inventory replenishment policies to maximize profitability and patient service levels (PSL) while minimizing costs associated with ordering, holding, stockouts, and disposal of expired goods. The study specifically targets a multi-echelon PSC network comprising manufacturers, distribution centers, and retailers, where demand is stochastic and replenishment lead times vary.

Methodology
The authors formulate the IM problem as a Markov Decision Process (MDP) within a multi-echelon network. The system involves a global agent (distribution center) and multiple local agents (retailers) that make sequential decisions regarding order quantities and emergency replenishments.

State Space: Defined by on-hand inventory levels (categorized by shelf-life stages), open orders, emergency orders, stochastic demand, and lead times.
Action Space: Modeled as a continuous action space, allowing for precise control over order quantities rather than being restricted to discrete levels.
Reward Function: Designed to maximize net profit by balancing sales revenue against ordering costs, holding costs, emergency replenishment penalties, and disposal costs, subject to PSL constraints.

To solve this complex MDP, the paper proposes a Hybrid Asynchronous Advantage Actor–Critic Distributed Proximal Policy Optimization (A3C-DPPO) algorithm.

Architecture: The framework utilizes asynchronous parallel learning where multiple local agents (retailers) interact with their environments to collect experience. These agents compute local gradients for both policy (actor) and value (critic) networks.
Hybrid Mechanism: Unlike standard A3C which updates global parameters asynchronously, this approach aggregates gradients at a central server (distribution center) and applies PPO (Proximal Policy Optimization) updates. The PPO component uses a clipped objective function to ensure stable policy updates, preventing drastic deviations that could destabilize the supply chain.
Gradient Aggregation: The global actor and critic networks aggregate gradients from all local agents using importance weighting and exponential moving averages to synchronize learning while maintaining stability.

Key Contributions

Continuous Action Space Formulation: The study moves beyond discrete or quantized action spaces common in previous literature, proposing a continuous action space that offers greater flexibility and precision in replenishment decisions under stochastic demand.
Hybrid A3C-DPPO Algorithm: The development of a novel hybrid algorithm that combines the exploration efficiency of asynchronous A3C with the training stability of PPO. This addresses the "curse of dimensionality" and the sensitivity of PSCs to policy changes.
Multi-Echelon Coordination: The framework explicitly models the coordination between distribution centers and retailers, optimizing decisions across the entire network rather than in isolation.
Real-World Validation: The algorithm is validated using real-world pharmaceutical inventory data from a hospital system serving two pharmacies, covering seasonal, non-seasonal, and pandemic-driven demand scenarios.

Results
Numerical experiments and real-world validation demonstrate the superiority of the proposed A3C-DPPO algorithm over baseline policies, including classical $(s, S)$ , Deep Q-Network (DQN), and standard PPO.

Convergence and Rewards: The hybrid algorithm converges faster (approximately 3,000 episodes) compared to PPO and DQN (over 4,500 episodes). Under Gamma and Poisson demand distributions, it achieved average rewards 28% and 35% higher than PPO and DQN, respectively.
Cost Reduction: In real-world validation, the proposed algorithm significantly reduced average inventory costs compared to baselines. For instance, it reduced costs by 61.5% for Tamiflu and 95.1% for Metformin compared to DQN, and by 37.3% and 82.8% compared to PPO, respectively.
Robustness: Sensitivity analysis showed that the algorithm maintains robust performance under varying demand variances and strict PSL constraints (up to 0.99), whereas traditional methods struggle with high variability and convergence instability.
Impact of Constraints: The study observed that while higher PSL targets increase holding costs, the algorithm effectively balances these costs against stockout penalties, optimizing the trade-off better than benchmarks.

Significance and Claims
The paper claims that the proposed hybrid DRL framework represents a significant advancement in dynamic inventory management for pharmaceutical supply chains. By integrating asynchronous learning with stable policy optimization, the method successfully handles the high-dimensional, stochastic nature of PSCs. The authors assert that their approach provides a practical, real-time decision-making tool that ensures high patient service levels while minimizing waste and operational costs. The study concludes that this hybrid architecture enables seamless adaptation to diverse demand distributions, offering a robust solution for the complex, dynamic environment of modern pharmaceutical logistics. Future work is suggested to explore fully connected networks and temperature-sensitive products, but the current study focuses on validating the efficacy of the A3C-DPPO approach in a standard multi-echelon setting.