Designing a double deep reinforcement learning selection tool for resilient demand prediction

Technical Summary: Designing a Double Deep Reinforcement Learning Selection Tool for Resilient Demand Prediction

Problem Statement
The paper addresses the critical challenge of demand forecasting in supply chain management, particularly in volatile environments characterized by events like the pandemic and regional conflicts. While numerous artificial intelligence (AI) forecasting architectures exist, no single method consistently performs optimally across all datasets due to varying data characteristics such as seasonality, noise, volatility, and intermittence. Traditional approaches often rely on manual expert selection or static optimization of hyperparameters, which are time-consuming, prone to error, and fail to adapt to real-time shifts in data dynamics. Furthermore, existing ensemble methods may not frequently review the selection of the appropriate solution, leading to accumulating errors over time. The core problem is the lack of an agile, automated mechanism to dynamically select the most suitable forecasting model from a committee of candidates based on the specific state of the data at any given moment.

Methodology
The authors propose a novel architecture, CRFFNN-ARIRBES, which functions as a double deep reinforcement learning (DDRL) agent to automate model selection.

1. Double Deep Reinforcement Learning (DDRL) Agent:

   • The system utilizes a Q-learning framework within a Markov Decision Process (MDP) to select a forecasting model from a "committee" of six neural network-based candidates (FFNN, LSTM, GRU, BiLSTM, CNN, and a CNN-LSTM hybrid).
   • To mitigate the overestimation of Q-values common in standard Deep Q-Networks, the authors employ a Double Deep Q-Network (DDQN) architecture. This involves two separate networks: an Online Q-network (updated frequently) and a Target Q-network (updated periodically) to provide a stable reference for action selection.
   • State Representation: The agent's state ($s$) is constructed by concatenating the historical demand of the previous 35 days ($d_{t-35} \dots d_{t-1}$) with the forecasted outputs of all six committee members ($f_{t1} \dots f_{t6}$).
   • Action: The action consists of selecting one specific forecasting model from the committee to generate the final prediction for the current time step.
   • Reward: The agent receives a reward based on the inverse of the Mean Squared Error (MSE) between the actual and predicted values, specifically formulated as $1 / e^{MSE}$, to penalize large errors more heavily.

2. CRFFNN Architecture:

   • Instead of a simple feedforward network, the Q-network utilizes a Convolutional Recurrent Feed-Forward Neural Network (CRFFNN).
   • Convolutional Layers: Extract spatial features from the input data.
   • Recurrent Layers: Capture temporal patterns and time-dependent relationships within the sequential data.
   • Feed-Forward Layers: Consolidate the extracted features to map them to the expected output (Q-values for each action).
   • This hybrid architecture allows the agent to better understand the complex spatial and temporal features of the environment state.

3. Average Reward Improvement Rate Based Early Stopping (ARIRBES):

   • To address the computational intensity and long training times associated with reinforcement learning, a novel early-stopping mechanism is introduced.
   • This method smooths the reward signal using a moving average. Training halts if the current average reward does not exceed the best historical average reward by a specific Improvement Rate (IR) threshold within a defined "patience" window. This prevents overfitting and significantly reduces training duration.

Key Contributions

1. DDRL for Dynamic Model Selection: The introduction of a double deep reinforcement learning architecture that dynamically selects forecasting models in real-time, utilizing a CRFFNN to process both historical demand and committee predictions for informed decision-making.
2. ARIRBES: The proposal of a novel early-stopping technique based on the rate of improvement in average rewards, designed to expedite training time without sacrificing performance.
3. Empirical Validation: A rigorous evaluation of the proposed approach against a forecasting committee and four automatic model selection strategies (Ensemble Averaging, Median, Stacking, and FFNN-DDQL) using two distinct datasets.

Experimental Results
The study was evaluated on two datasets: a public Corporación Favorita grocery sales dataset and a private client snack demand dataset (both with a 28-day forecasting horizon).

• Performance Metrics: The models were assessed using Symmetric Mean Absolute Percentage Error (SMAPE) and Mean Absolute Scaled Error (MASE).
• Accuracy: The proposed CRFFNN-ARIRBES method achieved the lowest Mean SMAPE and Mean MASE on both datasets compared to individual models (FFNN, LSTM, GRU, etc.) and other selection strategies (Stacking, FFNN-DDQL).
  • Favorita Dataset: Achieved a Mean SMAPE of 37.03 and Mean MASE of 1.79.
  • Snack Dataset: Achieved a Mean SMAPE of 48.52 and Mean MASE of 1.49.
• Stability: The proposed method demonstrated superior stability, evidenced by the lowest standard deviation in error metrics and narrow error distributions in violin plots across 10 different random weight initializations.
• Efficiency: The ARIRBES early-stopping mechanism drastically reduced training time. For the Favorita dataset, training time dropped from ~19,841 seconds (without early stopping) to ~5,120 seconds. For the snack dataset, it reduced from ~10,190 seconds to ~2,635 seconds, while maintaining or slightly improving forecast accuracy.

Significance and Claims
The paper claims that the proposed solution offers a robust and effective approach to the "no single model fits all" problem in demand forecasting. By leveraging the adaptability of reinforcement learning and the feature extraction capabilities of deep learning (CNN/RNN), the system can continuously adapt to changing data dynamics. The authors assert that their method outperforms state-of-the-art static selection and ensemble methods in terms of both accuracy and stability.

The study concludes that while the approach is computationally intensive due to the need to run the entire committee during training and inference, the introduction of the ARIRBES strategy makes the training process significantly more feasible. The authors modestly note that future work could involve incorporating macroeconomic variables, segmenting datasets into more homogeneous subgroups, and further optimizing the computational cost for large-scale industrial applications.