💻 computer science

Multi-Objective Constrained Deep Q-Networks for Supplier Selection and Order Allocation in Sustainable Supply Chains

This paper introduces ParetoShield-Q, a multi-objective constrained deep Q-learning framework that optimizes sustainable supplier selection and order allocation by integrating preference conditioning, distributional tail-risk management, and feasibility oracles within a Markov decision process, validated through extensive quantitative analysis on public demand and supplier datasets.

Original authors: Miao Li, Zihan Wang, Xuwen Cao, Yangcui Weng

Published 2026-11-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Miao Li, Zihan Wang, Xuwen Cao, Yangcui Weng

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every time a store manager decides which supplier to call for a shipment and how many boxes to order, they are solving a puzzle that has grown far more complex than simply finding the lowest price. In the modern world, a single decision ripples through a network of competing goals: keeping costs low, ensuring products arrive on time, reducing the carbon footprint of the journey, and preparing for the possibility that a supplier might suddenly fail. These choices cannot be made in isolation; ordering too much from a cheap but unreliable source might save money today but cause a shortage tomorrow, while being overly cautious might waste resources. The challenge lies in making a sequence of these decisions day after day, where the outcome of one choice changes the options available for the next, all while juggling strict rules about budget, capacity, and environmental limits.

Researchers have long tried to help with this by using computer models that rank suppliers or calculate the best order for a single week. However, these traditional methods often struggle when the future is uncertain or when the goals conflict in complicated ways. A new study introduces a different approach, one that treats supplier selection not as a static list to be checked, but as a dynamic, continuous process of adaptation. The researchers developed a system that learns from past patterns to make real-time decisions, balancing the need for speed, cost, and reliability without breaking the rules of the supply chain.

The core of this new system is a computer program designed to act like a seasoned procurement manager who never forgets a lesson. Instead of just looking at the price tag, the program considers a full picture of the situation: how much stock is currently sitting in the warehouse, what the weather or traffic might do to delivery times, and how much carbon dioxide the transport will generate. It then simulates thousands of possible futures to see which combination of suppliers and order sizes will lead to the best overall result. The system is built to handle a specific type of difficulty: the fact that you cannot order half a truckload or split a shipment in a way that violates a contract. It ensures that every single decision it makes is physically possible and legally valid before it is ever sent to a supplier.

To test this idea, the researchers created a realistic simulation using real-world data. They took actual daily sales records from thousands of products across ten different stores and combined them with a database of supplier details, such as shipping costs, defect rates, and production times. They did not simply ask the computer to find the cheapest option; instead, they gave it a set of priorities that could be adjusted, like a dial. One day, the system might be told to focus on minimizing carbon emissions, and the next day, to prioritize getting products to the shelf as fast as possible. The program had to learn how to shift its strategy instantly to meet these changing goals while staying within a strict budget and avoiding shortages.

The results of this simulation showed that the new system, which the authors call ParetoShield-Q, performed better than several established methods used in industry and academia. When compared to other computer models, it managed to fulfill 96.7 percent of customer demand, a higher rate than most of its competitors. It also kept the number of rule-breaking decisions, such as ordering more than a supplier could handle, down to zero. While a traditional mathematical method could find a slightly cheaper total cost, it often failed to adapt as quickly when the situation changed or when the priorities shifted. The new system, by contrast, maintained high performance across all the different goals, from saving money to reducing environmental impact, without needing to be retrained for every new scenario.

One of the most significant findings was how the system handled unexpected trouble. The researchers tested the model by simulating severe disruptions, such as a scenario where forty percent of the suppliers suddenly became unavailable or unreliable. In these tough conditions, the new system still managed to fulfill 86.5 percent of demand, whereas other methods dropped to fulfillment rates as low as 69 percent. This suggests that the system's ability to look ahead and consider the "tail risk"—the chance of rare but disastrous events—helps it build a more resilient supply chain. It does not just react to a problem; it anticipates the possibility of one and adjusts its orders to buffer against the shock.

The study also revealed that the system is efficient enough to be used in real time. As the number of potential suppliers grew from a handful to eighty, the time it took for the system to make a decision remained under one second. This speed is crucial because supply chain managers often need to make hundreds of decisions every day. The researchers noted that the system achieves this speed by using a smart search method that quickly discards impossible options before spending time calculating the best one. It is like a navigator who knows which roads are closed before even starting the journey, ensuring that the final route is always drivable.

Despite these promising results, the researchers are careful to note the limits of their work. The findings come from a simulation based on public data, not from a live trial in a real company's warehouse. The data they used for demand was real, but the details about supplier emissions and disruption probabilities were constructed for the experiment. This means that while the system proved it could work well in a controlled environment, its performance in the messy, unpredictable reality of global trade would need to be verified with actual business data. The study does not claim to have solved the problem of sustainable sourcing forever, but it offers a powerful new tool that learns, adapts, and respects the hard constraints of the physical world.

Ultimately, this work represents a shift in how we think about supply chains. It moves away from the idea of a static plan that is set in stone at the beginning of the month and toward a dynamic process that evolves with every new piece of information. By combining the ability to learn from data with a strict adherence to the rules of feasibility, the system offers a way to navigate the complex trade-offs of modern commerce. It shows that it is possible to design a decision-making process that is both smart enough to find the best balance between cost, speed, and sustainability, and disciplined enough to ensure that every decision can actually be carried out. As supply chains face increasing pressure to be greener and more reliable, tools like this provide a glimpse of how technology can help managers make the right choices, one order at a time.

Technical Summary: Multi-Objective Constrained Deep Q-Networks for Supplier Selection and Order Allocation in Sustainable Supply Chains

1. Problem Formulation

The paper addresses the complex challenge of sustainable supplier selection and order allocation (SSOA) within a sequential decision-making framework. Unlike static multi-criteria decision-making, this problem is modeled as a preference-conditioned constrained Markov Decision Process (MDP).

The core difficulty lies in the coupling of three distinct elements:

Mixed-Integer Action Space: Decisions involve binary supplier activation (admission) and non-negative integer order quantities.
Multi-Objective Trade-offs: The system must simultaneously optimize economic cost, environmental impact (embodied emissions), service reliability (fill rates), and supplier diversification.
Stochastic Constraints and Risks: The environment includes stochastic lead times, demand uncertainty, and potential disruptions. Crucially, the problem requires satisfying hard constraints (budget, capacity, carbon limits) in the current period while managing tail risks (e.g., shortage losses) over the future horizon.

The authors note that existing methods often fail to handle the intersection of mixed-integer feasibility, non-convex Pareto trade-offs, and distributional tail-risk requirements simultaneously.

2. Methodology: ParetoShield-Q

The authors propose ParetoShield-Q, a feasibility-conditioned Pareto Bellman formulation designed to generate executable supplier allocations. The methodology integrates four key components:

Distributional Multi-Objective Q-Learning: Instead of estimating expected values, the critic learns a vector distribution of outcomes ( $Z_\theta$ ) for cost, emissions, service, and risk. This uses quantile regression to capture the full distribution of cumulative returns, enabling the estimation of Conditional Value at Risk (CVaR) for shortage tails.
Exact Completion-Feasibility Oracle: To ensure operational admissibility, the method employs an exact mixed-integer oracle. Before an action is selected, the system tests whether a partial supplier-quantity assignment can be extended to a fully feasible plan satisfying all current constraints (budget, capacity, carbon). This prevents the selection of actions that are mathematically optimal but operationally impossible.
Augmented Chebyshev Action Selection: To handle non-convex Pareto frontiers and user preferences, the method uses an augmented Chebyshev functional. Given a preference vector $w$ , it selects the action that minimizes the maximum weighted deviation from a reference point, augmented by a weighted sum term to ensure strict monotonicity. This allows the system to select non-convex trade-offs without retraining for every weight configuration.
Tail-Sensitive Risk Control: A projected dual variable ( $\lambda$ ) interacts with the CVaR estimate of shortage losses. This acts as a soft penalty for exceeding a declared tail-risk budget, separating deterministic constraint enforcement from predictive uncertainty.

The training process utilizes a shared distributional critic conditioned on the preference vector and risk budget, updated via quantile-regression Bellman errors. Inference involves a beam search over autoregressive supplier-quantity sequences, where the beam is pruned strictly by the feasibility oracle.

3. Key Contributions

The paper outlines four primary contributions:

Structured Constrained Procurement MDP: A unified formulation that integrates binary admission, integer allocation, inventory dynamics, carbon limits, and disruption-aware risk into a single feasible-action set, explicitly quantifying conditions where the set is empty.
Feasible Pareto Bellman Decision Rule: A new operator that restricts maximization to actions passing an exact extendibility test, utilizing an augmented Chebyshev criterion to navigate non-convex trade-offs.
Tail-Sensitive Allocation with Conditional Guarantees: A mechanism combining quantile-based CVaR with a projected risk multiplier. The authors prove conditional one-step feasibility (actions are guaranteed to satisfy known constraints) and finite-candidate Pareto-efficiency properties under specific assumptions.
Scenario-Based Empirical Evaluation: A comprehensive evaluation protocol using public M5 demand data and a static supplier-attribute dataset, coupled through a defined procurement simulation to assess trade-offs, preference adaptivity, and disruption robustness.

4. Results

The study evaluates ParetoShield-Q against several baselines, including Weighted DQN, Double DQN, Constrained DQN, Preference MORL, NSGA-II with repair, and Rolling MILP.

Multi-Objective Performance: ParetoShield-Q achieved the highest normalized Hypervolume (HV) of 0.85 and a demand fulfillment rate of 96.7%, outperforming other learning-based methods. While Rolling MILP achieved the lowest absolute cost (117.9), ParetoShield-Q offered a superior balance across all objectives with 0.0% constraint violations, matching the feasibility of the MILP and NSGA-II baselines.
Preference Adaptivity: The model successfully adapted to different regimes (cost-oriented, carbon-oriented, reliability-oriented, risk-averse). For instance, the carbon-oriented setting reduced the emissions proxy to 62.6, while the reliability-oriented setting increased fulfillment to 98.1%.
Ablation Analysis: Removing the feasibility oracle increased constraint violations to 12.4% and dropped HV to 0.58. Replacing the Chebyshev scalarization with a weighted sum reduced HV to 0.74. Replacing CVaR with expected shortage increased tail loss significantly (from 22.5 to 32.3).
Disruption Robustness: Under a 40% supplier disruption shock, ParetoShield-Q maintained 86.5% fulfillment, significantly outperforming Constrained DQN (76.1%) and Weighted DQN (69.4%).
Computational Scaling: The method demonstrated efficient scaling. At 80 suppliers, ParetoShield-Q required 0.82 seconds per decision, compared to 35.50 seconds for Full MILP and 47.32 seconds for a plain combinatorial DQN.

5. Significance and Claims

The paper claims that ParetoShield-Q provides a coherent distinction between hard current-period admissibility and uncertain future sustainability performance. By using an exact oracle, the system guarantees that the selected procurement instruction satisfies all known parameters and rules, even if the Q-estimator is imperfect.

The authors emphasize that the method does not claim to eliminate stochastic risks or guarantee future delivery outcomes; rather, it ensures that the decision made is executable and respects the stated risk budget. The study positions the work as a step toward learning sequential replenishment decisions that are both preference-controllable and operationally safe, addressing the gap between theoretical multi-objective reinforcement learning and the mixed-integer realities of sustainable procurement.

The authors remain modest regarding external validity, noting that the evaluation relies on a semi-synthetic environment where public demand data is coupled with static supplier attributes and scenario-defined emissions. They state that future validation on longitudinal purchase orders and auditable supplier-level data is necessary to strengthen claims of external applicability.