🤖 AI

Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection

This paper introduces a differentiable projection framework that embeds a convex optimization module within deep reinforcement learning to enforce hard, interdependent constraints in sequential decision-making, achieving near-optimal performance and significant cost reductions in complex inventory planning problems where traditional methods struggle.

Original authors: Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts, Stefan Minner

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts, Stefan Minner

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, chaotic spaceship trying to navigate through a storm of asteroids. You have a brilliant navigator (a computer brain) who can predict the future and suggest the perfect course. But there's a catch: your ship has strict, unbreakable rules. You can't fly through an asteroid, you can't exceed your fuel limit, and your cargo hold has a maximum weight. If your navigator suggests a path that breaks even one of these rules, the ship crashes. This is the daily struggle of "Operations Research," a field of science dedicated to making the best possible decisions when resources are limited and the future is uncertain. For decades, computers solved these puzzles using rigid, slow math that could handle the rules but struggled with the chaos of real life. Meanwhile, a newer, flashier type of computer brain called "Deep Reinforcement Learning" (DRL) learned to fly by trial and error, getting faster and better at handling chaos, but it often crashed because it didn't know how to respect the hard rules.

The big question scientists have been asking is: Can we build a pilot that is both fast and smart like the new DRL brains, but also strictly obeys the hard safety rules like the old math? If we could, we could manage everything from global supply chains to factory floors much more efficiently, saving billions of dollars and preventing shortages. This paper steps into that exact corner of science, trying to bridge the gap between the flexible learning of AI and the rigid safety of traditional math.

The Paper's Story: Teaching a Robot to Play by the Rules

The authors of this paper, Patrick Helm, Jan-Niklas Doerr, Joren Gijsbrechts, and Stefan Minner, have built a new kind of "pilot" for complex decision-making problems. They call it a differentiable policy with differentiable projection. That sounds like a mouthful, so let's break it down with a simple analogy.

Imagine you are playing a video game where you control a robot that has to stack boxes. The robot's brain (a neural network) is very creative; it looks at the situation and shouts out a "target" for where it wants to put the next box. Sometimes, this target is perfect. But often, the robot is too excited and suggests stacking a box in mid-air or putting too many boxes in a tiny space. In the past, if the robot made a mistake, the game would either let it crash (bad) or force it to stop and calculate a new path from scratch (slow).

The authors' solution is a three-step "safety filter" that sits between the robot's brain and its hands:

The Dreamer: First, the robot's brain proposes a continuous, smooth target. It doesn't worry about the rules yet; it just dreams up the ideal move.
The Projector: Next, the move hits a "differentiable projection" module. Think of this as a magical, elastic wall. If the robot tries to push a box into a wall, the wall gently but firmly pushes the box back to the nearest safe spot. Crucially, this wall is "smart." It doesn't just shove the box; it calculates exactly how hard it pushed back and sends that information back to the robot's brain as a "lesson." This allows the robot to learn why it was wrong and how to adjust its dream next time, all without breaking the game's physics.
The Integer Mapper: Finally, the robot's hands can only grab whole boxes, not half-boxes. The system takes the smooth, safe position from the previous step and snaps it to the nearest whole number. But here's the trick: it uses a special "surrogate gradient" (a clever mathematical shortcut) to pretend the snapping process was smooth, so the robot's brain can still learn from the result.

What They Found and Why It Matters

The team tested this new pilot on some very tough problems: managing inventory in multi-layered supply networks (like a factory that makes parts for other factories). These are problems where resources are tight, demand changes wildly, and every decision affects the next.

In their simulations, the results were impressive. On small test cases where they knew the perfect answer, their new method was almost spot-on, missing the perfect score by less than 1% on average. When they moved to larger, more complex networks (like the ones used by major companies), their method beat the current "best" strategies by a significant margin. Specifically, it saved up to 9.75% in costs compared to the best existing inventory policies and outperformed complex rolling-horizon planning programs by at least 7.7%.

They also tested it on a real-world industry challenge from ASML, a giant in semiconductor manufacturing. In this high-stakes environment, their policy reduced average costs by 3.22% compared to the best-known benchmark. To put that in perspective, ASML's machines are worth hundreds of millions of dollars; saving even a tiny percentage on inventory and production costs translates to massive amounts of money.

What They Ruled Out

The paper is very clear about what doesn't work. They explicitly argue against two common shortcuts:

Just penalizing mistakes: Some methods try to teach a robot to follow rules by just giving it a "scolding" (a penalty) when it breaks them. The authors show this doesn't work well for hard constraints; the robot might still try to break the rules if the reward is high enough.
Simple rounding: If you just take a smooth number and round it down to the nearest whole number (like rounding 3.9 to 3), you often miss out on the best possible moves. Their "dual-informed" mapping is much smarter, ensuring the robot can actually reach the most efficient, "edge-of-the-possible" solutions that simple rounding misses.

The Bottom Line

This paper doesn't claim to have solved every inventory problem in the universe. Instead, it demonstrates that by embedding a smart, rule-following "projection" step directly inside the learning process, we can train AI agents that are both highly efficient and strictly compliant with complex, interdependent rules. The authors suggest that this approach is particularly valuable when resources are tight and demand is unpredictable—precisely the situations where traditional methods struggle the most. By keeping the learning process smooth and differentiable, they managed to teach a computer to make discrete, integer decisions (like "make 5 units") while respecting hard constraints, achieving results that are not just theoretically sound, but economically significant in real-world simulations.

Technical Summary: Learning Feasible Inventory Policies via Differentiable Projection

Problem Definition
The paper addresses constrained sequential decision processes characterized by large, combinatorial action spaces and interdependent feasibility constraints. While Mixed-Integer Linear Programs (MILPs) can handle such constraints flexibly, they scale poorly in stochastic environments. Conversely, Deep Reinforcement Learning (DRL) offers scalable decision rules but typically struggles with hard constraints. Existing DRL approaches either penalize constraint violations (Lagrangian relaxation), failing to guarantee strict feasibility, or rely on feasibility mechanisms (e.g., differentiable scaling) that break down when constraints are interdependent rather than simple. The core challenge is to learn a policy that enforces hard, state-dependent, and interdependent constraints (such as shared resource limits and material balances) while maintaining differentiability for end-to-end training via pathwise gradients.

Methodology
The authors propose a differentiable policy architecture that embeds a convex optimization module directly into the policy network. The policy consists of three sequential modules:

Differentiable Decision Rule: A neural network maps the system state to continuous action targets ( $z_t$ ).
Differentiable Projection (QP): A Quadratic Program (QP) projects the continuous targets onto the relaxed feasible set of the continuous relaxation ( $A_{rel}$ ). The QP minimizes the weighted squared Euclidean distance between the target and the feasible set. Crucially, the authors utilize implicit differentiation of the QP's optimality conditions (KKT conditions) to compute analytical sensitivities (Jacobian matrices) with respect to the inputs. This ensures that gradients flow back through the projection step without breaking the differentiability required for Hindsight Differentiable Policy Optimization (HDPO). The use of a QP (rather than an LP) ensures strong convexity, yielding a unique, smooth optimizer that avoids the degeneracy and discontinuous jumps associated with LP solutions.
Dual-Informed Integer Mapping: Since operational decisions (e.g., production quantities) must be integers, the continuous projected action ( $x^*_t$ ) is mapped to a discrete feasible action ( $x_t$ ). This mapping uses a greedy local search guided by the dual variables (shadow prices) from the QP. It starts with the component-wise floor of the projected action (guaranteed feasible due to the downward-closed nature of the constraints) and iteratively adds units to coordinates with the highest priority scores (derived from the objective improvement and dual prices) as long as feasibility is maintained. To enable gradient flow through this non-differentiable integer step, the authors employ a Straight-Through Estimator (STE), treating the mapping as an identity function during backpropagation.

Key Contributions
The paper provides three main theoretical and practical contributions:

Architecture: A novel end-to-end differentiable policy that combines a neural decision rule, a convex projection layer, and a dual-informed integer mapping. This allows the policy to handle hard, interdependent constraints with the flexibility of MILPs while retaining the scalability of DRL.
Theoretical Guarantees:
- Bounded Error: The two-step mapping (projection + integer mapping) introduces a bounded error relative to the exact integer projection, justifying the avoidance of per-state integer programming.
- Completeness: Unlike simple flooring, which fails to reach certain feasible actions on the "active boundary" (where constraints are tight) from a positive-measure set of targets, the proposed integer mapping is complete. Every feasible integer action is reachable from a non-negligible region of the continuous target space.
- Frontier Selection: The mapping ensures that when targets are ambitious, the resulting action lies on the integer frontier (fully utilizing resources), preventing the waste of capacity.
Empirical Validation: The method is applied to multi-echelon production-inventory planning under shared resource and material constraints, a domain where existing DRL methods struggle.

Results
The authors evaluate the policy across three experimental settings:

Small Instances (3 items): Compared against the optimal policy derived via relative value iteration, the proposed method achieves an average optimality gap of 0.91%, outperforming the best-known base-stock policy (SBS*) on 22 of 24 instances.
Larger Instances (10 items, Tempelmeier & Derstroff benchmarks): The method outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program (MSP) by at least 7.7%. The gains are most significant in tightly capacitated networks with high demand variability and interdependent resources.
Industry-Scale Case Study (ASML 2025 Research Challenge): On a realistic 14-item benchmark, the policy reduces average costs by 3.22% and 2.54% (across two demand settings) relative to the best-known benchmark (a forecast-based echelon base-stock policy with shortfall allocation). Furthermore, the policy's cost distribution first-order stochastically dominates the baseline, indicating a higher probability of achieving lower costs.

Significance and Claims
The paper claims that traditional inventory policies (like base-stock) struggle in tightly capacitated systems with high demand variability because they separate stocking and rationing decisions. In contrast, the proposed learning-based approach optimizes these decisions jointly, delivering economically significant savings precisely where planning is most difficult.

The authors emphasize that their approach bridges the gap between the formulation-driven workflow of MILPs (where constraints are added to the model) and the scalability of DRL. By embedding the constraint handling within the policy via a differentiable projection, the method remains robust to changes in resource structures without retraining the neural architecture. The results suggest that DRL can deliver substantial economic value in sequential decision problems with interdependent hard constraints, a class of problems widespread in operations research but previously difficult to solve with learning-based methods. The work is supported by the Deutsche Forschungsgemeinschaft (DFG).

The Paper's Story: Teaching a Robot to Play by the Rules

What They Found and Why It Matters

What They Ruled Out

The Bottom Line

Technical Summary: Learning Feasible Inventory Policies via Differentiable Projection

More like this