💻 computer science

A Calibration-Aware Reference Architecture for Resilient Supply Chain Planning: Integrating Conformal Probabilistic Demand Forecasting with Scenario-Based Stochastic Vehicle Routing

This paper presents a reference architecture and open-source implementation that bridges machine learning and operations research in supply chain planning by integrating conformal probabilistic forecasting with scenario-based stochastic routing, empirically demonstrating through a synthetic backtest that uncalibrated quantile forecasts lead to significant under-coverage and stockout risks, thereby validating the necessity of calibration-aware interfaces for reliable decision support.

Original authors: Joseph Javier Sánchez Acuña

Published 2026-07-28

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Joseph Javier Sánchez Acuña

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, floating city that moves every day to deliver pizza, medicine, and toys to thousands of neighbors. This is the world of supply chains: the invisible web of trucks, warehouses, and schedules that keeps our modern life running. But here's the problem: the future is messy. Sometimes a storm hits, sometimes a factory breaks, and sometimes everyone suddenly orders extra pepperoni. To keep the city running, captains need to guess what will happen tomorrow.

For a long time, scientists have used two different toolkits to solve this. One toolkit is Machine Learning, which is like a super-smart detective that looks at past patterns to guess the future. The other is Operations Research, which is like a master chess player who figures out the best moves to make with the pieces you have. Usually, these two toolkits talk to each other, but they often speak different languages. The detective hands the chess player a single number (like "we need 100 pizzas"), and the chess player builds a plan based on that number. The big question this paper asks is: What if the detective's guess is slightly wrong? What if the "100 pizzas" is actually a 90% guess, but the detective acts like it's a 100% guarantee? If the chess player builds a plan based on a shaky guess, the whole city might run out of food or get stuck in traffic. This paper builds a new, safer way to connect the detective and the chess player, making sure they check their work before setting sail.

The Detective's Mistake: When "90%" Doesn't Mean 90%

The authors of this paper decided to test a very common habit in the supply chain world. Usually, when a computer predicts demand, it might say, "I'm 90% sure we won't need more than 500 units." This is called a quantile forecast. It's like a weather app saying, "There's a 90% chance it won't rain."

The researchers built a digital simulation of a logistics network with 16 customers over 400 days. They let a standard computer program (a "gradient-boosting ensemble") act as the detective. They asked it to predict the demand and then checked if the "90% sure" predictions were actually right.

The result was a surprise. The computer was lying to them, but not on purpose—it just didn't know how to calibrate its confidence. Even though the computer claimed to be 80% sure of its prediction intervals, the real world only matched those predictions 77.5% of the time. That's a gap of 2.47 percentage points.

Think of it like a weather app that says, "There's an 80% chance of no rain," but it actually rains 22.5% of the time. If you plan a picnic based on that app, you're going to get wet. In the supply chain, this means the trucks were being sent out with less space than they actually needed. The plan looked safe on paper, but in reality, the trucks were under-protected. The paper calls this a "negative result" because it proves that the old way of doing things is risky. The "90%" number the computer gave wasn't a real guarantee; it was just a guess that happened to be too optimistic.

The Chess Player's New Strategy: Testing the Plan Before Moving

Once they realized the detective was unreliable, the authors built a new architecture to fix the bridge between the detective and the chess player. They didn't just throw away the old tools; they added a "calibration layer."

Imagine you are the chess player again. Instead of just taking the detective's single number and moving your pieces, you now have a simulation lab.

The Calibration Layer: Before the detective gives you a number, you run it through a special filter (called Conformal Quantile Regression). This filter checks the detective's past mistakes and adjusts the prediction intervals so they are statistically more likely to match reality. The paper notes that this method provides "distribution-free marginal coverage under exchangeability," meaning it corrects the confidence levels based on the data's behavior, rather than promising a perfect fix in every possible scenario.
The Monte Carlo Lab: Instead of planning for one single future, the chess player now generates thousands of "what-if" scenarios. They run the same truck route against 400 different possible futures to see how often the trucks actually get stuck.
The "Value" Check: The authors wanted to know if all this extra math was actually worth it. They compared a "smart" plan (which used the thousands of scenarios) against a "dumb" plan (which just used the average guess). But they made sure both plans had the exact same number of trucks and the same amount of gas. This is crucial. If the smart plan wins just because it had more trucks, that's cheating. By keeping the trucks equal, they could measure the true value of having better information.

What They Found (and What They Didn't)

The paper is very honest about what it proved and what it just built.

The Proven Facts:
In their simulation of 16 customers over 400 days, the uncalibrated detective got worse and worse as time went on. As they moved through five different time periods (called "folds"), the quality of the predictions dropped by 16.1%. Even though the computer had more data to learn from (growing from 3,104 to 5,568 observations), it still got confused. This proves that just "more data" isn't enough if the method doesn't check its own confidence. The gap between what the computer said and what actually happened was real and measurable.

The "Under Construction" Parts:
The authors built a full toolbox that includes these new features: a way to measure the "Value of the Stochastic Solution" (how much money you save by using the smart plan), a way to check if the trucks are actually full enough, and a system to explain why the computer made a guess. However, they admit that while they built the tools to measure these things, they haven't run the final numbers for the "smart plan" vs. "dumb plan" comparison in this specific report yet. They labeled these as "Tier 2" components—meaning the engine is built and the test harness is ready, but the final race results aren't in the paper.

Why This Matters

The most important takeaway isn't a new super-algorithm; it's a new attitude. The paper argues that in supply chains, we stop assuming our plans are safe and start measuring them.

Before this, if a computer said, "We are 90% safe," everyone believed it. Now, the authors show that without a "calibration check," that 90% might actually be 77%. That difference is the difference between a truck arriving on time and a truck running out of gas in the middle of nowhere.

The authors created a "reference architecture"—a blueprint for how to build these systems so that every step is checked. They even made the code open-source, so anyone can download it, run the tests, and see for themselves. They didn't claim to have solved the world's supply chain problems, but they did prove that the way we usually check our work is broken, and they handed us a new, more honest way to fix it.

In short: Don't trust the detective's first guess. Check the math. Run the simulation. And always, always make sure your trucks have enough room for the rain that might actually fall.

Technical Summary: A Calibration-Aware Reference Architecture for Resilient Supply Chain Planning

1. Problem Statement

The paper addresses a critical methodological gap at the interface between machine learning (ML) and operations research (OR) in supply chain planning. While ML and OR are frequently combined, the transition is often unguarded: point or quantile forecasts are treated as deterministic inputs for optimizers, service levels are asserted rather than measured, and the marginal value of stochastic planning over deterministic baselines is rarely quantified.

The authors identify three specific, unexamined claims in typical integrated pipelines:

Calibration Claim: That a model's nominal quantile (e.g., $q_{0.90}$ ) accurately reflects the probability of exceedance (10%).
Service-Level Claim: That routing to a per-customer upper quantile guarantees route-level capacity feasibility.
Value Claim: That stochastic machinery outperforms deterministic mean-demand planning, rather than simply benefiting from the implicit "slack" of high-quantile inputs.

The paper argues that these claims are often empirically false and proposes a reference architecture designed to test, rather than assume, these properties.

2. Methodology and Architecture

The proposed architecture is a modular, open-source reference implementation (approx. 2,100 lines) designed for reproducible research. It integrates three distinct layers, each instrumented to validate specific assumptions:

A. Calibration-Aware Forecasting Layer

Base Models: Utilizes Gradient-Boosted Decision Trees (XGBoost) and a single-layer LSTM trained with pinball loss for quantile regression.
Conformalized Quantile Regression (CQR): Wraps base models to produce distribution-free, calibrated prediction intervals. It employs time-respecting per-customer calibration splits to ensure conformity scores are computed on unseen, chronologically subsequent data.
Post-Processing: Applies a monotone rearrangement operator to eliminate quantile crossing (where $q_{0.1} > q_{0.5}$ ).
Adaptation: Includes an optional adaptive conformal update to handle non-stationary regimes (trends, seasonality, exogenous drift).
Diagnostics: Reports pinball loss, approximate Continuous Ranked Probability Score (CRPS), empirical coverage, and interval width.

B. Measurably Constrained Stochastic Routing Layer

Sample Average Approximation (SAA): Replaces the heuristic of "solving deterministically on an inflated upper-quantile" with an SAA procedure. Scenarios are generated (truncated Normal distribution fitted to CQR intervals) and a fixed route plan is evaluated via Monte Carlo simulation.
Recourse Modeling: Implements a two-stage recourse cost model where capacity violations trigger a return-to-depot penalty.
Capacity Building: A wrapper that re-solves the routing problem at the capacity required to meet a target service level and re-validates against an independently seeded scenario draw to prevent overfitting.
Distance Backend: Supports both Haversine (default for synthetic data) and OSRM (Open Source Routing Machine) for realistic road-network distances.

C. Decision-Theoretic Evaluation Layer

Value Metrics: Computes the Value of the Stochastic Solution (VSS) and Expected Value of Perfect Information (EVPI).
Controlled Comparison: Crucially, the deterministic and stochastic plans are evaluated at identical vehicle capacity. This isolates the value of distributional information from the value of additional slack.
Significance Testing: Employs Diebold–Mariano tests (for pairwise comparisons) and Friedman–Nemenyi tests (for multiple models) to determine statistical significance.

D. Reproducibility Scaffold

Validation: Uses rolling-origin (walk-forward) cross-validation to detect temporal degradation.
Explainability: Provides SHAP attributions (with caveats regarding feature dependence) and windowed occlusion for LSTM models.
Tooling: Includes a 53-function test suite, pinned dependencies, Docker containerization, and optional MLflow tracking.

3. Key Results (Executed Evidence)

The paper distinguishes between "Tier 1" (executed and archived) and "Tier 2" (implemented but not archived) results. The reported empirical findings are based on a five-fold rolling-origin backtest on a synthetic 16-customer, 400-day logistics network.

Systematic Under-Coverage (Tier 1): An uncalibrated quantile gradient-boosting ensemble achieved an empirical coverage of 0.775 against a nominal target of 0.800. This represents a mean shortfall of 2.47 percentage points, with under-coverage occurring in four of the five folds. The authors note this is a "negative result" about the widely used baseline, demonstrating that nominal quantile levels are not calibrated coverage levels and treating them as such understates stockout risk.
Progressive Degradation (Tier 1): Forecast quality (measured by CRPS) degraded monotonically across successive origins. Despite a 79% increase in training data (from 3,104 to 5,568 observations), the CRPS increased by 16.1% from the first to the last fold. This trend was only visible through rolling-origin evaluation; a single train/test split would have concealed this systematic degradation.
Uncertainty Quantification: The paper explicitly states that the coverage figures characterize the uncalibrated baseline. The architecture successfully detected this failure mode, providing the empirical motivation for the conformal layer.

4. Significance and Claims

The paper modestly positions itself as a methodology and reference-architecture contribution rather than a claim of state-of-the-art forecasting accuracy or routing optimality. Its significance lies in:

Explicit Interface Testing: Converting implicit assumptions (calibration, service levels, stochastic value) into explicit, testable measurements.
Honest Empirical Reporting: Providing a "negative result" that challenges the common practice of using uncalibrated quantile forecasts as deterministic inputs, showing that such practices produce plans that are "more reliable than they are."
Reproducibility as Substance: Demonstrating that rigorous validation protocols (rolling-origin, significance testing, controlled capacity comparison) are necessary to reveal systematic trends and avoid false positives that single-split evaluations miss.
Open Implementation: Releasing a complete, containerized pipeline with a comprehensive test suite to allow other researchers to extend the work (e.g., toward decision-focused learning or distributionally robust optimization) against a documented, auditable baseline.

The authors conclude that the architecture's primary contribution is the integration of established techniques into an auditable pipeline where the transition from learning to optimization is measured, not assumed.