🤖 AI

Bridging Forecast Accuracy and Inventory KPIs: A Simulation-Based Software Framework

This paper introduces a decision-centric simulation framework that bridges the gap between statistical forecast accuracy and operational inventory KPIs in the automotive aftermarket, demonstrating that improvements in traditional error metrics do not necessarily translate to better cost and service outcomes and providing a tool for evaluating models based on their real-world business impact.

Original authors: So Fukuhara, Abdallah Alabdallah, Nuwan Gunasekara, Slawomir Nowaczyk

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: So Fukuhara, Abdallah Alabdallah, Nuwan Gunasekara, Slawomir Nowaczyk

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive fleet of delivery trucks. Your job is to make sure every truck has the right spare parts (like tires, engines, or filters) ready when they break down. If you have too many parts sitting in your warehouse, you waste money on storage and risk them becoming obsolete. If you have too few, the trucks sit idle, customers get angry, and you have to pay for expensive emergency deliveries.

The big question is: How do you know how many parts to keep?

Usually, companies try to answer this by using computer programs to predict when a part will break. For decades, experts have judged these prediction programs based on how "statistically accurate" they are. It's like grading a weather forecaster solely on whether they said "20% chance of rain" when it actually rained 15% of the time. They use complex math scores (like MAE or RMSE) to say, "This model is the best because its numbers are closest to the truth."

The Problem:
The authors of this paper argue that this approach is misleading. Just because a prediction model is mathematically "perfect" doesn't mean it saves you money in the real world. A model might be great at guessing the exact number of broken parts but terrible at helping you decide how many to stock, leading to huge costs.

The Solution: A "Flight Simulator" for Spare Parts
To fix this, the researchers built a special software framework. Think of it as a flight simulator for inventory management.

The Engine (Synthetic Data Generator): Instead of using real, secret company data, they built a digital engine that creates fake but realistic truck fleets. They simulate how trucks age, how weather (like winter cold or summer heat) affects parts, and how a dealership might suddenly grow from 50 trucks to 100. This creates a "training ground" where they can test different scenarios without risking real money.
The Pilot (Forecasting Models): They let different computer models (some modern AI, some old-school math) try to predict when parts will break in this fake world.
The Landing (Inventory Simulator): This is the most important part. The software takes the predictions and asks: "Okay, based on these predictions, how much money did we spend on storage, emergency shipping, and lost customers?"

The Big Surprise
When they ran the simulation, they found a counter-intuitive result: The "best" math models often cost the most money.

The "Smart" Models: Modern AI models (like XGBoost) were very good at getting the numbers right statistically. They had the lowest "error scores."
The "Old School" Models: Simple, older methods (like Croston's method) had higher error scores. They were "worse" at predicting the exact numbers.

However, when they looked at the final bill:
The simple, "worse" models actually saved the company the most money. The "smart" AI models, despite their high accuracy, led to decisions that resulted in higher costs.

The Analogy
Imagine you are trying to guess how many ice cream cones you will sell tomorrow.

Model A predicts you will sell 42.3 cones. It is mathematically very precise.
Model B predicts you will sell 40 cones. It is slightly less precise.

If you buy 42.3 cones (which you can't do, so you buy 43), you might throw away 3 unsold cones. If you buy 40, you might sell out, but you didn't waste any. In the real world, the "worse" prediction (Model B) might actually lead to a better business outcome because it aligns better with how you actually stock your shelves.

The Takeaway
The paper concludes that we need to stop judging forecasting models just by their "test scores" (statistical accuracy). Instead, we should judge them by their real-world impact (how much money they save or lose).

They have made their "flight simulator" software open-source, so other researchers can use it to test their own ideas. The goal is to shift the focus from "Who has the best math?" to "Who helps us run the business most efficiently?"

Technical Summary: Bridging Forecast Accuracy and Inventory KPIs

Problem Statement
In the automotive aftermarket, particularly for heavy-duty vehicle spare parts, demand is characterized by high intermittency, sparsity, and uncertainty. Efficient inventory management is critical to balancing the economic and environmental costs of overstocking (capital tie-up, obsolescence) against understocking (downtime, emergency shipments, customer dissatisfaction). While Artificial Intelligence and Machine Learning (AI/ML) offer advanced predictive capabilities, the evaluation of forecasting models in this domain remains dominated by statistical accuracy metrics (e.g., MAE, RMSE, $R^2$ ). A significant gap exists in understanding how these statistical metrics translate into operational Key Performance Indicators (KPIs), such as total inventory cost and service levels. Consequently, models with superior statistical accuracy may not yield optimal operational outcomes, and the relationship between forecast error and inventory performance is poorly understood.

Methodology
To address this gap, the authors propose a decision-centric, closed-loop simulation software framework designed to evaluate forecasting models based on their downstream impact on inventory decisions. The framework consists of three integrated modules:

Synthetic Demand Generator:
- Structure: A Discrete Time Simulation (DTS) system modeled hierarchically (Dealer $\to$ Truck $\to$ Part).
- Mechanism: Demand is generated by simulating part failures using survival models. The time-to-failure is treated as a random variable governed by hazard functions.
- Models: Four parametric survival models are implemented: Exponential, Weibull, Log-logistic, and Gompertz. These allow for heterogeneous failure dynamics (e.g., memoryless vs. aging components).
- Factors: The generator incorporates seasonality (modeled via Radial Basis Functions to capture regional and component-specific variations), usage intensity (Normal vs. Hard), and concept drift (sudden or slow changes in fleet size).
- Output: Daily demand time series for various part types across multiple dealers, replicating real-world intermittency (measured by Average Demand Interval and Coefficient of Variation).
Flexible Forecasting Module:
- This module accepts the synthetic demand data and hosts various predictive models.
- Models Tested: The study evaluates three machine learning models (XGBoost, Random Forest, SVR) and four statistical time-series methods (ARIMA, Croston's method, TSB, SBA).
- Training: Models are trained on the first two years of synthetic data to forecast the third year. Hyperparameters are tuned to minimize Mean Absolute Error (MAE).
Inventory Control Simulator (Cost Simulator):
- Mechanism: A Discrete-Event Simulation (DES) that processes the forecasts to drive inventory decisions.
- Policy: It employs a standard inventory policy (calculating safety stock, reorder points, and order quantities) based on established formulas.
- Cost Components: The simulator computes total operational costs, including holding costs, order costs, rush order costs, transportation costs, and badwill costs (penalties for service level shortfalls).
- Output: Operational KPIs, specifically Total Cost and Service Level (fill rate).

Experimental Setup
The authors generated 48 synthetic scenarios over a three-year horizon ( $T=1095$ days) with a daily time-step. Scenarios varied by:

Concept Drift: None, sudden, slow, or combined.
Fleet Size: Four ranges of trucks per dealer ( $N_t \in [5, 100]$ ).
Part Lifetime: Three ranges of median time-to-failure ( $t_m \in [100, 730]$ days).
Configuration: Four dealers, with fixed parameters for part price, lead time, and supplier capacity.

Key Results
The experiments revealed a significant disconnect between statistical accuracy and operational performance:

Accuracy vs. Cost: Models that achieved the best statistical accuracy (XGBoost and Random Forest, which minimized MAE and RMSE) resulted in higher total inventory costs compared to classical intermittent demand methods.
Counter-Intuitive Findings: Croston's method, which performed worse in statistical metrics (higher MAE/RMSE), achieved the lowest total inventory cost. Random Forest was second best in cost, while SVR and XGBoost incurred the highest costs.
Simpson's Paradox: Analysis of the relationship between error metrics (MAE, RMSE) and total cost revealed a weak or negative correlation within specific scenarios. Lower statistical error did not guarantee lower operational costs.
Model Rankings: The ranking of models changed dramatically when shifting the evaluation criterion from predictive accuracy to operational cost. Classical methods designed for intermittent demand (Croston, SBA, TSB) proved more robust for inventory optimization than complex ML models in this specific context.

Significance and Contributions
The paper claims two primary contributions:

A Modular, Open-Source Framework: The authors provide a reusable, transparent software framework that bridges demand forecasting and inventory management. By coupling a synthetic demand generator with an inventory cost simulator, it lowers the barrier for researchers to conduct rigorous, reproducible experiments that evaluate models based on operational relevance rather than just statistical error.
Empirical Quantification of the Accuracy-Performance Gap: The study empirically demonstrates that improvements in conventional accuracy metrics do not necessarily translate into better cost–service trade-offs. It highlights the risk of relying solely on statistical error measures for model selection in spare parts logistics.

Conclusion
The authors conclude that the evaluation of forecasting models in the automotive aftermarket must shift from a purely predictive focus to a decision-centric approach. The proposed framework operationalizes the link between forecasting and inventory, suggesting that future methodological development should prioritize operational relevance. The software and experimental results are released as open-source to facilitate further research in this domain.

Technical Summary: Bridging Forecast Accuracy and Inventory KPIs

More like this