📈 economics

Experimental Designs for Multi-Item Multi-Period Inventory Control

This paper investigates the systematic biases inherent in standard A/B testing designs for multi-item, multi-period inventory systems due to temporal carryover and cannibalization, proposing and validating a pairwise experimental design that effectively mitigates these interference effects.

Original authors: Xinqi Chen, Xingyu Bai, Zeyu Zheng, Nian Si

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinqi Chen, Xingyu Bai, Zeyu Zheng, Nian Si

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a massive, floating grocery store that sails through time. Every day, you have to decide how much of every single item—from bananas to batteries—to stock on your shelves. If you guess wrong and run out, customers leave empty-handed (a "lost sale"). If you guess too high, you waste money storing things that might rot. Now, imagine you want to test a new, smarter way to make these guesses. You can't just try it on the whole ship at once because if it fails, you lose everything. So, you decide to run a "test drive." You split your crew into two groups: one group keeps using the old guessing method (the "control"), and the other tries the new method (the "treatment"). This is called an A/B test, the gold standard for figuring out if a new idea actually works.

But here's the tricky part: your ship has a limited amount of space in its cargo hold. If the "new method" group orders too many bananas, they might take up all the space, leaving no room for the "old method" group's apples. Or, if the new method leaves a huge pile of unsold bananas today, those bananas might still be there tomorrow, messing up the test for the next day. In the world of science, this is called "interference"—where the test of one thing accidentally changes the results of another. The big question is: How do you run a fair test on a crowded, time-traveling grocery ship without the results getting messed up by the cargo hold's limits or the leftover fruit?

This paper dives deep into that exact problem, specifically for inventory systems with lost sales and capacity limits. The authors, Xinqi Chen and colleagues, act like detectives investigating three different ways to run these tests: Switchback (where the whole ship switches between old and new methods every few days), Item-Level Randomization (where specific items are permanently assigned to old or new methods), and Pairwise Randomization (a mix where every single item on every single day gets its own coin flip). They discovered that the "best" way to test depends entirely on what is wrong with the old guessing method.

If the old method was just consistently underestimating how many people want to buy things (a "mean bias"), the Switchback test is a trap. It tends to make the new method look worse than it really is because of leftover inventory from the "good" days carrying over into the "bad" days. Meanwhile, the Item-Level test tends to make the new method look too good because the new items hog the limited shelf space, starving the old items and making the new method look like a hero. In this case, the authors found that Pairwise Randomization is the sweet spot, balancing out these errors.

However, if the old method was actually guessing the right average number of sales but was just very "jittery" or unpredictable (a "dispersion" problem), the story flips. Now, the Switchback test makes the new, steadier method look even better than it is (because the new method leaves less messy leftover inventory), while the Item-Level test actually works perfectly and gives an unbiased answer. The Pairwise test, in this scenario, starts to suffer from the same "leftover" bias as the Switchback.

The authors didn't just guess these outcomes; they built complex mathematical models to prove the direction of these biases and ran thousands of computer simulations to verify them. They even tested their theories on real-world data from a fresh-food retail network (FreshRetailNet-50K), where they simulated stockouts and customers swapping items when their first choice was gone. The results held up: the same mechanisms that caused bias in their simple math models also caused bias in the messy, real world.

So, what's the takeaway for the captain of the floating grocery store? Don't just pick a testing method because it sounds cool. If you are trying to fix a method that consistently underestimates demand, avoid switching the whole ship back and forth; instead, try assigning different items to different methods, or use the clever "Pairwise" mix. But if you are testing a method that is just more consistent and less jittery, you can safely stick items to their methods for the long haul. The paper warns that blindly using the popular "Switchback" method in inventory systems is risky and often leads to systematic errors, proving that in the chaotic world of inventory, one size does not fit all.

Technical Summary: Experimental Designs for Multi-Item Multi-Period Inventory Control

Problem Statement
While randomized experiments (A/B testing) are the gold standard for evaluating interventions in digital platforms and economics, they remain underutilized in inventory management. Traditional evaluation in this domain relies on simulation, which faces significant limitations: inventory data is censored (unmet demand is unobservable), post-stockout customer behaviors (substitution, abandonment) are complex to model, and high-fidelity simulations are computationally expensive. This paper addresses the gap by analyzing A/B testing strategies for evaluating inventory control policies in multi-item, multi-period systems with lost sales and warehouse capacity constraints. The core challenge is that standard experimental designs often violate the Stable Unit Treatment Value Assumption (SUTVA) due to interference: temporal carryover effects in time-based designs and cannibalization effects across items under shared capacity constraints.

Methodology and Model Formulation
The authors model a periodic-review inventory system with $N$ items and a shared capacity $B$ over $T$ periods. Unmet demand is lost. The system employs a Myopic Clairvoyant Policy, which optimizes expected single-period rewards given demand estimates, serving as a heuristic for non-stationary environments.

The study compares three experimental designs, all maintaining a marginal treatment probability $p$ :

Switchback (SW): All items share the same treatment assignment within a period, which randomizes over time.
Item-Level Randomization (IR): Each item is assigned to treatment or control for the entire horizon.
Pairwise Randomization (PR): Every item-period cell is independently randomized.

The authors analyze the bias of the Inverse Probability Weighting (IPW) estimator for the Global Treatment Effect (GTE) under two distinct forecasting improvement scenarios:

Scenario 1 (Mean Bias): The treatment corrects a systematic downward bias in demand forecast means (e.g., correcting for censoring), while forecast dispersion remains constant.
Scenario 2 (Dispersion Reduction): The treatment reduces the dispersion (variance) of forecast errors while preserving the mean.

Theoretical analysis is conducted using Lagrangian relaxation for the capacity-constrained optimization, deriving conditions under which the system remains in specific structural regimes (slack or interior). For Scenario 2, the authors employ a mean-field limit approach where $N \to \infty$ and capacity scales proportionally, allowing for asymptotic bias characterization.

Key Contributions and Results

Bias Characterization in Scenario 1 (Mean Bias):
- Switchback (SW): Systematically underestimates the GTE (negative bias). This occurs because improved forecasts increase order-up-to levels, generating more leftover inventory that carries over into subsequent control periods, artificially depressing the control group's performance in the estimator.
- Item-Level (IR): Systematically overestimates the GTE (positive bias). Under tight capacity, treated items with higher targets crowd out control items, reducing the inventory available to the control group relative to the global control counterfactual.
- Pairwise (PR): The bias is upper-bounded by the IR bias. It combines the downward force of temporal carryover (like SW) and the upward force of cross-sectional competition (like IR). Depending on the dominance of these forces, PR can be less biased than IR but may still exhibit negative bias if carryover is strong.
Bias Characterization in Scenario 2 (Dispersion Reduction):
- Switchback (SW): Systematically overestimates the GTE (positive bias). The treatment's lower dispersion leads to less expected leftover inventory compared to the control. In a switchback design, this reduced leftover in treatment periods carries over, inflating the treatment-control contrast.
- Item-Level (IR): Is asymptotically unbiased. Since items are permanently assigned, there is no temporal switching. In the mean-field limit, the capacity multiplier becomes deterministic and shared, eliminating spillover distortions.
- Pairwise (PR): Exhibits the same asymptotic positive bias as Switchback, inheriting the intertemporal leftover-carryover mechanism.
Empirical Validation:
- Controlled Simulations: Synthetic experiments using uniform demand distributions confirm the theoretical signs of bias across tight, medium, and loose capacity regimes.
- Trace-Driven Experiments: Using the FreshRetailNet-50K dataset (50,000 store-product time series with stockout annotations), the authors validated these mechanisms in a realistic environment. They incorporated latent demand recovery and stockout substitution. The results confirmed that the theoretical bias directions persist even with real-world noise and substitution effects.

Significance and Practical Implications
The paper demonstrates that the direction and magnitude of experimental bias in inventory systems are highly sensitive to the type of forecasting improvement (mean vs. dispersion) and the experimental design chosen.

Based on these findings, the authors offer the following practical recommendations:

Use Item-Level Randomization (IR) when capacity constraints are loose or when the treatment and control differ primarily in forecast-error dispersion (Scenario 2). In these regimes, IR exhibits low bias and lower sampling variability.
Use Pairwise Randomization (PR) when the treatment and control differ substantially in estimation bias (Scenario 1) and capacity is relatively tight. PR can achieve lower bias and sampling variability than both IR and Switchback in these specific conditions.
Exercise Caution with Switchback Designs (SW). In multi-period inventory systems, SW experiments are prone to systematic bias due to inventory carryover and typically exhibit high sampling variability. The authors advise against their use in such settings unless the specific interference mechanisms are well-understood and accounted for.

Conclusion
This study provides a rigorous theoretical and empirical framework for designing A/B tests in inventory management. It highlights that standard experimental designs, while effective in other domains, can yield misleading conclusions in inventory systems due to unique interference structures (temporal carryover and capacity-induced cannibalization). By characterizing these biases, the paper enables practitioners to select experimental designs that align with their specific operational regimes and the nature of the inventory policies being tested.

More like this