Offline Dynamic Inventory and Pricing Strategy: Addressing Censored and Dependent Demand

This paper proposes a novel data-driven framework using offline reinforcement learning and survival analysis to estimate optimal pricing and inventory control policies in sequential decision-making environments characterized by censored and dependent demand, overcoming challenges like missing profit information and non-stationarity through high-order Markov decision process approximations and finite-sample regret guarantees.

The Gist

Imagine you are running a small, trendy bakery. You want to figure out the perfect price for your croissants and exactly how many to bake each morning to make the most money.

Here's the tricky part: You don't have a crystal ball to predict the future. Instead, you only have a dusty old notebook (your offline dataset) filled with records from the past few years. But there's a catch: that notebook is full of holes.

The Two Big Problems

1. The "Sold Out" Mystery (Censored Demand)
Let's say on a rainy Tuesday, you baked 20 croissants and sold them all by 10 AM. Your notebook says "Sold: 20." But did 20 people buy them because that's all they wanted? Or did 50 people show up, but 30 left empty-handed because you ran out?
You don't know the real demand; you only know the sales. In the real world, this is called censored demand. It's like trying to guess how many people wanted to buy a concert ticket when the venue was full, but you only have a list of the people who actually got in. You are missing the data on the people who were turned away, so you don't know if you should have baked more.

2. The "Mood Swing" Effect (Dependent Demand)
Usually, we assume that what happens today has nothing to do with yesterday. But in this paper, the authors realize that demand is dependent.
Think of it like a viral trend. If you run out of croissants on Monday, people might get frustrated and stop coming on Tuesday. Or, if you have a huge sale on Monday, people might get excited and come back on Tuesday. The past changes the future. This makes the problem much harder because the rules of the game keep shifting based on what happened before.

The Solution: A Time-Traveling Detective

The authors propose a clever way to solve this using the "dusty notebook" without needing to run new experiments (which would be risky and expensive).

The "High-Order" Memory Trick
Standard computer models (like a simple Markov Decision Process) usually only look at today to decide what to do tomorrow. They have short memories.
But because demand depends on the past, the authors built a model with a longer memory. They created a "High-Order MDP."

• Analogy: Imagine a detective solving a crime. A normal detective asks, "Who was here right now?" This new detective asks, "Who was here right now, and who was here yesterday, and who was here the day before?"
• They specifically track how many times in a row you ran out of stock. If you sold out three days in a row, the model knows this is a special situation that requires a different strategy than if you sold out just once.

The "Survival Analysis" Connection
To fill in the missing "Sold Out" data, they borrowed a tool from medicine called Survival Analysis.

• Analogy: Doctors use this to predict how long a patient will survive after a treatment, even if some patients drop out of the study early. The bakery owners use this to predict "how long" the demand would have lasted if they had infinite croissants, even though the notebook says "we ran out." It helps them estimate the invisible customers who left empty-handed.

The Result: A New Playbook

By combining these ideas with Offline Reinforcement Learning (learning from old data rather than trial-and-error), they created two new algorithms.

Think of these algorithms as a GPS for your bakery:

1. They look at your old, messy notebook.
2. They fill in the blanks where you ran out of stock.
3. They remember how yesterday's mood affects today's customers.
4. They calculate the exact price and baking quantity to maximize your profit.

Why This Matters

Before this paper, if you wanted to learn the best strategy, you might have to guess, run experiments, and potentially lose money by running out of stock too often. This paper says, "You don't need to guess. You can learn the perfect strategy just by carefully studying your past mistakes and successes, even if your records are incomplete."

It's the first time anyone has successfully taught a computer to be a master baker and pricing expert using only a broken, incomplete notebook from the past.

Technical Summary

Here is a detailed technical summary of the paper "Offline Dynamic Inventory and Pricing Strategy: Addressing Censored and Dependent Demand" based on the provided abstract.

1. Problem Definition

The paper addresses a complex offline sequential decision-making problem involving simultaneous feature-based pricing and inventory control. The setting is characterized by two critical, challenging features:

• Dependent Demand: The current demand is not independent; it relies on past demand levels, introducing temporal dependencies.
• Censored Demand: The system operates under a "lost sales" model. If demand exceeds available inventory, the excess is lost. Consequently, the observed sales data is censored (we only observe the minimum of demand and inventory), leading to missing information regarding the true demand distribution and potential profit.

Objective: The goal is to learn an optimal pricing and inventory control policy that maximizes long-term profit using only a historical offline dataset. This dataset includes past prices, ordering quantities, inventory levels, covariates (features), and the censored sales observations.

2. Methodological Challenges

The authors identify three primary obstacles that prevent the direct application of standard Markov Decision Process (MDP) or Reinforcement Learning (RL) techniques:

1. Loss of Markov Property: Due to demand dependence on past levels and the presence of censoring, the standard state definition (current inventory and price) is insufficient to predict future outcomes. The system history is required, breaking the standard Markov assumption.
2. Missing Profit Information: Censoring obscures the true demand, making it impossible to directly calculate the profit associated with unobserved demand scenarios.
3. Non-Stationarity: The optimal policy is non-stationary because the underlying demand dynamics and the information state evolve over time based on the history of censoring events.

3. Proposed Methodology

To overcome these challenges, the paper proposes a novel framework combining Offline Reinforcement Learning and Survival Analysis:

• High-Order MDP Approximation:
  The authors reformulate the problem by approximating the optimal policy through a high-order MDP. The state space is expanded to include the number of consecutive censoring instances. This transformation allows the system to recover the Markov property by explicitly tracking the "memory" of the censoring history, which is crucial for estimating the underlying demand distribution.

• Specialized Bellman Equation:
  The problem is reduced to solving a specialized Bellman equation tailored to this high-order MDP structure. This equation accounts for the censored nature of the data and the dependent demand dynamics.

• Data-Driven Algorithms:
  Inspired by techniques from offline RL and survival analysis (which deals with censored time-to-event data), the authors propose two novel algorithms to solve these Bellman equations. These algorithms are designed to:

  • Handle the censored data without bias.
  • Estimate the value function and optimal policy directly from the offline dataset.
  • Leverage the feature-based nature of the demand to generalize across different covariates.

4. Key Contributions

• Theoretical Framework: This is the first data-driven approach specifically designed to learn optimal pricing and inventory policies in a sequential environment with both censored and dependent demand.
• Algorithmic Innovation: The development of two new algorithms that bridge the gap between survival analysis (for handling censoring) and offline RL (for policy optimization in high-order MDPs).
• Theoretical Guarantees: The authors establish finite-sample regret bounds, providing a mathematical proof of the algorithms' effectiveness and convergence properties.
• Practical Implementation: The paper includes numerical experiments demonstrating the efficacy of the proposed methods in estimating optimal policies compared to baselines. The code is publicly available on GitHub.

5. Significance and Impact

This research fills a critical gap in operations management and machine learning literature. Traditional inventory models often assume independent demand or fully observed sales, which rarely holds true in real-world retail and supply chain scenarios where stockouts (censoring) and demand momentum (dependence) are common.

By successfully addressing the "censored and dependent" challenge in an offline setting (where active exploration is impossible), this work enables businesses to:

• Optimize pricing and inventory strategies using historical data without risking customer loss through trial-and-error.
• Accurately model complex demand dynamics that were previously difficult to handle with standard MDPs.
• Improve long-term profitability by recovering lost information from censored sales records.

In summary, the paper provides a robust, theoretically grounded, and empirically validated solution for one of the most difficult problems in dynamic inventory management: learning from incomplete, history-dependent data.