Technical Summary: FreshRetailNet-LT

Problem Statement

Accurate demand estimation is fundamental to inventory and pricing policies for perishable retail products. However, a critical challenge arises from censored sales data caused by stockout events. When a product is out of stock, observed sales drop to zero, not because demand has ceased, but because supply is exhausted. This creates a systematic bias where forecasting models trained on raw sales data learn to underestimate true latent demand. This underestimation perpetuates a vicious cycle: poor forecasts lead to insufficient inventory, causing more stockouts, which further corrupts demand measurements.

Existing public retail datasets (e.g., M5, Walmart, Favorita) fail to address this effectively due to two primary limitations:

1. Lack of Stockout Annotations: They do not distinguish between true zero demand and zero demand caused by inventory depletion, treating all zero sales as accurate demand observations.
2. Insufficient Temporal Granularity: Most datasets aggregate data at daily or weekly intervals. This obscures intraday demand patterns (e.g., morning/evening peaks) and dilutes the visibility of stockout events, which are often short-lived but critical for perishable goods.

Furthermore, standard imputation and forecasting methods often assume data is "Missing At Random" (MAR). In perishable retail, stockouts are "Missing Not At Random" (MNAR), driven by structured covariates like promotions, weather, and time of day, rendering standard approaches ineffective.

Methodology

The paper introduces FreshRetailNet-LT, a large-scale benchmark designed to facilitate research into latent demand recovery and forecasting under censoring. The methodology is structured around a two-stage framework:

1. Dataset Construction

FreshRetailNet-LT aggregates multi-source operational data from 1,057 stores across 18 major Chinese cities over a period of two years (June 2023 – July 2025).

• Scale: Contains over 20,000 store-product time series covering 576 perishable SKUs (vegetables, meat, seafood, fruits, etc.).
• Resolution: Hourly granularity, capturing detailed sales trajectories and stock status.
• Annotations: Uniquely provides verified hourly stockout labels, distinguishing between periods of genuine zero demand and periods where demand was censored due to stockouts.
• Covariates: Includes rich contextual data such as promotional discount rates, precipitation, temperature, and statutory holidays.

2. Two-Stage Modeling Approach

The authors propose a pipeline to first recover latent demand and then forecast it:

• Stage 1: Latent Demand Recovery: The goal is to reconstruct the unobserved demand ($d$) during stockout periods. The model takes observed sales ($y$), covariates (promotions, weather, etc.), and the stockout indicator ($s$) as input. It estimates the missing demand ($\hat{d}$) only when $s=0$ (stockout), while retaining observed sales when $s=1$.
  • Baselines Tested: Multi-periodicity models (TimesNet), attention-based imputation (SAITS, ImputeFormer, iTransformer), uncertainty-aware generators (GPVAE, CSDI), and interpretable baselines (DLinear).
• Stage 2: Censoring-Robust Forecasting: The recovered demand series is aggregated to a daily level to create a de-biased training set. This data is then used to train forecasting models for a 7-day horizon.
  • Baselines Tested: Similar Scenario Average (SSA), Temporal Fusion Transformer (TFT), DLinear, and the foundation model Chronos2 (zero-shot and LoRA fine-tuned).

Key Contributions

The paper claims three primary contributions:

1. Verified Stockout Event Annotations: It is the first open-source benchmark providing hourly, verified stockout labels. This enables researchers to directly distinguish between true zero demand and supply-constrained zero demand, a capability previously requiring indirect inference.
2. Multi-Year Longitudinal Coverage with High Granularity: The dataset captures both fine-grained intraday fluctuations (e.g., bimodal daily peaks) and long-term seasonal dynamics over two years, addressing the temporal resolution gap in existing perishable retail datasets.
3. Multi-Model Covariates: It uniquely integrates hyperlocal weather, holiday effects, and promotional discounts, supporting research into climate-resilient supply chains and dynamic pricing under perishability constraints.

Experimental Results

The authors evaluated the framework using Weighted Absolute Percentage Error (WAPE) for accuracy and Weighted Percentage Error (WPE) for bias, alongside a Decoupling Score ($\rho_{DS}$) to measure the removal of spurious correlations between stockouts and demand.

• Latent Demand Recovery:

  • TimesNet achieved state-of-the-art performance with a WAPE of 21.35% and a WPE of -8.00%.
  • Crucially, TimesNet achieved a Decoupling Score of -0.04, significantly closer to zero than raw sales data (-0.60), indicating it successfully eliminated the artificial link between stockouts and demand suppression.
  • Generative models like GPVAE showed higher errors (48.55% WAPE), suggesting they may over-smooth the volatile demand patterns of fresh produce.

• Forecasting Performance:

  • Using raw sales data, the TFT model exhibited a systematic underestimation bias (WPE of -6.69%).
  • When trained on demand recovered by TimesNet, TFT's bias was reduced to near-zero (-0.13%), and WAPE improved from 25.10% to 23.47%.
  • Similar improvements were observed across other models (SSA, DLinear, Chronos2) when using recovered demand, confirming that correcting the censoring bias enhances forecasting reliability.
  • Chronos2 (a foundation model) benefited from recovery but did not match the adaptability of specialized architectures like TFT, likely due to a lack of domain-specific adaptation to complex retail covariates.

Significance and Claims

The paper positions FreshRetailNet-LT as a pivotal resource for advancing perishable retail operations research. Its significance lies in:

• Breaking the Censoring Cycle: By providing ground-truth annotations for stockouts, it allows for the development of models that can break the self-reinforcing cycle of demand underestimation and inventory waste.
• Standardizing Benchmarks: It establishes a targeted benchmark for "inventory-aware forecasting," addressing the specific MNAR mechanisms found in fresh retail that general-purpose datasets miss.
• Enabling Causal Analysis: The dataset's structure supports causal inference regarding how promotional, climatic, and operational perturbations interact with perishability risks.
• Practical Utility: The authors claim the dataset provides immediate solutions for inventory optimization and a platform for future methodological innovation, bridging the gap between machine learning theory and the practical constraints of perishable supply chains.

The paper concludes that while the dataset addresses long-standing limitations, challenges remain in modeling sparse, long-tail SKUs and developing end-to-end joint optimization frameworks, pointing to these as future research directions.