Technical Summary: Integrable Elasticity via Neural Demand Potentials

Problem Statement

In multiproduct retail pricing, a central challenge is learning stable, reproducible, and economically plausible price elasticities at scale. Elasticities are derivatives of the demand map; consequently, small-scale irregularities in an estimated demand surface can be amplified into large, noisy, or economically implausible fluctuations in elasticity estimates. This issue is exacerbated in multiproduct settings where the elasticity matrix couples all Stock-Keeping Units (SKUs), and cross-price effects are often weak, heterogeneous, and difficult to identify from observational data due to correlated pricing and promotional activity.

Existing approaches face a trade-off:

• Classical structural models (e.g., AIDS, QUAIDS) impose rigid functional forms that may fail to capture complex, state-dependent substitution patterns.
• Standard machine learning models (e.g., Gradient Boosting, generic neural networks) often treat elasticity as a post-hoc derivative of a fitted predictor. While flexible, these models do not guarantee that the resulting elasticity field corresponds to a single, globally coherent demand function. If elasticities are parameterized directly or derived numerically from an irregular surface, the model may produce locally plausible elasticities that are not globally integrable, leading to path-dependent demand reconstructions.

Methodology: The Integrable Context-Dependent Demand Network (ICDN)

The authors propose the Integrable Context-Dependent Demand Network (ICDN), a "demand-first" neural architecture. Instead of modeling elasticities directly, ICDN learns log-demand as a smooth, context-conditioned function of log-prices. Elasticities are then obtained as exact, analytic derivatives of this learned surface, guaranteeing mathematical integrability (i.e., the existence of a potential function).

Core Architecture

1. Demand-First Formulation:
   The model defines log-demand $y(u, x)$ as a function of log-prices $u = \log p$ and contextual covariates $x$ (store, time, promotions, product metadata). The elasticity matrix $E$ is defined as the Jacobian of this surface: $E_{ij} = \partial y_i / \partial u_j$. By construction, if the surface is sufficiently smooth, the elasticity field is integrable.

2. Structured Price Representation:
   Rather than using a generic multilayer perceptron, ICDN decomposes the demand surface into interpretable components:

   • Linear Components: Locally linear log-log terms for own-price ($\beta_{ii} u_i$) and directed cross-price ($\beta_{ij} u_j$) effects.
   • Nonlinear Spline Refinements: Product-specific truncated-power cubic splines $B_i(u_i)$ applied to log-prices. These allow the slope and curvature of demand to vary across price regions specific to each SKU's empirical support.
   • Spline-Spline Interactions: A term $B_i(u_i)^\top U^{(ij)}(x) B_j(u_j)$ captures nonlinear relative price-positioning effects between products $i$ and $j$.

3. Contextual Conditioning and Sparsity:

   • Tokenized Encoding: Each SKU is mapped to a latent vector $h_i$ via a shared encoder processing contextual tokens (store, seasonality, lifecycle, etc.).
   • Directed Interactions: Cross-price parameters are generated from ordered pairs $(h_i, h_j)$, allowing for asymmetric effects ($E_{ij} \neq E_{ji}$).
   • Sparse Attention: To ensure scalability in high-dimensional settings, a sparse neighbor selector identifies a small set of relevant competitors for each SKU using scaled dot-product attention and metadata bonuses. This reduces the complexity of cross-price interactions from $O(n^2)$ to $O(nk)$.

4. Analytic Derivatives:
   Because the spline bases have closed-form derivatives, the model computes elasticities analytically without relying on dense numerical Jacobian or Hessian evaluations. This ensures computational tractability and transparency.

Training and Regularization

The training objective balances predictive accuracy with economic regularity:

• Robust Fit: Uses a Huber loss on log-demand to handle outliers and heteroskedasticity.
• Smoothness Penalty: Penalizes the squared second derivative (curvature) of the own-price response to prevent artificial oscillations in the demand surface.
• Elasticity-Band Penalty: Imposes soft constraints on the implied elasticities. Own-price elasticities are encouraged to be negative (within $[-5, 0]$), and cross-price elasticities are discouraged from being unusually large (within $[-1, 1]$), reflecting empirical priors for beer demand.
• Sign Constraints: The linear own-price coefficient is constrained to be non-positive via a softplus transformation, though the total elasticity can still vary locally due to nonlinear terms.

Key Contributions

1. Demand-First Principle: Formulates elasticity estimation as a problem of learning a smooth demand surface, ensuring that derived elasticities are mathematically integrable and globally consistent.
2. Structured Neural Architecture (ICDN): Introduces a model that combines explicit price bases, analytic spline derivatives, and sparse, directed cross-price interactions, making it practical for high-dimensional retail panels.
3. Analytic Tractability: Leverages closed-form derivatives of spline bases to compute elasticities efficiently, avoiding the computational bottlenecks of generic automatic differentiation on dense matrices.
4. Contextual Heterogeneity: Implements SKU-specific conditioning and sparse attention to capture heterogeneous and directional substitution patterns without imposing symmetry.

Empirical Results

The authors evaluate ICDN on the Dominick's Finer Foods scanner dataset for the beer category, comparing it against a classical directed log-log OLS benchmark.

• Predictive Performance: ICDN demonstrates superior out-of-sample generalization. Across five temporal validation folds, ICDN achieves higher $R^2$ and lower Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) in the majority of matched store-SKU-fold triplets compared to the benchmark.
• Elasticity Stability:
  • Own-Price: ICDN produces significantly more stable elasticity estimates. In bootstrap resampling, ICDN yields narrower confidence intervals (median width 1.018 vs. 2.683 for the benchmark) and lower standard deviations. It also adheres strictly to the economic prior of downward-sloping demand (100% negative estimates vs. 99.1% for the benchmark).
  • Cross-Price: While cross-price effects are inherently noisier, ICDN produces more reproducible estimates with narrower confidence intervals and lower temporal dispersion. Notably, ICDN yields predominantly positive cross-price elasticities (91% of pairs), consistent with substitution, whereas the benchmark shows a mixed signal.
• Uncertainty Diagnostics: The paper notes that while ICDN improves stability, uncertainty calibration is imperfect. Own-price intervals are slightly conservative (over-coverage), while cross-price intervals show under-coverage relative to fold-based variation, suggesting that while the estimates are more stable, they do not perfectly capture ground-truth causal uncertainty.

Significance and Claims

The paper claims that ICDN offers a principled approach to retail demand modeling that bridges the gap between the flexibility of deep learning and the structural requirements of economic theory. By treating elasticities as derivatives of a single, smooth demand potential, the model avoids the inconsistency of learning elasticities as independent local objects.

The authors emphasize that their results are based on observational data and should not be interpreted as causal identification of price interventions. Instead, the contribution lies in producing more stable, reproducible, and economically coherent elasticity estimates that are better suited for operational tasks such as scenario simulation, revenue optimization, and pricing diagnostics. The model provides a framework where business priors (e.g., downward-sloping demand) are embedded directly into the training objective, ensuring that the learned demand surface remains interpretable and robust to noise.