Neural Demand Estimation with Habit Formation and Rationality Constraints

This paper introduces a flexible neural demand system that integrates habit formation and enforces economic rationality constraints to improve out-of-sample prediction accuracy and welfare analysis, as demonstrated by significant gains in modeling Dominick's analgesics data compared to static models.

The Gist

Imagine you are trying to understand how a family decides what to buy at the grocery store every week. Do they switch brands because the price changed? Or do they keep buying the same brand because they've always bought it, or because they have a whole box of it in the pantry?

For decades, economists have used mathematical "maps" to predict these choices. The most famous map is called the AIDS model (Almost Ideal Demand System). Think of this map as a very rigid, old-school GPS. It's good at giving directions when the roads are straight and simple, but it gets confused when the traffic is chaotic, the roads twist, or when the driver has a habit of taking the same route every day regardless of the traffic.

This paper introduces a brand-new, AI-powered GPS for understanding shopping habits. It's called a "Neural Demand System," and it has two superpowers that the old maps lack: Habit Memory and Rationality Checks.

Here is the breakdown in simple terms:

1. The Problem: The "Static" Map vs. Real Life

Old models assume that every time you walk into a store, your brain is a blank slate. They think, "Oh, the price of ibuprofen went up, so I'll buy less of it."

But in real life, we aren't blank slates.

• Habit: If you've been buying Brand X for 10 years, you might ignore a small price hike because you're loyal.
• Stockpiling: If you bought a huge pack of ibuprofen last week, you won't buy any this week, even if the price drops.
• The Trap: Old models see you buying less and think, "Aha! The price went up, so demand dropped!" But actually, you just have a full pantry. The old map confuses habits with price sensitivity.

2. The Solution: The "Neural" GPS with a Memory

The authors built a new model using Neural Networks (the same tech behind Siri or self-driving cars). Instead of forcing the data into a rigid box, this AI learns the patterns directly from the data.

But AI can be wild and unpredictable. To keep it honest, the authors added Economic Rules (like a strict coach):

• The "Adding-Up" Rule: If you spend 100% of your budget, the shares of all items must add up to 100%. The AI is built so it cannot break this rule.
• The "Rationality" Check: The AI is penalized if it predicts weird behavior, like buying more of something when the price goes up (unless it's a weird "Giffen good," which is rare). This ensures the map makes economic sense.

3. The Secret Sauce: The "Habit Stock"

This is the paper's biggest innovation. The new model doesn't just look at today's prices. It looks at yesterday's shopping cart.

Imagine the model has a backpack (the "Habit Stock"). Every time you buy something, a little bit of that item goes into the backpack. The backpack decays over time (you use up the ibuprofen), but it remembers your recent history.

• Without the backpack: The model thinks, "Price is high, buy less."
• With the backpack: The model sees, "Price is high, BUT they just bought a huge pack last week, so they are still full. They will keep buying even if the price is high."

4. The Real-World Test: The Painkiller Aisle

The authors tested this on real data from Dominick's Finest Foods, looking at painkillers like Aspirin, Tylenol (Acetaminophen), and Advil (Ibuprofen).

What they found:

• The Old Map (Static): It thought people were very sensitive to price changes. It predicted that if Advil got 10% more expensive, people would switch to Tylenol immediately.
• The New Map (With Habits): It realized people are actually less sensitive to price because of loyalty and habits.
• The Result: When they calculated the "pain" (welfare loss) of a price hike, the new model said, "Ouch, that hurts consumers 15-16% more than the old model thought." Why? Because the old model thought people would easily switch brands to save money. The new model realized people are stuck in their habits and will pay the higher price, making the price hike much more painful for their wallets.

5. The "Placebo" Check

To make sure the AI wasn't just cheating by having "more data," they ran a trick test. They took the history data and shuffled it (like mixing up a deck of cards so the past doesn't match the present).

• Result: The model crashed. Its predictions got worse.
• Meaning: This proved that the improvement wasn't just because the AI had more numbers to look at; it was because it was actually using the correct timeline of habits.

The Big Takeaway

This paper teaches us that to understand how people spend money, we can't just look at the price tag today. We have to look at who they are and what they bought yesterday.

By giving the computer a "memory" of past habits, we get a much clearer picture of the economy. It turns out that when prices go up, people don't switch brands as easily as we thought because they are stuck in their routines. This changes how we calculate the cost of living, the impact of taxes, and how much a price hike actually hurts a family's budget.

In short: The old map said, "People will switch if prices rise." The new map says, "People are creatures of habit; they will pay the price, and that hurts them more than we thought."

Technical Summary

1. Problem Statement

The paper addresses three critical challenges in modern demand estimation:

1. High-Dimensionality and Flexibility: Classical parametric demand systems (e.g., Almost Ideal Demand System - AIDS, QUAIDS) often struggle with complex substitution patterns and high-dimensional price variations.
2. State Dependence: Static models assume period-by-period choices are conditionally independent given current prices and income. They fail to account for habit formation, stockpiling, and reference dependence, which create path dependence in consumer behavior.
3. Welfare Analysis and Integrability: Flexible machine learning models often lack the economic discipline (integrability conditions like Slutsky symmetry and monotonicity) required to compute valid welfare metrics (e.g., Compensating Variation). Without these constraints, welfare calculations may be economically incoherent.

The core objective is to develop a flexible, semi-parametric neural demand system that estimates continuous budget shares on the probability simplex, incorporates state dependence (habits), and enforces economic rationality to support accurate welfare analysis.

2. Methodology

A. The Neural Demand System

The author proposes estimating the conditional distribution of budget shares $w_t$ given an economic state $s_t$ by minimizing the Kullback-Leibler (KL) divergence between observed and predicted shares.

• Architecture: A Multi-Layer Perceptron (MLP) acts as a "preference scorer" $V_\psi(s)$, taking prices, expenditure, and state variables as inputs.
• Output Mapping: The scores are transformed into budget shares via a Softmax function:
  $$\hat{w}_i(s) = \frac{\exp(V_\psi(s)_i)}{\sum_j \exp(V_\psi(s)_j)}$$
  This construction enforces the adding-up constraint (shares sum to 1) by design, avoiding the need for complex constrained optimization.
• Training Objective: The loss function combines data fit with economic regularization:
  $$\mathcal{L}(\psi) = D_{KL}(w_{obs} || \hat{w}) + \lambda_{mono} \mathcal{L}_{mono} + \lambda_{slut} \mathcal{L}_{slut}$$
  • Monotonicity Penalty: Penalizes upward-sloping Marshallian demand (Giffen behavior).
  • Slutsky Symmetry Penalty: Penalizes violations of the symmetry condition in the Slutsky substitution matrix ($S_{ij} = S_{ji}$), which is necessary for rational utility maximization.

B. Habit Formation and State Dependence

To capture dynamic effects without committing to a parametric utility form, the model augments the state vector with a habit stock $\bar{x}_t(\delta)$:
$$\bar{x}_t(\delta) = \delta \bar{x}_{t-1}(\delta) + (1-\delta)x_{t-1}$$
where $\delta$ is a decay parameter and $x_{t-1}$ is past consumption. The neural scorer conditions on this habit stock non-parametrically, allowing the data to determine how past consumption influences current price sensitivity.

C. Handling Endogeneity and Identification

• Control Function (CF): To address price endogeneity (e.g., promotions), the model uses a control function approach. First-stage residuals of log-prices are included as inputs to the neural network. Counterfactual predictions are evaluated by setting these residuals to zero.
• Weak Identification of $\delta$: The habit decay parameter $\delta$ is not point-identified from the within-sample fit because different $\delta$ values can produce habit stocks that are observationally similar (collinear) over the data support.
  • Solution: The author uses a profile criterion (minimizing out-of-sample KL loss over a grid of $\delta$) to select $\hat{\delta}$.
  • Robustness: Instead of relying on a single point estimate, the paper reports an identified set of near-minimizers and computes welfare bounds over this set.

3. Key Contributions

1. Neural Estimator on the Simplex: A novel estimator for continuous budget allocation that uses KL minimization and Softmax mapping. It scales to high dimensions while preserving accounting identities (adding-up) by construction.
2. Theory-Disciplined Flexibility: Integrates consumer theory not as a rigid parametric form but as regularization penalties (Slutsky symmetry, monotonicity) and diagnostic checks. This allows for flexible approximation while ensuring the resulting comparative statics are economically coherent.
3. State-Dependent Demand Framework: Extends neural demand to include habit formation. It provides a practical methodology for handling the weak identification of the habit decay parameter via profiling and set-valued inference.
4. Separation of Substitution vs. Selection: Demonstrates how conditioning on state variables separates true structural substitution from persistent segmentation (state selection), a common confounder in scanner data.

4. Results

A. Simulation Evidence

The model was validated against synthetic data with known ground truth (CES, Leontief, Stone-Geary, and Habit Formation DGPs).

• Accuracy: The neural demand system recovered elasticities and Compensating Variation (CV) with near-exact accuracy in static settings, outperforming LA-AIDS, QUAIDS, and linear models, especially in non-linear environments (Leontief).
• Habit Formation: In the habit-formation DGP, static models (even flexible ones) failed to recover the true demand structure, exhibiting high out-of-sample error. Adding the habit stock reduced RMSE by ~80% and KL divergence by an order of magnitude.
• Endogeneity: The control function correction was essential for recovering causal elasticities when prices were endogenous.
• Parameter Recovery: The profiling procedure successfully identified the correct persistence regime, yielding an identified set that contained the true $\delta$ in 100% of simulation seeds.

B. Empirical Application: Dominick's Analgesics

The model was applied to weekly scanner data for OTC analgesics (Aspirin, Acetaminophen, Ibuprofen) across 93 stores.

• Predictive Performance: The habit-augmented neural model reduced out-of-sample RMSE by 33% compared to the LA-AIDS baseline and by ~27% compared to the static neural model. A "shuffled history" placebo test confirmed these gains were due to genuine temporal dependence, not just extra covariates.
• Elasticities and Substitution:
  • Static models showed a strong, upward-sloping relationship between Aspirin prices and shares (suggesting Giffen-like behavior or strong substitution).
  • The habit model revealed this was largely state-segment correlation: once the habit stock was held fixed, the Aspirin price-share gradient flattened, and the apparent substitution between Aspirin and Ibuprofen collapsed to near zero.
  • Robust substitution was found only between Acetaminophen and Ibuprofen.
• Welfare Analysis:
  • For a 10% price increase in Ibuprofen, the habit-augmented model predicted a Compensating Variation (CV) loss ~15–16% larger than the static model.
  • This indicates that ignoring habit formation leads to a significant underestimation of the welfare cost of price shocks in repeat-purchase categories.

5. Significance

This paper bridges the gap between neural econometrics and constrained semi-parametric demand analysis.

• Policy Relevance: It demonstrates that ignoring state dependence (habits) in repeat-purchase markets can lead to erroneous conclusions about substitution patterns and welfare impacts.
• Methodological Advancement: It provides a scalable, interpretable framework for estimating demand systems that are both flexible enough to fit complex data and disciplined enough to support rigorous welfare analysis.
• Robustness: By treating the habit decay parameter as a partially identified structural hyperparameter and reporting welfare bounds over an identified set, the approach offers a robust alternative to point-estimation in dynamic demand modeling.

The code and implementation are open-source, facilitating further research in neural demand estimation with economic constraints.