Conjugating Variational Inference for Large Mixed Multinomial Logit Models and Consumer Choice

1. Problem Statement

The paper addresses the computational and statistical challenges associated with estimating Mixed Multinomial Logit (MMNL) models on large-scale datasets.

• The Challenge: MMNL models are the standard for capturing unobserved heterogeneity in consumer choice (random coefficients). However, Bayesian inference for these models becomes computationally prohibitive as the number of observations ($n$), the number of groups (e.g., stores, $S$), and the dimension of random coefficients ($w$) increase.
• Limitations of Existing Methods:
  • MCMC: While accurate, Markov Chain Monte Carlo methods scale poorly with high-dimensional random coefficient vectors and large datasets.
  • Standard Variational Inference (VI):
    • Data Augmentation VI (DAVI): Assumes independence between global parameters and local random effects. This factorization is often unrealistic, leading to biased estimates and underestimation of uncertainty.
    • Amortized VI (AVI): Uses neural networks to map data to variational parameters. While scalable, it struggles with highly imbalanced group sizes and may fail to capture complex dependencies in the posterior.
• Specific Application Gap: Estimating complex variants like Nested Logit (MNestL) and Bundle Choice (B-MMNL) models with unrestricted covariance matrices for random coefficients is currently infeasible for large datasets using existing VI methods.

2. Methodology: Conjugating Variational Inference (CVI)

The authors propose a new VI method called Conjugating Variational Inference (CVI) to overcome these bottlenecks.

Core Innovation

The method constructs a Gaussian approximation to the conditional posterior of the random coefficients ($\alpha_i$) by introducing "approximate conjugacy."

1. Second-Order Taylor Expansion: Instead of using the exact log-likelihood (which is intractable for closed-form updates), CVI approximates the log-likelihood $\log p(y_i | \theta, \alpha_i)$ using a second-order Taylor expansion around an auxiliary center $a_i$ and a proxy parameter $\vartheta$.
   $$\log \tilde{p}(y_i | \vartheta, \alpha_i) \approx \text{const} + g_i^\top (\alpha_i - a_i) - \frac{1}{2}(\alpha_i - a_i)^\top H_i (\alpha_i - a_i)$$
   where $g_i$ and $H_i$ are the gradient and negative Hessian evaluated at the expansion point.
2. Conjugate Approximation: By combining this quadratic likelihood approximation with a Gaussian hierarchical prior, the conditional posterior $p(\alpha_i | y_i, \vartheta)$ becomes a multivariate normal distribution with a closed-form solution.
3. Proxy Parameter Strategy: A key distinction from prior work (e.g., Tan, 2021) is the use of a proxy parameter $\vartheta$ (updated intermittently) rather than the full global parameter vector $\theta$ for the expansion. This avoids computing expensive third-order derivatives and nested optimizations at every iteration, significantly improving scalability.

Algorithmic Details

• Variational Family: The global parameters $\theta$ are approximated using a factorized mean-field Gaussian with a low-rank plus diagonal covariance structure (to reduce parameter count). The random coefficients $\alpha_i$ are approximated using the derived Gaussian conditional.
• Optimization: The Evidence Lower Bound (ELBO) is maximized using Stochastic Gradient Descent (SGD) with the re-parameterization trick.
• Update Schedule: The auxiliary parameters ($a_i$ and $\vartheta$) are updated every $\kappa$ iterations (rather than every step) to align the Taylor expansion centers with the current variational mean, balancing convergence speed and computational cost.
• Handling Non-Concavity: For Nested Logit models, where the Hessian may not be positive definite, the method employs a spectral decomposition to project the Hessian onto the nearest symmetric positive semi-definite matrix.

3. Key Contributions

1. Novel Algorithm (CVI): Introduction of a scalable VI method that effectively handles high-dimensional random coefficients and unrestricted covariance matrices by leveraging approximate conjugacy via Taylor expansions.
2. Scalability: The method is designed to handle datasets with millions of observations and tens of thousands of random coefficient groups, outperforming existing VI methods (DAVI, AVI) in both speed and accuracy.
3. Model Flexibility: The framework is successfully applied to three distinct model specifications:
   • Standard MMNL.
   • Nested Logit (MNestL).
   • Bundle Choice (B-MMNL), which models the joint selection of complementary goods (e.g., pasta and sauce).
4. Empirical Application: The first large-scale application of mixed bundle choice models to a real-world scanner panel dataset (over 1 million transactions), demonstrating the practical utility of the method.

4. Results

Simulation Studies

The authors tested CVI against DAVI, AVI, and MCMC (ground truth for small samples) across three model types:

• Accuracy: CVI consistently achieved higher ELBO values (indicating better posterior approximation) than DAVI and AVI for MMNL and Bundle models. For Nested Logit, it performed comparably to DAVI.
• Speed: CVI was significantly faster. In large-scale simulations (1M–5M observations), CVI was 107% to 328% faster than DAVI and AVI.
• Heterogeneity Capture: CVI provided more accurate estimates of the heterogeneity (variance of random coefficients) compared to DAVI, which tended to underestimate it.

Consumer Choice Application (Pasta Data)

Using a dataset of 381 stores and ~550k–1.2M transactions:

• Predictive Accuracy: Mixed models (MMNL, MNestL, B-MMNL) significantly outperformed fixed-coefficient benchmarks. The B-MMNL model (including sauce bundles) provided the highest predictive accuracy for pasta choices, demonstrating that bundle information improves single-item prediction.
• Heterogeneity Drivers:
  • Heterogeneity in price sensitivity was driven primarily by brand preferences rather than pasta type.
  • Store Characteristics: Larger stores and stores in "Area 2" exhibited higher price elasticity (more negative). Premium stores (higher average transaction prices) exhibited lower price elasticity.
• Bundle Effects: The estimated complementary effects ($\gamma_r$) for pasta and sauce were predominantly negative. The authors attribute this to inventory dynamics (staggered restocking) rather than a lack of complementarity; households rarely buy pasta and sauce in the same trip because they deplete at different rates.

5. Significance and Implications

• Computational Breakthrough: CVI enables the estimation of complex, high-dimensional mixed logit models that were previously computationally intractable, particularly those with unrestricted covariance structures and bundle choices.
• Marketing Insights: The application reveals that store-level heterogeneity is a critical factor in consumer choice, driven by store size, location, and positioning. It also highlights the importance of modeling bundles to capture true consumer utility, even when same-trip purchases are infrequent due to inventory constraints.
• Methodological Advancement: By decoupling the need for nested optimization at every step, CVI offers a robust alternative to amortized VI for settings with imbalanced data and complex dependencies, bridging the gap between the speed of VI and the accuracy of MCMC.

In conclusion, the paper establishes Conjugating Variational Inference as a superior tool for large-scale Bayesian estimation in discrete choice modeling, offering a practical solution for researchers and practitioners dealing with massive, heterogeneous consumer choice data.