Identification and Estimation of Demand Models with Endogenous Product Entry and Exit

This paper proposes a new semiparametric two-step estimator that utilizes latent propensity scores to correct for endogenous product entry and exit bias in demand models, demonstrating that conventional correction methods substantially underestimate price elasticities in the airline industry.

The Gist

Imagine you are trying to figure out how much people love a specific brand of ice cream. You go to the grocery store, look at the shelves, and see that Brand A is there, but Brand B is missing. You ask the store manager, "Why isn't Brand B here?"

The manager says, "Well, we only stock brands that we think will sell really well in this neighborhood."

Here is the problem: The manager knows things you don't. They know that the neighborhood has a lot of families with kids who love Brand B, but you (the researcher) only see the empty shelf. If you try to calculate how much people love Brand B just by looking at the stores where it is available, you will get the wrong answer. You might think, "Oh, people only buy it in rich neighborhoods," when really, they would buy it everywhere, but the manager just didn't stock it in the poor neighborhoods because they thought it wouldn't sell.

This is the core problem of Endogenous Selection Bias. Firms (like airlines or ice cream companies) make smart decisions about where to sell based on secret information about demand. If you ignore this, your math is broken.

The Old Ways of Fixing It (And Why They Failed)

For a long time, economists tried to fix this with two main strategies, both of which had big flaws:

1. The "Naive" Guess: They assumed companies were blind. They assumed companies didn't know anything about future demand when they decided to enter a market.
   • The Metaphor: It's like assuming the ice cream manager picks flavors by throwing darts at a board. We know this isn't true; managers are smart.
2. The "Super-Computer" Model: They built a massive, complex simulation of the entire economy, guessing exactly how companies compete, what their costs are, and what they know.
   • The Metaphor: This is like trying to solve a Rubik's cube by building a robot that simulates every possible move in the universe. It works, but it's incredibly heavy, slow, and if you guess the robot's rules wrong, the whole thing collapses.

The New Solution: The "Secret Club" Analogy

The authors of this paper (Aguirregabiria, Iaria, and Sokullu) came up with a clever, simpler way to fix the math without needing to guess the companies' secret rules.

Imagine that every city has a hidden "vibe" or "personality" that we can't see. Let's call these Secret Clubs.

• Club A loves spicy food.
• Club B loves sweet food.
• Club C is undecided.

When a company decides to open a store, they are looking at the city and thinking, "Is this a Club A city? If so, I'll open a spicy restaurant."

The Problem: We can't see the clubs. We only see the restaurants that opened.
The Old Mistake: We assumed all cities were the same (or that companies were blind).
The New Insight: The authors realized that if we look at patterns of who opens stores where, we can figure out the existence of these Secret Clubs.

If Pizza Hut and Taco Bell always open together in the same cities, but never open alone, it suggests there is a "Secret Club" of cities that loves both. If they open separately, it suggests different clubs.

How Their Method Works (The Two-Step Dance)

The authors propose a Two-Step Semiparametric Estimator. Think of it like a detective solving a mystery in two phases.

Step 1: The Pattern Hunter (Finding the Secret Clubs)

Instead of guessing what the companies know, the researchers look at the patterns of entry across thousands of markets.

• They ask: "When does Airline X enter a route? When does Airline Y enter?"
• They use a statistical trick called a Finite Mixture Model. This is like saying, "Okay, let's pretend there are 3 types of hidden market vibes (Secret Clubs). Let's see if the data fits better with 1 type, 2 types, or 3 types."
• By looking at how airlines' decisions correlate (e.g., "Southwest enters when Delta enters, but United doesn't"), they can mathematically reconstruct the probability that a market belongs to a specific Secret Club, even though they can't see the club directly.

They call this the Latent Propensity Score. It's a fancy way of saying: "The hidden likelihood that a company enters, based on the secret vibes of the market."

Step 2: The Correction (Fixing the Ice Cream Math)

Now that they have estimated these "Secret Clubs" and the probabilities of entry, they go back to the demand equation (the ice cream math).

• They add a "correction term" to their formula. This term accounts for the fact that the company only entered because they knew the market was a "Secret Club" that loved their product.
• This allows them to strip away the bias. They can now say, "Okay, we know the manager only stocked Brand B in rich neighborhoods because they knew rich people loved it. Let's adjust the numbers to see how much everyone would have loved it."

Why This Matters: The Airline Example

The authors tested this on US Airlines.

• The Old Way: When they used standard methods, they thought airlines were very sensitive to price. They thought, "If we raise the price by $1, demand drops a little bit."
• The New Way: When they used their "Secret Club" method, they found the opposite. The demand was actually much more sensitive to price.
  • The Result: The old methods were underestimating how much people hate high prices. They thought airlines had more power to raise prices than they actually did.
  • The "Lerner Index" (a measure of market power): The old method said airlines had huge power (68%). The new method said their power was much lower (15%).

The Big Takeaway

This paper is like giving economists a new pair of glasses.

• Before: They looked at the market and saw only the visible products, assuming the missing ones were just "not wanted."
• Now: They can see the invisible patterns of why products are missing. They realize that missing products aren't random; they are the result of smart companies hiding their secrets.

By using this new method, we get a much truer picture of how much people actually value products, how much power companies really have, and what would happen if we changed the rules (like in a merger). It's a smarter, more flexible way to understand the hidden logic of the market.

Technical Summary

1. The Problem: Endogenous Selection Bias in Demand Estimation

The paper addresses a fundamental econometric challenge in estimating demand systems for differentiated products: endogenous product entry and exit.

• The Issue: Firms strategically introduce products in markets where they anticipate higher demand, utilizing private information about unobserved demand shocks ($\xi_{jt}$) that researchers do not observe. Consequently, the observed sample of products is non-random.
• The Consequence: Standard regression-based demand estimation suffers from selection bias because the error term ($\xi_{jt}$) is correlated with the product availability indicator ($a_{jt}$).
• Why Standard Methods Fail:
  • Propensity Score Failure: In oligopoly settings, entry decisions depend on the equilibrium profitability of all competing products. The unobserved demand shocks of rivals ($\xi_{-j}$) affect firm $j$'s entry decision. This creates a multi-dimensional selection problem where the standard propensity score (probability of entry conditional on observables) is not a sufficient statistic for selection bias. The monotonicity conditions required for standard control function approaches (e.g., Heckman, Das et al.) are violated.
  • Joint Estimation Limitations: Existing solutions often involve jointly estimating the full structural model (demand, pricing, and entry). While consistent, these methods rely heavily on strong parametric assumptions regarding supply-side structures (cost functions, competition nature) and distributional assumptions, making demand identification fragile to supply-side misspecification.

2. Methodology: A Semiparametric Two-Step Approach

The authors propose a new semiparametric two-step estimator that corrects for selection bias without imposing strong supply-side assumptions. The core innovation is the use of latent propensity scores.

A. Theoretical Framework

• Model: The demand system follows the standard Berry-Levinsohn-Pakes (BLP) framework. The entry game is modeled as a Bayesian Nash Equilibrium (BNE) where firms possess private information about entry costs and potentially common knowledge about demand/cost fundamentals ($\kappa_t$).
• Latent Propensity Scores: The authors define the latent propensity score $P_j(x_t, \kappa_t)$ as the probability of entry conditional on both observables ($x_t$) and the unobserved common state ($\kappa_t$).
• Key Insight (Proposition 3): The selection bias term in the demand equation, $\lambda_j(x_t) = E[\xi_{jt} | a_{jt}=1, x_t]$, can be expressed as a convolution (mixture) of these latent propensity scores and the conditional mean of unobservables given the latent state:
  $$\lambda_j(x_t) = \int \left[ \frac{P_j(x_t, \kappa)}{P_j(x_t)} \right] \mu_j(\kappa) f_\kappa(\kappa) d\kappa$$
  where $\mu_j(\kappa) = E[\xi_{jt} | \kappa_t = \kappa]$.

B. Identification Strategy

The identification proceeds in two steps using a sieve approximation (finite mixture model):

1. Step 1: Estimating Latent Types and Probabilities.
   • The unobserved heterogeneity $\kappa_t$ is approximated by a finite set of $L$ discrete "latent market types."
   • The joint distribution of entry decisions across all products is modeled as a nonparametric finite mixture.
   • Using the cross-product correlation in entry decisions, the model nonparametrically identifies the distribution of latent types ($f_\kappa$) and the type-specific entry probabilities ($P_{j,\ell}$).
   • This step exploits the fact that while individual entry decisions are noisy, the joint pattern of entry across products reveals the underlying latent market structure.
2. Step 2: Demand Estimation with Control Functions.
   • The selection bias term is approximated using the estimates from Step 1. It becomes a linear combination of constructed regressors (control functions) based on the latent propensity scores.
   • The demand parameters are estimated via Generalized Method of Moments (GMM), controlling for both the endogenous product availability (via the control function) and price endogeneity (via standard BLP instruments).

C. Estimation Algorithm

• First Step: Maximum Likelihood Estimation (MLE) using the Expectation-Maximization (EM) algorithm to estimate the mixture weights and type-specific logit parameters.
• Second Step: GMM estimation of demand parameters. Standard errors are corrected using a linearized bootstrap procedure that accounts for the first-step estimation error without re-running the computationally expensive EM algorithm in every bootstrap iteration.

3. Key Contributions

1. New Identification Conditions: The paper establishes that demand parameters are identified in the presence of endogenous entry even when standard monotonicity conditions fail. It proves that the selection bias is identifiable as a mixture of latent propensity scores.
2. Decoupling Supply and Demand: Unlike joint structural estimation, this method identifies demand parameters independently of supply-side assumptions (e.g., cost functions, competition mode). It only requires assumptions about the information structure at the time of entry.
3. Handling Non-Monotonicity: It provides a solution for selection problems where the selection rule is non-monotonic due to strategic interdependence among firms (oligopoly effects), extending the literature on sample selection bias.
4. Computational Tractability: The two-step sieve approach avoids the need to solve for equilibrium outcomes in every iteration of the estimation, significantly reducing computational complexity compared to full structural estimation.

4. Empirical Results: US Airline Industry

The authors apply the method to US airline route data (2012–2013), comparing their "Finite Mixture" estimator against OLS, standard 2SLS (ignoring selection), and standard semiparametric control function methods (assuming $L=1$).

• Elasticity Estimates:
  • OLS: Severely biased (Own-price elasticity $\approx -1.6$).
  • Standard 2SLS (No Selection Correction): Still biased (Elasticity $\approx -5.5$).
  • Standard Semiparametric ($L=1$): Similar to 2SLS, failing to capture the full selection bias.
  • Proposed Method ($L=3$): Reveals significantly higher price elasticities (Elasticity $\approx -7.6$).
• Lerner Index: The implied market power (Lerner Index) drops drastically from ~69% (OLS) to ~15% (Proposed Method), suggesting that previous studies significantly overestimated market power by ignoring endogenous entry.
• Marginal Costs: The method also corrects selection bias in marginal cost estimation, though the authors find that in this specific dataset, entry decisions are driven more by demand-side unobservables than cost-side unobservables.
• Model Fit: The finite mixture model with $L=3$ latent types provides a significantly better fit to the joint entry distribution than models with fewer types, and the results stabilize at $L=3$ or $L=4$.

5. Significance and Implications

• Policy Relevance: The finding that demand is more elastic and market power is lower than previously thought has profound implications for antitrust policy and merger analysis. Standard models may overstate the anti-competitive effects of mergers because they underestimate consumer sensitivity to price.
• Robustness: The method offers a robust alternative to structural models that are sensitive to misspecification of supply-side parameters. It allows researchers to estimate demand consistently even when the exact nature of competition or cost structures is unknown.
• Generalizability: The framework is applicable to other industries with endogenous product availability (e.g., supermarkets, software, media) and can be extended to dynamic entry/exit games and models with two-sided matching.

In summary, this paper provides a rigorous, computationally feasible, and assumption-light solution to the long-standing problem of endogenous product selection in demand estimation, fundamentally altering the empirical understanding of market power and demand elasticity in differentiated product markets.