Estimating the Number of Components in Finite Mixture Models via Variational Approximation

1. Problem Statement

The paper addresses the fundamental problem of model selection in Finite Mixture Models (FMMs), specifically determining the true number of components ($K^*$) when the data is generated from an over-specified model (where the candidate number of components $K > K^*$).

• The Challenge of Singularity: FMMs are "singular" statistical models because they suffer from parameter non-identifiability (e.g., label switching, merging of components). In over-specified settings, the Fisher Information Matrix (FIM) becomes non-invertible.
• Limitations of Existing Methods:
  • BIC (Bayesian Information Criterion): While consistent for regular models, BIC relies on asymptotic normality and an invertible FIM. In singular models, the standard penalty term ($-\frac{m}{2}\log n$) is incorrect, often leading to over-penalization or underestimation.
  • sBIC (singular BIC): Based on algebraic geometry (Real Log Canonical Threshold, RLCT), sBIC requires knowledge of the true parameter ordering and complex algebraic structures that are often intractable to compute for general FMMs.
  • Bayesian Methods with Strong Priors: Some approaches rely on rapidly decaying priors to force extra components to zero, but this can lead to overly small models if data is limited.
  • Likelihood-based Penalties: Methods like SCAD penalize mixing weights but may miss small clusters.

The authors propose using Variational Bayes (VB) and the Evidence Lower Bound (ELBO) derived from Mean-Field (MF) approximation as a computationally efficient and theoretically consistent alternative for model selection in singular FMMs.

2. Methodology

The core methodology involves analyzing the large-sample properties of the ELBO under the Mean-Field approximation for FMMs.

• Model Setup:
  • Data $X_n$ is i.i.d. from a true mixture $p^*(x) = \sum_{k=1}^{K^*} w^*_k g(x; \eta^*_k)$.
  • The candidate model assumes $K$ components ($K \le K_{max}$).
  • Prior: A symmetric Dirichlet prior $\text{Diri}(\phi_0, \dots, \phi_0)$ is placed on mixing weights $w$, and a smooth, positive prior is placed on component parameters $\eta$.
• Variational Inference:
  • The authors use Block Mean-Field (MF) approximation, where the variational distribution factorizes as $q(\theta, S_n) = q_\theta(\theta) \otimes q_{S_n}(S_n)$.
  • The goal is to maximize the ELBO: $L(q_{Z_n}) = \mathbb{E}_{q}[\log p(X_n, S_n, \theta)] - \mathbb{E}_{q}[\log q_{Z_n}]$.
  • The selected number of components $\hat{K}$ is the $K$ that maximizes the optimal ELBO value $\hat{L}_K$.
• Theoretical Analysis:
  • The authors establish matching upper and lower bounds for the ELBO as $n \to \infty$.
  • They utilize Laplace approximation to analyze the conditional posterior of parameters given latent indicators.
  • Crucially, they leverage the Hellinger distance and Wasserstein distance relationships to control the behavior of mixing weights in over-specified models.

3. Key Contributions

A. Theoretical Bounds on ELBO

The paper proves that for sufficiently large $n$, the ELBO satisfies:
$$L(\hat{q}_{Z_n}) = \log p^*(X_n) - \lambda \log n + O_P(1)$$
where $\lambda$ is the effective penalty coefficient. The value of $\lambda$ depends on the prior hyperparameter $\phi_0$:

1. Singular Regime ($\phi_0 < \frac{m+1}{2}$):
   $$\lambda = (K - K^*)\phi_0 + \frac{mK^* + K^* - 1}{2}$$
   Here, the penalty for extra components is proportional to $\phi_0$. If $\phi_0$ is small, the penalty is mild but sufficient to drive redundant weights to zero.
2. Regular Regime ($\phi_0 \ge \frac{m+1}{2}$):
   $$\lambda = \frac{mK + K - 1}{2}$$
   This matches the standard BIC penalty structure, where the effective dimension is the full parameter count.

B. Model Selection Consistency

The authors prove Corollary 4: Maximizing the ELBO yields a consistent estimator for the number of components ($\hat{K} \to K^*$ in probability) as $n \to \infty$, provided $\phi_0$ is chosen appropriately. This holds even without conjugate priors, relying only on smooth, positive priors.

C. Stability and "Emptying" of Redundant Components

A major by-product of the analysis is the behavior of the variational posterior under over-specification:

• Singular Regime ($\phi_0 < \frac{m+1}{2}$): The variational approximation inherits the stability of the true posterior. Redundant components ($K - K^*$) are "emptied out" (their mixing weights converge to 0) at a rate of $O(n^{-1})$ (up to logarithmic factors). This is faster than the standard parametric rate.
• Regular Regime ($\phi_0 \ge \frac{m+1}{2}$): The weights tend to spread across all $K$ components, leading to instability and slower convergence.

D. Parameter Estimation Rates

Despite the over-specification, the authors show that the parameters associated with the $K^*$ non-empty components converge at the parametric rate of $O(n^{-1/2})$ (up to logarithmic factors), even for weakly identifiable models (e.g., location-scale Gaussian mixtures). This is achieved without requiring the strong identifiability assumptions (second-order or higher) typically needed in the literature.

4. Key Results and Empirical Validation

• Simulation Studies:
  • Regime Verification: Experiments confirm the theoretical transition at the threshold $\phi_0 = (m+1)/2$. Below this threshold, the ELBO penalty slope changes linearly with $\phi_0$; above it, the slope becomes constant.
  • Weight Behavior: Box plots of mixing weights show that for $\phi_0 < 3$ (in a specific setting), redundant weights drop to near zero as $n$ increases, while for $\phi_0 > 3$, they remain distributed across all components.
  • Convergence: The Wasserstein distance between the estimated and true mixing measures decreases at the $n^{-1/2}$ rate for small $\phi_0$.
• Comparison with State-of-the-Art:
  • The ELBO-based method outperforms BIC (which underestimates in small samples) and Group-Sort-Fuse (GSF) (which struggles with weakly identifiable models and close clusters).
  • Real Data Application: Applied to Single-Cell RNA Sequencing data (Zheng et al., 2017). The ELBO method selected 10 components, correctly identifying sub-clusters within cell types (e.g., Monocytes and NK cells) that BIC merged or missed. This aligns with biological ground truth and previous clustering studies.
• Computational Efficiency: The ELBO method (optimized via Coordinate Ascent Variational Inference, CAVI) is significantly faster than Nested Sampling and comparable to BIC (EM algorithm), making it scalable for large datasets.

5. Significance

1. Bridging Theory and Practice: The paper provides the first rigorous theoretical justification for using the ELBO from Mean-Field VB for model selection in singular models, filling a gap where standard BIC fails and sBIC is computationally intractable.
2. Relaxed Assumptions: Unlike previous works that require strong identifiability conditions or specific conjugate priors, this framework works with general smooth priors and weakly identifiable models (like location-scale Gaussians).
3. Super-Efficiency: It demonstrates that variational approximations can inherit the "super-efficiency" properties of the true posterior (emptying redundant components at rate $n^{-1}$), leading to faster parameter convergence in over-specified settings.
4. Practical Guidance: The authors provide a clear heuristic for the prior hyperparameter $\phi_0$: choosing $\phi_0 < (m+1)/2$ (typically around 1) ensures stable model selection and accurate parameter estimation, while large $\phi_0$ leads to instability.

In summary, this work establishes Variational Bayes not just as a computational tool for approximation, but as a statistically consistent and efficient framework for model selection in complex, singular mixture models.