Bayesian Variable Selection with the Quasi-Posterior

1. Problem Statement

The paper addresses a critical limitation in Bayesian Variable Selection (BVS): the reliance on the correct specification of a full likelihood function.

• The Issue: Traditional BVS requires defining a complete probability distribution $F(y|X)$ for the data. In practice, this distribution is almost always misspecified (e.g., assuming Gaussian errors when data is heavy-tailed, or assuming Poisson when data is overdispersed).
• Consequences: Misspecification leads to:
  • Reduced Power: Failure to detect truly active variables (e.g., in heavy-tailed distributions).
  • False Positives: Selecting unnecessary complex models (e.g., in the presence of inliers).
  • Inconsistency: The posterior may not converge to the true model as sample size increases.
• Current Alternatives: Existing robust approaches either use complex non-parametric models (reducing interpretability) or modify the updating rule (e.g., "tempered" posteriors) but often require tuning a "learning rate" hyperparameter, which lacks a principled method for variable selection and uncertainty quantification.

2. Methodology

The authors propose the Model Quasi-Posterior, a generalized Bayesian framework that replaces the full likelihood with a quasi-likelihood.

Core Framework

Instead of specifying a full density $f(y|x)$, the method only requires specifying the mean function $\mu(x^T\beta)$ and the variance function $V(\mu)$.

• Quasi-Likelihood: Based on the work of Wedderburn (1974) and McCullagh (1983), the log quasi-likelihood is defined as:
  $$\ell_\psi(y_i; x_i, \beta) = \int_a^{\mu(x_i^T\beta)} \frac{y_i - t}{\psi V(t)} dt$$
  where $\psi$ is a dispersion parameter.
• The Quasi-Posterior: The posterior distribution for the regression coefficients $\beta$ is constructed by combining a prior $\pi(\beta)$ with the exponential of the log quasi-likelihood:
  $$\tilde{\pi}(\beta|y, X) \propto \pi(\beta) \exp\{n Q_n(y, X; \beta, \hat{\psi})\}$$
  where $Q_n$ is the average log quasi-likelihood and $\hat{\psi}$ is a consistent estimator of the dispersion parameter derived from the data.

Model Selection Strategy

To perform variable selection, the authors define a Model Quasi-Posterior over the model space $\gamma$ (where $\gamma_j=1$ if variable $j$ is included):

1. Prior: A Spike-and-Slab prior is used, where coefficients are either exactly zero (spike) or drawn from a Normal distribution (slab).
2. Marginal Quasi-Likelihood: The marginal likelihood for a model $\gamma$ is obtained by integrating out $\beta_\gamma$.
   • Linear Regression: A closed-form solution exists (analogous to the standard Gaussian marginal likelihood but with fixed variance $\hat{\psi}$).
   • General GLMs: A Laplace approximation is used to approximate the intractable integral.
3. Inference: A random-scan Gibbs sampler is employed to explore the model space. The algorithm alternates between updating inclusion indicators $\gamma_j$ and the sparsity level $w$. A caching strategy is implemented to store Hessian matrices and modes, significantly reducing computational cost.

3. Key Contributions

1. Principled Variable Selection without Full Likelihood: The paper establishes the first instance of generalized Bayesian variable selection that provides asymptotically valid uncertainty quantification without requiring a correctly specified full likelihood. It only requires correct specification of the first two moments (mean and variance).
2. Theoretical Guarantees:
   • Consistency: Proves that the quasi-likelihood maximizer is $\sqrt{n}$-consistent.
   • Asymptotic Equivalence: Demonstrates that the model quasi-posterior shares the same asymptotic distribution as the standard Bayesian posterior when the model is correctly specified.
   • Bayes Factor Behavior: Shows that the quasi-Bayes factor behaves asymptotically like the standard Bayes factor: it converges to zero exponentially for under-specified models and penalizes over-specified models via a complexity term (similar to BIC), ensuring consistent model selection.
3. Automatic Learning Rate: Unlike other generalized Bayesian methods that require tuning a "learning rate" parameter, the quasi-posterior naturally estimates this via the dispersion parameter $\hat{\psi}$, which is derived directly from the data residuals.
4. Computational Efficiency: The derivation of closed-form marginal quasi-likelihoods for linear regression and the use of Laplace approximations with caching make the method scalable and computationally tractable.

4. Results

The authors validate the method through extensive simulations and real-world applications.

Simulation Studies

The method (QP) was compared against standard Bayesian BVS (assuming correct likelihoods) and Lasso-penalized methods under three misspecification scenarios:

1. Overdispersed Count Data:
   • Scenario: Data generated from a Gamma distribution (overdispersed) but modeled with Poisson or Negative Binomial (NB).
   • Result: QP significantly outperformed Poisson and NB models. Poisson models failed to control False Discovery Rate (FDR), while NB models suffered from low power. QP maintained high power (>90%) and low FDR (<5%).
2. Heavy-Tailed Errors:
   • Scenario: Linear regression with Student-$t$ errors ($\nu=3$) vs. Gaussian assumption.
   • Result: Standard Bayesian methods lost power to detect active variables. QP maintained superior power and F1 scores while controlling FDR.
3. Inliers:
   • Scenario: Data containing "inliers" (points with zero error), which cause standard Gaussian posteriors to shrink variance and create false positives.
   • Result: QP halved the FDR of the standard Bayesian approach (mombf) at large sample sizes without sacrificing power, effectively identifying that the inliers did not require a complex model structure.

Real Data Applications

1. NMES 1988 (Healthcare Utilization):
   • Context: Predicting physician office visits (count data) with potential overdispersion.
   • Finding: QP and NB models selected similar variables, but QP provided a better fit to the empirical mean-variance relationship (lowest MSE for variance) compared to Poisson and NB.
2. DLD Dataset (Genomics):
   • Context: Predicting gene expression levels.
   • Finding: QP identified a consistent set of active genes with the standard Gaussian Bayesian method but demonstrated robustness in small sample subsampling experiments, maintaining higher power and F1 scores when $n < 150$.

5. Significance

This paper represents a significant advancement in robust Bayesian statistics:

• Bridging the Gap: It successfully bridges the gap between the interpretability and theoretical guarantees of Bayesian methods and the robustness of quasi-likelihood approaches.
• Practical Utility: It provides a "plug-and-play" solution for variable selection in Generalized Linear Models (GLMs) where the true data distribution is unknown or complex, removing the need for practitioners to guess the correct likelihood family.
• Theoretical Foundation: By proving asymptotic validity, it legitimizes the use of quasi-likelihoods for model selection, a task previously dominated by likelihood-based or penalized frequentist methods.
• Implementation: The authors provide an R package (modelSelectionQP) implementing these methods, making the approach accessible for social science, genomics, and econometric applications.

In summary, the Model Quasi-Posterior offers a robust, theoretically sound, and computationally efficient alternative to traditional Bayesian variable selection, particularly in scenarios where model misspecification is a concern.