Technical Summary: Bayesian Variable Selection in Generalized Linear Models

Problem Statement

Covariate selection in Generalized Linear Models (GLMs) is a critical statistical challenge. Including irrelevant predictors leads to overfitting and reduced interpretability, while omitting relevant ones results in biased estimates. While frequentist approaches (e.g., LASSO, Ridge) and existing Bayesian methods (e.g., Spike-and-Slab, continuous shrinkage priors) address this, they face specific limitations. Many Bayesian approaches rely on non-fully conjugate formulations, are restricted to linear Gaussian models, or lack theoretical guarantees regarding posterior consistency for both variable selection indicators and model parameters in the broader GLM context.

Methodology

The authors propose a fully Bayesian hierarchical and conjugate framework for covariate selection applicable to any distribution within the exponential family. The core of the methodology involves:

1. Binary Inclusion Indicators: The standard GLM is extended by introducing a binary random vector $z \in \{0, 1\}^p$, where $z_j = 1$ indicates the inclusion of the $j$-th covariate and $z_j = 0$ indicates exclusion. The linear predictor is modified to $\eta_i = x_i^\top (\beta \circ z)$, where $\circ$ denotes the element-wise product.
2. Conjugate Priors:
   • Indicators ($z$): Modeled via a Bernoulli distribution with parameters $c$, where $c$ follows a Beta prior.
   • Regression Coefficients ($\beta$): Modeled conditionally on $z$ using conjugate priors derived from the work of Chen and Ibrahim (2003). The prior for $\beta$ is adapted to depend on $z$, effectively splitting the coefficients into active ($\beta^{(1)}$) and inactive ($\beta^{(0)}$) sets.
   • Scale Parameter ($\tau$): Assigned a conjugate prior consistent with the specific exponential family member.
3. Posterior Inference:
   • The framework yields closed-form expressions for the full conditional posterior distributions of all parameters ($\beta, z, c, \tau$).
   • An efficient Gibbs Sampling algorithm is derived to sample from the joint posterior.
   • For GLMs where the conjugate posterior of $\beta$ does not correspond to a standard distribution (e.g., Poisson), the authors employ a Laplace approximation within a Sampling-Importance-Resampling (SIR) scheme to draw samples from the intractable density.

Key Contributions

1. First Fully Conjugate Bayesian Variable Selection for GLMs: The paper presents the first framework that achieves full conjugacy for variable selection across the entire exponential family, not just linear Gaussian models.
2. Theoretical Guarantees: The authors establish posterior consistency for both the inclusion indicators ($z$) and the active regression coefficients ($\beta^{(1)}$). They prove that as the sample size $n \to \infty$, the posterior probability concentrates on the true model $z^*$ and the active coefficients converge to the true parameter values $\beta^*$ with asymptotic normality.
3. Computational Efficiency: By leveraging conjugacy and a tailored Gibbs sampler (augmented with Laplace-SIR for non-Gaussian cases), the method avoids the need for exhaustive model exploration (which is infeasible for high dimensions) or complex non-conjugate MCMC schemes.
4. Implementation: A corresponding R package is developed to implement the proposed Gibbs sampling algorithm.

Empirical Results

The method was validated on both synthetic and real-world datasets across Poisson, Logistic, and Linear models.

• Synthetic Data: Experiments included scenarios with informative, noisy, and correlated covariates. The proposed method (BayesVS-GLM) demonstrated:
  • High accuracy in recovering the true sparsity pattern ($z^*$), often outperforming Maximum Likelihood Estimation (MLE) and standard Bayesian GLMs (without selection).
  • Accurate estimation of active coefficients, with Relative Mean Squared Error (RelMSE) comparable to an "Oracle" model that knows the true inclusion vector.
  • Correct behavior for inactive coefficients, where the posterior distribution remains diffuse and consistent with the prior, effectively shrinking irrelevant coefficients to zero without inducing bias in the active set.
• Real-World Data:
  • Crabs Dataset (Poisson): Successfully identified relevant predictors (width, spine condition) while discarding noise variables (random indices), achieving competitive predictive accuracy (MAE, RMSE).
  • Heart Disease Dataset (Logistic): Identified clinically relevant variables (chest pain type, exercise-induced angina) consistent with medical literature, achieving balanced accuracy and F1 scores comparable to or slightly better than baselines.
  • Pollution Dataset (Linear): Explored a large model space ($2^{15}-1$), identifying consistent variables (e.g., non-white population percentage) while revealing uncertainty in others, with predictive performance matching baseline models.

Significance and Claims

The paper claims to unify various Bayesian variable selection perspectives into a single coherent framework applicable to the exponential family. Its primary significance lies in providing theoretical guarantees of posterior consistency for variable selection in GLMs, a property previously established mainly for linear Gaussian models or specific shrinkage priors.

The authors emphasize that their approach induces a natural decomposition of the posterior into active and non-active components due to the conjugate prior structure, rather than through explicit sparsity-inducing constructions. This allows the method to capture information from observations only for truly active covariates while reverting to the prior for inactive ones. The work is presented as a valid post-model Bayesian selection procedure that balances computational efficiency with rigorous theoretical foundations.