Objective Model Prior Probabilities in Variable Selection

1. Problem Statement

The paper addresses the critical issue of selecting objective prior probabilities for models in Bayesian variable selection. In a setting with $k$ potential variables, there are $2^k$ possible models. The goal is to assign a prior probability $Pr(M_\gamma)$ to each model $M_\gamma$ (defined by an indicator vector $\gamma$) that reflects a state of "objective" ignorance while incorporating necessary parsimony (a preference for simpler models).

The authors identify two historical "objective" defaults that fail in practice:

1. Uniform Prior: Assigns equal probability ($2^{-k}$) to every model. This leads to a prior distribution heavily concentrated on models with size $k/2$. In high-dimensional settings (e.g., genomics with $k=1,000,000$), this implies a near-certainty that half the variables are relevant, which contradicts scientific reality and lacks multiplicity control.
2. Jeffreys Prior: Assigns equal probability ($1/(k+1)$) to each model size (dimension), distributing that mass equally among models of that size. While this provides multiplicity control, it assigns equal weight to the null model ($d=0$) and the full model ($d=k$). This lack of penalization for large models often leads to the selection of overly complex models (e.g., the full model) even when data suggests otherwise, violating the principle of parsimony.

The core challenge is to find prior specifications that are objective (not arbitrarily tuned to specific data) yet sufficiently parsimonious to penalize large models without being overly restrictive.

2. Methodology

The authors evaluate eight different model prior specifications using a combination of numerical simulations and theoretical analysis.

A. The Framework

• Model Space: $M_\gamma$ represents a linear model with $k$ covariates.
• Likelihood: Standard General Linear Model with Intrinsic Priors for parameters (leading to a specific Bayes Factor formulation).
• Evaluation Metrics:
  • Posterior Inclusion Probabilities (PIP): $Pr(x_j | y)$.
  • Highest Posterior Probability Model (HPM): The single most probable model.
  • Median Probability Model (MPM): The model containing all variables with $PIP > 0.5$.
  • Parsimony: Measured by how the prior mass $M(d)$ (total probability of models of size $d$) decays as $d$ increases.

B. The Candidates

The study compares the two problematic defaults (Uniform, Jeffreys) against six alternative "objective" candidates:

1. Harmonic Prior: $Pr(M_\gamma) \propto (1+|\gamma|)^{-1} / \binom{k}{|\gamma|}$. Assigns mass to dimensions proportional to $1/(d+1)$.
2. CMG Prior: Based on a Poisson mixture with an intrinsic prior on the Poisson rate.
3. Half-p Jeffreys Prior: Assumes the inclusion probability $p \sim Uniform(0, 0.5)$.
4. Half-k Jeffreys Prior: A smoothed version of Jeffreys that caps the prior mass at size $k/2$ and distributes the remainder.
5. Beta(1, 2) Prior: Assumes $p \sim Beta(1, 2)$.
6. Hierarchical Beta Prior: A hyperprior on the Beta parameter $\alpha$ ($\pi(\alpha) \propto \alpha^{-3/2}$), resulting in a complex prior for $p$.

C. Analysis Tools

• Numerical: Used the R package BayesVarSel for exact enumeration (small $k$) and Gibbs sampling/branch-and-bound for larger $k$. A specific case study involved an infant obesity dataset ($k=47$).
• Theoretical: Analyzed the asymptotic behavior of the prior mass $M(d)$ and the prior variable inclusion probability $Pr(x_j)$ as $k \to \infty$.

3. Key Contributions

1. Systematic Critique of Defaults: The paper rigorously demonstrates why Uniform and Jeffreys priors are unsuitable for modern high-dimensional variable selection, highlighting the "full model" paradox of the Jeffreys prior.
2. Introduction of New Objective Priors: The authors formalize and analyze several alternatives (Harmonic, Half-p, Hierarchical Beta) that attempt to balance objectivity with parsimony.
3. Distinction between HPM and MPM: The study provides strong empirical evidence that the Median Probability Model (MPM) is more stable and robust across different prior choices than the Highest Posterior Probability Model (HPM). In the obesity example, the HPM varied drastically (from 8 to 47 variables) depending on the prior, while the MPM remained stable (differing by at most 2 variables).
4. Theoretical Characterization: The paper derives large-$k$ approximations for the prior mass $M(d)$ and inclusion probabilities, revealing how different priors behave asymptotically (e.g., CMG and Harmonic priors induce decreasing inclusion probabilities, while others maintain constant probabilities).

4. Results

Numerical Findings (Obesity Example, $k=47$)

• Uniform & Jeffreys: Failed to produce parsimonious results. The Jeffreys prior selected the full model (47 variables) as the HPM, and the Uniform prior suggested roughly half the variables were important.
• Harmonic Prior: Surprisingly, also selected the full model as the HPM, indicating it was not parsimonious enough for this dataset.
• CMG Prior: Extremely parsimonious; it effectively "killed off" all models with dimension $>14$, resulting in very low inclusion probabilities for most variables.
• Half-p / Half-k Priors: Enforced strict parsimony, eliminating models larger than size 34.
• Beta(1,2) & Hierarchical Beta: Performed well, selecting models with 8–12 variables. The HPMs for these priors were consistent with each other and scientifically plausible.
• Stability of MPM: Regardless of the prior (except CMG), the MPM remained remarkably stable, reinforcing the recommendation to use MPM over HPM for variable selection.

Theoretical Findings

• Prior Inclusion Probabilities:
  • Uniform/Jeffreys: $Pr(x_j) = 0.5$ (Too high).
  • Beta(1,2): $Pr(x_j) = 1/3$.
  • Hierarchical Beta: $Pr(x_j) \approx 0.22$.
  • CMG/Harmonic: $Pr(x_j)$ decreases as $O(1/k)$ or $O(1/\log k)$.
• Parsimony vs. Objectivity: The authors argue that "objectivity" in variable selection does not necessarily require a constant inclusion probability of 0.5. If the goal is to avoid overfitting, priors that induce decreasing inclusion probabilities (like Harmonic or CMG) or moderate constant probabilities (like Beta(1,2)) are more appropriate.

5. Significance and Conclusions

The paper concludes that there is no single "perfect" objective prior, but rather a trade-off based on the scientific context:

• Rejected: Uniform and Jeffreys priors are deemed untenable for variable selection due to lack of parsimony. CMG is rejected for being too parsimonious (too aggressive).
• Recommended Candidates:
  • Harmonic Prior: Good if one desires decreasing inclusion probabilities, but may lack sufficient parsimony in some cases.
  • Half-p Prior: Excellent for strict parsimony, though it may be too restrictive by eliminating large models entirely.
  • Beta(1, 2) & Hierarchical Beta: These are the most balanced choices. They offer a reasonable degree of parsimony without being overly aggressive. The Hierarchical Beta favors smaller models slightly more, while Beta(1,2) is computationally simpler.

Final Takeaway: The authors advocate for the use of Beta(1,2), Hierarchical Beta, or Half-p priors depending on the desired level of parsimony. Furthermore, they strongly suggest reporting the Median Probability Model (MPM) rather than the HPM, as the MPM provides a more robust and stable selection of variables across different prior specifications.