Technical Summary: Positive-definiteness in separable priors

Problem Statement

In Bayesian statistics, priors for symmetric positive-definite (PD) matrices, such as covariance, precision, or correlation matrices, are often constructed by assuming independent entries and subsequently truncating the joint distribution to the space of PD matrices. This class of priors, termed PD-separable ($p^+$), is defined as $p^+(\Theta) = p(\Theta)I(\Theta \succ 0) / c$, where $p(\Theta)$ is a separable prior with independent entries, $I(\cdot)$ is the indicator function for positive-definiteness, and $c = P_p(\Theta \succ 0)$ is the normalizing constant.

While conceptually simple and computationally convenient (facilitating posterior sampling and mode estimation via penalized likelihood methods), the authors argue that this construction suffers from significant interpretability issues. Specifically, the truncation can induce unintended consequences:

1. Marginal Distortion: The marginal distributions of the entries under $p^+$ ($\pi^+_{ij}$) may differ significantly from the untruncated marginals ($\pi_{ij}$), altering prior beliefs about variance and sparsity.
2. Shrinkage Properties: The truncation can drastically alter shrinkage behavior, potentially inflating diagonal entries or shrinking off-diagonal entries toward zero more aggressively than intended.
3. Sparse Inference Bias: In sparse settings (e.g., graphical models), the truncation probability $c_z$ depends on the sparsity pattern. This causes the induced prior on sparsity structures, $p^+(z) \propto c_z p(z)$, to systematically favor sparser models compared to the intended prior $p(z)$, potentially biasing posterior inference.

Methodology

The paper employs a combination of probabilistic bounds, random matrix theory (specifically Wigner matrix theory), and analysis of truncated distributions to characterize the behavior of $p^+$ and its marginals.

1. Characterization of Truncation Effects:

   • The authors relate the Total Variation (TV) distance and Kullback-Leibler (KL) divergence between the truncated ($p^+$) and untruncated ($p$) distributions directly to the normalizing constant $c$.
   • They derive expressions for the marginal density $\pi^+_{ij}$ in terms of the untruncated density $\pi_{ij}$ and the conditional probability $c_{ij}(\theta_{ij}) = P_p(\Theta \succ 0 | \theta_{ij})$.
   • They establish that while $c \to 1$ implies $p^+ \to p$, the marginals can remain close even if $c$ is small, provided the conditional probabilities $c_{ij}$ are relatively stable.

2. Controlling the Normalizing Constant ($c$):

   • The paper derives bounds for $c$ based on the relationship between diagonal entries ($\theta_{ii}$) and off-diagonal entries ($\theta_{ij}$). A sufficient condition for $\Theta \succ 0$ is $\theta_{ii} > \sum_{j \neq i} |\theta_{ij}|$.
   • Using Wigner matrix theory, the authors analyze the asymptotic behavior of the minimum eigenvalue $\lambda_{\min}(\Theta)$ as the dimension $k \to \infty$. They establish conditions on the ratio of the diagonal mean ($\mu$) to the off-diagonal standard deviation ($\sigma$) and the dimension $k$ that ensure $c \to 1$.
   • Specifically, for fixed diagonals, $c \to 1$ if $\mu / (\sigma \sqrt{k}) > 2$. For stochastic diagonals (e.g., Exponential), stricter conditions on $\sigma$ relative to $k$ are required.

3. Sparse Settings:

   • The authors extend the analysis to sparse matrices where $\theta_{ij} = 0$ with positive probability. They define $c_z = P_p(\Theta \succ 0 | Z=z)$, the probability of positive-definiteness given a specific sparsity pattern $z$.
   • They prove that $c_z$ is a decreasing function of the density of $z$ (number of non-zero entries) and the off-diagonal variance $\sigma^2$.
   • Using concentration inequalities and random matrix theory, they derive lower bounds for $c_z$ that depend on the maximum row degree $d_z$ of the sparsity pattern.

Key Contributions and Results

1. Quantifying the Distance Between $p$ and $p^+$

The paper provides explicit bounds showing that the TV and KL distances between $p^+$ and $p$ are functions of $c$.

• Proposition 1: $TV(p^+, p) = 1 - c$ and $KL(p^+ || p) = -\log c$.
• Proposition 4: Even if $c \to 0$ (meaning the joint distributions are far apart), the mean marginal distance can still vanish if $-\log c = o(k^2)$ (for off-diagonals) or $o(k)$ (for diagonals). This implies that careful parameter selection can preserve marginal interpretability even if the joint truncation probability is low.

2. Conditions for $c \approx 1$

The authors provide precise asymptotic conditions to ensure $c$ is close to 1, thereby making the prior interpretable:

• Fixed Diagonal: If $\theta_{ii} = \mu$, then $c \to 1$ if $\liminf_{k \to \infty} \frac{\mu}{\sigma \sqrt{k}} > 2$. If the ratio is $< 2$, $c \to 0$.
• Stochastic Diagonal: If $\theta_{ii} \sim \text{Exp}(\lambda)$, the condition is stricter: $\sigma$ must be $o(k^{-3/2})$ for $c \to 1$, compared to $o(k^{-1/2})$ for fixed diagonals.
• General Scale Families: $c$ is strictly decreasing in the off-diagonal scale parameter $\sigma$.

3. Impact on Sparse Inference

The paper demonstrates that in sparse graphical models, the truncation introduces a bias toward sparser structures:

• Proposition 7: The expected minimum eigenvalue $\lambda_{\min}(\Theta)$ decreases as the sparsity pattern $z$ becomes denser. Consequently, $c_z$ decreases as the number of non-zero entries increases.
• Corollary 3 & Theorem 3: The authors derive lower bounds for $c_z$ involving the maximum degree $d_z$. For fixed diagonals, $c_z \ge 1 - 2k e^{-\mu^2 / (2\sigma^2 d_z)}$.
• Implication: If the prior variance $\sigma^2$ is not set carefully (specifically, if $\sigma$ is too large relative to $\mu$ and $d_z$), $c_z$ becomes small for dense structures. This causes the induced prior $p^+(z) \propto c_z p(z)$ to assign significantly higher mass to sparse models than the original prior $p(z)$, potentially leading to over-selection of sparse graphs in posterior inference.

4. Parameter Selection Guidelines

The paper offers practical guidelines for setting prior hyperparameters to mitigate these effects:

• Variance Control: The off-diagonal variance $\sigma^2$ must be sufficiently small relative to the diagonal mean $\mu$ and the matrix dimension (or sparsity degree).
• Diagonal Distribution: Using diagonal distributions with lighter tails near zero (e.g., Gamma or Inverse-Gamma) rather than Exponential can relax the constraints on $\sigma$ required to keep $c \approx 1$.
• Re-parameterization: The authors suggest parameterizing $\Theta = D^{1/2} \tilde{\Theta} D^{1/2}$ and placing a PD-separable prior on $\tilde{\Theta}$ (containing partial correlations), which may offer better interpretability and scale invariance.

Significance and Claims

The paper claims that while PD-separable priors are popular for their computational convenience, their interpretability is often compromised unless parameters are set with rigorous attention to the matrix dimension and sparsity structure.

• Interpretability: The authors argue that without ensuring $c \approx 1$ (or that marginals remain close), the hyperparameters of the prior do not reflect the intended beliefs about the data-generating process (e.g., variance or sparsity proportion).
• Sparse Learning: A critical contribution is the demonstration that truncation systematically biases sparse inference. The paper shows that unless $\sigma$ is scaled appropriately with $k$ and the sparsity level, the posterior will be driven by the truncation artifact rather than the data or the intended prior $p(z)$.
• Practical Guidance: The paper provides the first rigorous asymptotic conditions (involving Wigner matrix limits) for setting the off-diagonal variance in high-dimensional sparse settings to ensure the prior behaves as intended. It suggests that standard choices (like fixed $\sigma$ or Exponential diagonals) may lead to pathological behavior in high dimensions and proposes specific scaling laws (e.g., $\sigma = o(k^{-1/2})$ for fixed diagonals) to avoid them.

The authors conclude that careful parameterization is essential to ensure that the "separable" assumption remains a valid approximation of the prior beliefs after the positive-definiteness constraint is enforced.