Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry

Technical Summary: Generalized Guarantees for Variational Inference in the Presence of Even and Elliptical Symmetry

Problem Statement
Variational Inference (VI) approximates a target density $p$ by finding the best match $q$ within a tractable family of distributions $\mathcal{Q}$, typically by minimizing a divergence $D(p||q)$. While different divergences (e.g., reverse KL, forward KL, $\alpha$-divergences) often yield different approximations, it remains unclear under what conditions VI provides provably accurate estimates of statistical properties, such as the mean and correlation matrix. Previous work established that minimizing the reverse Kullback-Leibler (KL) divergence recovers the mean and correlation matrix of a target $p$ if $p$ is log-concave and exhibits even or elliptical symmetry. However, these guarantees were limited to specific divergences and required strong assumptions (log-concavity) that exclude heavy-tailed distributions common in practice.

Methodology
The authors generalize these guarantees to a broad class of objective functions and relax the regularity conditions on the target density.

1. Objective Functions: The analysis focuses on $f$-divergences, a broad family defined as $D_f(p||q) = \int f(p(z)/q(z))q(z)dz$, which includes the reverse and forward KL divergences, $\alpha$-divergences, and smoothed total variation distances.
2. Symmetry Definitions: The paper formalizes even symmetry (symmetry about a point $\nu$), elliptical symmetry (symmetry defined by a scatter matrix $M$), and partial symmetry (symmetry along a subset of coordinates $\sigma$).
3. Variational Family: The approximation family $\mathcal{Q}$ is assumed to be an elliptical location-scale family, where distributions are elliptically symmetric about a location parameter $\nu$ with a scatter matrix proportional to a scale matrix $S$.
4. Theoretical Framework:
   • Joint Diagonalization: The authors employ a coordinate transformation (Lemma 1) that simultaneously diagonalizes the scatter matrices of $p$ and $q$ via congruence. This reformulates the $f$-divergence in terms of transformed variables, simplifying the analysis of stationary points.
   • Unimodality vs. Log-concavity: Instead of requiring $p$ to be log-concave, the authors assume $p$ and $q$ are strictly unimodal (their radial functions are strictly decreasing). This allows the inclusion of heavy-tailed distributions like the Student-$t$ and Cauchy distributions.
   • Chain Rule for Partial Symmetry: For partial symmetries, the authors utilize a chain rule property of $f$-divergences (Definition 9) to decompose the divergence into conditional and marginal components, allowing for the analysis of coordinates with and without symmetry.

Key Contributions
The paper presents three main theorems that generalize previous results:

• Theorem 1 (Mean Recovery under Even Symmetry): For any strictly convex $f$-divergence, if the target $p$ is elliptically symmetric and strictly unimodal, the unique minimizer of the divergence with respect to the location parameter $\nu$ is the true mean of $p$ ($\nu = \mu$). This holds even if $p$ is heavy-tailed and not log-concave.
• Theorem 2 (Recovery of Mean and Scatter Matrix under Elliptical Symmetry): Under the same conditions as Theorem 1, plus the requirement that the radial function of $p$ is radially decreasing in its tail, any stationary point of the divergence with respect to the location-scale parameters $(\nu, S)$ satisfies $\nu = \mu$ and $S = \gamma^2 M$ for some $\gamma > 0$. Consequently, VI recovers the correlation matrix of $p$.
• Theorem 3 (Partial Symmetry Guarantees): If $p$ exhibits symmetry only along a subset of coordinates $\sigma$ (with the conditional distribution $p(z_\sigma | z_{\bar{\sigma}})$ being elliptically symmetric with a linear conditional mean and constant scatter matrix), the variational approximation recovers the conditional mean and scatter matrix along $\sigma$. If the conditional mean is constant, the marginal mean and correlation along $\sigma$ are also recovered.

Results

• Theoretical: The proofs demonstrate that symmetry-matching solutions are stationary points for all strictly convex $f$-divergences. The authors show that for unimodal targets, these stationary points are unique minimizers for the mean and scale parameters.
• Numerical Experiments: The authors evaluated VI on synthetic targets (multivariate Student-$t$, elliptical funnel, crescent distribution) and Bayesian hierarchical models (education data, disease modeling, genetic data).
  • They measured the asymmetry of the posterior distributions and found that VI yields lower absolute errors in mean estimation for more symmetric targets.
  • In models with partial symmetry (e.g., the elliptical funnel), VI accurately recovered the mean and correlations along the symmetric coordinates ($\theta$) even when it failed to recover the mean of the asymmetric coordinate ($\tau$).
  • Experiments confirmed that parameterization strategies (e.g., non-centered parameterization or marginalization) that reduce asymmetry in hierarchical models lead to better VI performance.

Significance and Claims
The paper claims to provide a unified theoretical framework showing that symmetry-matching properties of VI are robust across a wide range of objective functions ($f$-divergences) and do not rely on the restrictive log-concavity assumption. The authors emphasize that these results offer "prescriptions for practitioners," suggesting that VI performance in Bayesian hierarchical models can be improved by reparameterizing models to exploit partial symmetries (e.g., using non-centered parameterizations or collapsed posteriors). The work extends the understanding of when VI is provably accurate, moving beyond specific divergence choices to a broader class of objectives and more realistic target distributions (including heavy-tailed ones). The authors note that while they focus on theoretical guarantees, their findings complement existing work on post-hoc diagnostics and corrections.