Technical Summary: Likelihood-Informed Dimension Reduction Across Tempered Bayesian Posteriors

Problem Statement

Scientific computer simulations often involve high-dimensional parameters and expensive black-box forward maps, creating a "model-data gap" that necessitates Bayesian inversion. However, the curse of dimensionality makes posterior sampling (e.g., via Markov chain Monte Carlo) computationally prohibitive, often requiring $10^5$ or more simulator evaluations. To mitigate this, modern workflows employ dimension reduction to preprocess data before training emulators or sampling.

Existing Likelihood-Informed Subspace (LIS) methods aim to identify low-dimensional subspaces that capture the information relevant to the posterior distribution. These methods typically fall into two categories:

1. Data-Free (DF-LIS): Optimizes bounds based on the prior distribution, independent of specific observations.
2. Data-Based (DB-LIS): Optimizes bounds based on the specific posterior distribution (using the observed data $y^\dagger$).

While DB-LIS is theoretically optimal for a specific observation, it is often restrictive in practice. It requires sampling from the full posterior to construct diagnostic matrices, which is the very task being solved. Furthermore, standard LIS methods often assume access to exact gradients of the forward map, which are unavailable or uninformative for chaotic, stochastic, or legacy systems. Additionally, practical applications often utilize tempered sampling algorithms (e.g., Ensemble Kalman Inversion) that generate a sequence of intermediate distributions rather than a single posterior sample, yet current LIS methods do not effectively leverage this accumulated information.

Methodology

The authors propose a generalized framework termed $\alpha$-LIS (Likelihood-Informed Subspace), which bridges the gap between DF-LIS ($\alpha=0$) and DB-LIS ($\alpha=1$) by operating on a family of $\alpha$-tempered (or annealed) posterior distributions defined as:
$$\pi^{(\alpha)}_{X|Y}(x | y^\dagger) \propto \pi_X(x) \pi_{Y|X}(y^\dagger | x)^\alpha, \quad \alpha \in [0, 1]$$

1. Theoretical Generalization ($\alpha$-LIS)

The core theoretical contribution is the derivation of diagnostic matrices that minimize an upper bound on the Kullback–Leibler (KL) divergence between the true and reduced posteriors, where the expectation is taken over a distribution $p_\alpha$ that interpolates between the prior ($\alpha=0$) and the delta distribution at the observation ($\alpha=1$).

• Input Space: The reduced input subspace is defined by the leading eigenvectors of a generalized diagnostic matrix $H^\alpha_{X|y^\dagger}$:
  $$H^\alpha_{X|y^\dagger} = \int \nabla G(x)^\top \Gamma^{-1} \left[ (1-\alpha)\Gamma + \alpha^2 (y^\dagger - G(x))(y^\dagger - G(x))^\top \right] \Gamma^{-1} \nabla G(x) \, \pi^{(\alpha)}(x | y^\dagger) \, dx$$
  This matrix interpolates non-linearly between the data-free and data-based diagnostic matrices.
• Output Space: The reduced output space is defined as the minimizer of a trace objective function $J(V_s)$, which similarly incorporates the $\alpha$-tempered distribution. Unlike the input space, this does not yield a closed-form eigenvalue problem for $\alpha > 0$, requiring numerical optimization (e.g., on the Grassmann manifold).

2. Practical Extensions

To address real-world constraints, the authors propose two key extensions:

• Accumulated Information: Instead of using samples from a single $\alpha_k$, the method aggregates diagnostic matrices (or integrates the objective function) over a sequence of tempered distributions $0 = \alpha_0 < \dots < \alpha_k$. This leverages all available samples generated by sequential tempering algorithms (like Ensemble Kalman Inversion) to build a more robust subspace, particularly when sample sizes are limited.
• Gradient-Free Approximation: Recognizing that gradients $\nabla G$ are often unavailable or noisy, the authors propose using Statistical Linearization (SL). This approximates the gradient using the empirical covariance of the input-output pairs: $\nabla G(\cdot) \approx [(C_{xx})^\dagger C_{xG}]^\top$. This allows the construction of $\alpha$-LIS subspaces without requiring explicit derivatives of the forward map.

Key Contributions

The paper outlines three primary contributions:

1. Theoretical Generalization (C1): The definition of $\alpha$-LIS, a 1-parameter family that unifies DF-LIS and DB-LIS. The authors prove that this framework generalizes existing methods and propose extensions to handle non-differentiable likelihoods and the accumulation of data from tempered sequences.
2. Numerical Assessment (C2): A broad experimental validation showing that while $\alpha=1$ is theoretically optimal, $\alpha < 1$ (specifically $\alpha \approx 0.5$) often yields near-optimal or superior performance in practical settings with limited samples, noise, or gradient approximations. The "accumulated" approach is shown to be particularly robust.
3. Framework Integration (C3): The implementation of these algorithms within the Calibrate, Emulate, Sample (CES) framework. The authors demonstrate how $\alpha$-LIS can be used to reduce dimensions before training an emulator, thereby accelerating derivative-free sampling in chaotic systems (specifically the Lorenz '96 model).

Experimental Results

The authors validate their approach across linear, linear-exponential, Darcy flow, and Lorenz '96 problems.

• Linear and Linear-Exponential Models: In settings with large sample sizes, $\alpha=0.5$ performs comparably to $\alpha=1$. However, in scenarios with moderate sample sizes or noise, $\alpha=1$ becomes numerically non-robust, while intermediate $\alpha$ values (e.g., 0.5) and the accumulated approach significantly outperform both $\alpha=0$ and $\alpha=1$.
• Noisy Evaluations and Gradient Approximations: When forward map evaluations are noisy or gradients are approximated via Statistical Linearization, the performance of $\alpha=1$ degrades sharply. The accumulated $\alpha$-LIS approach maintains robustness, often outperforming Principal Component Analysis (PCA) and standard LIS variants.
• Darcy Flow and Lorenz '96: In complex, nonlinear inverse problems, the accumulated approach using $\alpha$-LIS with statistical linearization successfully reduces dimensionality to enable effective emulator training. In the Lorenz '96 application, the method successfully recovered the forcing parameters in a chaotic regime where direct MCMC sampling is intractable.

Significance and Claims

The paper claims that $\alpha$-LIS offers a "best of both worlds" performance, balancing the theoretical optimality of data-based methods with the robustness of data-free methods. The authors emphasize that:

• Robustness over Optimality: In practical, constrained settings (limited data, noise, lack of derivatives), the theoretically optimal $\alpha=1$ is often suboptimal. Intermediate values of $\alpha$ provide a necessary regularization against Monte Carlo errors and noise.
• Utilization of Tempered Sequences: The method effectively leverages the full sequence of distributions generated by modern tempering algorithms, rather than discarding intermediate samples.
• Applicability to CES: The approach is particularly well-suited for the CES framework, where it enables the training of accurate, low-dimensional emulators using the approximate samples generated during the calibration step.

The authors conclude that while $\alpha$-LIS is not a silver bullet for all noise and approximation scenarios, it significantly improves upon existing LIS methods in challenging, realistic settings and provides a flexible tool for high-dimensional Bayesian inverse problems.