Adaptive Nonparametric Perturbations of Parametric Models with Generalized Bayes

1. Problem Statement

Parametric Bayesian models are powerful tools for incorporating scientific knowledge and achieving data efficiency. However, they suffer from a critical flaw: model misspecification. If the assumed parametric likelihood $p_\theta$ does not contain the true data-generating distribution $p_0$, standard Bayesian inference can yield unreliable results, as the posterior may concentrate on a "best-fit" parameter that is still fundamentally wrong.

While fully Bayesian nonparametric models offer robustness, they often lack the data efficiency of parametric models and are computationally expensive to infer, particularly when calculating Bayes factors to compare models. The paper addresses the challenge of achieving both robustness (convergence to the truth even if the model is wrong) and efficiency (fast convergence rates when the model is correct) without the prohibitive computational cost of full nonparametric inference.

2. Methodology

The authors propose a two-stage approach: first, a theoretical framework called Nonparametric Perturbations of Parametric (NPP) models, and second, a computationally tractable approximation called Generalized NPP (gNPP).

A. Nonparametric Perturbations (NPP)

The NPP model introduces a "spike-and-slab" prior over the data distribution:

1. Spike: With probability $\eta$, the data is generated from the parametric model $p_\theta$.
2. Slab: With probability $1-\eta$, the data is generated from a flexible nonparametric distribution $\tilde{p}$ (e.g., a Dirichlet Process or Polya Tree) centered at $p_\theta$.

Theoretical Properties:

• Robustness: If the parametric model is misspecified ($p_0 \notin \mathcal{M}_{pm}$), the mixing weight $\eta_n$ (posterior probability of the parametric model) converges to 0. The posterior then relies on the nonparametric component, ensuring convergence to the true functional $\psi(p_0)$.
• Efficiency: If the parametric model is well-specified ($p_0 \in \mathcal{M}_{pm}$), $\eta_n$ converges to 1. The posterior inherits the parametric convergence rate of $O(n^{-1/2})$ (Bernstein-von Mises theorem), avoiding the slower rates typical of nonparametric models.

Limitation: Computing the NPP posterior requires calculating the marginal likelihood of the nonparametric model and the Bayes factor, which is often intractable for complex nonparametric priors.

B. Generalized NPP (gNPP)

To overcome computational barriers, the authors propose the gNPP, which mimics the statistical behavior of the NPP using Generalized Bayes principles.

1. Generalized Bayes Factor (gBF): Instead of computing marginal likelihoods, the gBF uses a statistical divergence $d(p, p_0)$ (e.g., Wasserstein distance, Maximum Mean Discrepancy (MMD), or Kernelized Stein Discrepancy (KSD)) to measure the mismatch between the parametric model and the data.

   • The gBF is defined as:
     $$\text{gbf}_n := \Xi \left( \frac{E[d_n(p_\theta, p_0)]}{E[d_n(p_\theta, p_0) \mid x_{1:n}](n+1)^{-r}} \right) \frac{\eta}{1-\eta}$$
   • Here, $E[\cdot \mid x_{1:n}]$ is the posterior expectation under the parametric model. The term $(n+1)^{-r}$ is a hyperparameter controlling the rate of convergence.
   • Key Insight: This approach avoids the nonparametric model entirely during the weighting step. It only requires samples from the parametric posterior to estimate the divergence.

2. Semiparametric Perturbation: Instead of a fully nonparametric "slab," the gNPP uses a semiparametric model (e.g., Bayesian Additive Regression Trees - BART) that is consistent for the target functional $\psi(p_0)$ but computationally efficient.

3. Final Posterior: The gNPP posterior is a weighted mixture:
   $$\hat{\Pi}(\psi(p) \mid x_{1:n}) = \hat{\eta}_n \Pi_{pm}(\psi(p_\theta) \mid x_{1:n}) + (1-\hat{\eta}_n) \hat{\Pi}_{pert}(\psi(p) \mid x_{1:n})$$
   where $\hat{\eta}_n$ is derived from the gBF.

3. Key Contributions

• Theoretical Guarantees: The authors prove that the gNPP posterior is model selection consistent. It achieves the optimal parametric rate ($O(n^{-1/2})$) when the model is correct and remains robust (converging to the truth) when misspecified.
• Computational Tractability: By replacing the standard Bayes factor with a divergence-based Generalized Bayes factor, the method eliminates the need to compute marginal likelihoods for nonparametric models. It only requires the ability to sample from the parametric posterior.
• Flexibility: The framework is general and not limited to causal inference. It supports various divergence metrics (Wasserstein, MMD, KSD) and semiparametric perturbation models.
• Empirical Validation: The paper demonstrates that gNPP outperforms pure parametric models in misspecified settings and pure nonparametric models in small-data settings, while maintaining calibration.

4. Results

Synthetic Data Experiments

• Setup: The authors tested the method on Gaussian data (well-specified) and Skew-Normal data (misspecified).
• Findings:
  • Well-specified: The gNPP matched the fast convergence of the parametric model.
  • Misspecified: The gNPP successfully shifted weight to the nonparametric component, converging to the true median, whereas the parametric model failed.
  • Calibration: The gNPP maintained correct frequentist coverage (95% credible intervals contained the truth 95% of the time) in both scenarios, whereas the parametric model was miscalibrated under misspecification.
  • Hyperparameters: The rate parameter $r$ in the gBF was shown to be critical; setting $r < 1/2$ (for MMD/KSD) ensures consistency.

Real-World Application: Single-Cell RNA Sequencing (scRNAseq)

• Task: Estimating the causal effect of gene expression (e.g., FOXP3) on other genes (e.g., GZMH, GZMK) in ovarian cancer patients.
• Challenge: Parametric models often fail to capture the complex, non-Gaussian nature of gene expression data, leading to biased causal estimates.
• Results:
  • For the gene GZMH, the parametric model suggested a strong negative effect of FOXP3, but diagnostics (low $R^2$, non-normal residuals) indicated misspecification. The gNPP down-weighted the parametric model (mixing weight $\approx 0.18$) but confirmed the direction of the effect, providing a robust conclusion.
  • For GZMK, the gNPP adjusted the uncertainty, reducing the posterior probability of a positive effect compared to the parametric model, reflecting the model's inability to explain the data perfectly.
  • The method successfully handled small sample sizes (sub-sampled datasets) by leveraging the parametric model's efficiency when data was scarce, while switching to the nonparametric model as data volume increased and misspecification became apparent.

5. Significance

This paper offers a practical solution to the robustness-efficiency trade-off in Bayesian inference.

1. Bridging the Gap: It bridges the gap between the theoretical robustness of nonparametric models and the practical efficiency of parametric models.
2. Scalability: By avoiding the computation of nonparametric marginal likelihoods, it makes robust Bayesian inference feasible for high-dimensional and large-scale problems (like genomics) where full nonparametric inference is impossible.
3. Generalizability: The framework is applicable beyond causal inference to any functional estimation problem where model misspecification is a concern.
4. Scientific Impact: In fields like precision medicine, where models are often imperfect approximations of biological reality, gNPP provides a mechanism to ensure that inferences (e.g., drug targets) are reliable even when the underlying statistical model is not perfectly specified.

In summary, the authors introduce a novel Generalized Bayes framework that adaptively mixes parametric and semiparametric models based on data-driven divergence checks, achieving state-of-the-art robustness without sacrificing computational scalability.