Synthetic Priors

1. Problem Statement

The paper addresses the challenge of prior elicitation in Bayesian inference for Generalized Linear Models (GLMs), particularly logistic regression.

• The Core Difficulty: Practitioners (e.g., clinicians, engineers) naturally reason about response probabilities at specific covariate values (e.g., "What is the failure rate at 31°F?" or "What is the response rate at the maximum tolerated dose?"). However, standard Bayesian priors are placed on the log-odds coefficients ($\beta$), which are abstract, cognitively difficult to calibrate, and sensitive to reparameterization.
• The Consequence: Specifying informative priors on $\beta$ often leads to the use of "flat" (non-informative) priors, which can result in extreme extrapolations, wide credible intervals, and inefficient inference, especially in small-sample settings.

2. Methodology: Synthetic Priors

The authors propose Synthetic Priors, a framework that encodes expert knowledge as "imaginary observations" (pseudo-data) rather than abstract hyperparameters. This approach is grounded in Good's [1950] device of imaginary observations, which posits that every conjugate prior is equivalent to a likelihood function derived from pseudo-data in the same exponential family.

A. The Conditional Means Prior (BCJ Prior)

Building on Bedrick et al. [1996], the method elicits priors on the conditional mean response ($\tilde{\mu}_j = \sigma(\tilde{x}_j^\top \beta)$) at $p$ expert-chosen design points ($\tilde{x}_1, \dots, \tilde{x}_p$).

• Elicitation: Experts specify independent Beta priors for the probability of success at each design point: $\tilde{\mu}_j \sim \text{Beta}(a_j, b_j)$.
• Synthetic Data Interpretation: Through a change of variables, this induces a prior on $\beta$ that is mathematically equivalent to the likelihood of $p$ independent binomial observations:
  • At design point $\tilde{x}_j$, there are $n_j = a_j + b_j$ "imaginary" trials.
  • There are $a_j$ "imaginary" successes.
• Posterior Computation: The posterior distribution for $\beta$ is computed by combining the real data likelihood with the synthetic data likelihood. The result is a standard logistic regression problem on an augmented dataset of size $n + p$.

B. Exact Inference via Pólya-Gamma Augmentation

To handle the non-conjugacy of the logistic likelihood, the authors utilize Pólya-Gamma (PG) data augmentation [Polson et al., 2013].

• Mechanism: PG augmentation introduces latent variables $\omega_i$ to conditionally linearize the logistic likelihood, transforming the problem into a conditionally Gaussian model.
• Algorithm: The authors develop an exact conjugate Gibbs sampler:
  1. Sample latent PG variables $\omega_i$ given current $\beta$.
  2. Sample $\beta$ from a multivariate normal distribution using the augmented precision matrix.
• Key Advantage: The synthetic observations enter the algorithm identically to real observations. No Metropolis-Hastings steps or tuning parameters are required. The prior sample size $n_j$ directly controls the weight of the synthetic data in the posterior precision matrix.

3. Key Contributions

The paper makes four primary theoretical and practical contributions:

1. Theoretical Unification: It identifies Good's device as the unifying principle for synthetic priors. It demonstrates that Ridge Regression (Gaussian linear models) and the BCJ Prior (logistic models) are both instances of encoding priors as imaginary observations.
   • Ridge: Equivalent to $p$ imaginary observations at the origin with response 0.
   • BCJ: Equivalent to $p$ imaginary binomial observations at expert-chosen design points.
2. Exact Computational Framework: It establishes that combining the BCJ prior with Pólya-Gamma augmentation yields an exact Gibbs sampler. This eliminates the need for approximate inference methods (like MCMC with Metropolis steps) for logistic regression with informative priors.
3. Structural Hierarchy: The paper organizes four frameworks into a hierarchy based on how they mediate the variance–bias tradeoff via a single scalar parameter:
   • Ridge Regression (Gaussian)
   • BCJ Prior (Logistic/Expert elicitation)
   • Catalytic Priors (Huang et al., 2020; data-adaptive)
   • Prediction-Powered Inference (Angelopoulos et al., 2023; frequentist)
   • Distinction: Unlike PPI, synthetic priors do not offer frequentist validity guarantees if the expert beliefs are wrong, but they offer superior Bayesian efficiency when beliefs are accurate.
4. Empirical Validation: The approach is validated on two benchmark problems, demonstrating improved decision-making and precision.

4. Results

Case Study 1: Challenger O-Ring Data

• Context: Predicting O-ring failure probability based on launch temperature ($n=23$).
• Setup: A flat prior vs. a BCJ prior anchored at 31°F (Challenger launch temp) with a prior mean of 80% failure, and 81°F with a prior mean of 10%.
• Findings:
  • The flat prior produced an extreme posterior mean of 98.8% failure at 31°F with a very wide credible interval (89–100%).
  • The BCJ prior moderated this to 87.3% (95% CrI: 62–99%), reflecting the engineer's belief that failure is likely but not certain.
  • The BCJ prior significantly narrowed credible intervals at intermediate temperatures (e.g., at 50°F, the upper bound dropped from 100% to 87%) by constraining the slope of the dose-response curve.

Case Study 2: Phase II Atopic Dermatitis Trial

• Context: A simulated dose-finding trial ($N=300$) for an IL-inhibitor.
• Setup: A two-point BCJ prior (Placebo: 10% response; Top Dose: 30% response) vs. a flat prior.
• Findings:
  • Precision: The BCJ prior reduced the width of 95% credible intervals by 3–6% compared to the flat prior.
  • Decision Making: The probability of correctly identifying a clinically meaningful dose (response > 5% over placebo) increased by 1.2 to 2.2 percentage points for the BCJ prior.
  • Bias: Both priors exhibited similar bias due to model misspecification (fitting a linear logistic model to an Emax data-generating process), but the BCJ prior provided tighter uncertainty quantification around that bias.

5. Significance and Implications

• Bridging the Gap: The method bridges the gap between domain expertise and statistical modeling. It allows practitioners to specify priors in the language of their domain (probabilities) while maintaining rigorous Bayesian inference on the model parameters.
• Computational Efficiency: By reducing informative prior specification to data augmentation, the method leverages standard, highly optimized GLM solvers and exact Gibbs samplers, making informative Bayesian analysis accessible and computationally cheap.
• Robustness: The framework allows for easy sensitivity analysis. Practitioners can adjust the "weight" ($n_j$) of the synthetic data to see how conclusions change, moving from a flat prior ($n_j=0$) to a highly informative prior.
• Limitations: The paper acknowledges that unlike Prediction-Powered Inference (PPI), synthetic priors rely on the correctness of the expert's beliefs. If the expert's elicited probabilities are systematically wrong, the posterior will be biased. However, this is a standard trade-off in Bayesian analysis, and the framework supports robustness checks.

In summary, Synthetic Priors provide a principled, computationally exact, and practically intuitive method for incorporating expert knowledge into GLMs, unifying concepts from ridge regression, Bayesian elicitation, and modern data augmentation techniques.