Technical Summary: Bayesian Surrogate Training on Multiple Data Sources

Problem Statement

Surrogate models are widely used to approximate complex, computationally expensive simulation models for tasks such as uncertainty quantification, sensitivity analysis, and solving inverse problems. However, a fundamental limitation exists: surrogate models are typically trained exclusively on simulation data. Since the underlying simulation model is often an imperfect simplification of real-world systems (missing processes, suffering from misspecifications, or having incorrect boundary conditions), the surrogate inherits these inaccuracies. Conversely, real-world measurement data contains valuable information about the true system but is often scarce, noisy, and lacks full knowledge of input parameters (e.g., unknown forcing variables) that are required by the simulation model.

Existing methods for integrating these data sources face several challenges:

1. Asymmetry: Simulation data has fully specified inputs, while real-world data often has unknown inputs.
2. Homogeneity Assumptions: Many approaches assume homogeneous data sources or rely on specific surrogate families (e.g., Gaussian Processes) with simplifying assumptions.
3. Lack of Weighting Strategies: There is limited literature on how to appropriately weight heterogeneous data sources during training to reflect varying levels of confidence or trust.
4. Indirect Integration: Some methods do not use real-world data directly for surrogate training but rather for post-hoc calibration or discrepancy modeling.

The paper proposes a general framework to address these issues by training Bayesian surrogate models on multiple data sources (simulation and real-world) using novel weighting strategies that operate independently of the chosen surrogate family.

Methodology

The authors propose a framework that combines two distinct data sources:

• Simulation Data ($D_S$): Generated from a physics-based model $M$ with known inputs $x_S$ and uncertain parameters $\omega_S$.
• Real-World Data ($D_R$): Generated from an unknown process $G$ with known inputs $x_R$ but unknown parameters $\omega_R$ and measurement noise.

The framework introduces two probabilistic approaches for weighted integration, controlled by a weighting factor $\beta \in [0, 1]$:

1. Posterior Predictive Weighting (PW)

This method trains two separate surrogate models independently:

• A simulation-based surrogate ($f_{M_S}$) trained on $D_S$ and used to infer unknown parameters from $D_R$.
• A data-driven surrogate ($f_{M_R}$) trained solely on $D_R$.

The final prediction is a weighted sum of their posterior predictive distributions:
$$p_{pw, \beta}(\hat{y}_R | \hat{x}_R, D_S, D_R) = \beta \cdot p(\hat{y}_R | \hat{x}_R, D_S, D_R, f_{M_S}) + (1 - \beta) \cdot p(\hat{y}_R | \hat{x}_R, D_R, f_{M_R})$$
Here, $\beta$ controls the relative influence of the simulation-based model. This approach allows for visualizing the discrepancy between the two sources by observing the superposition of their predictive distributions.

2. Power-Scaling of Likelihoods (PS)

This method trains a single surrogate model by directly integrating both data sources into the likelihood function via power-scaling. The joint posterior is defined as:
$$p_{ps}^\beta(\omega'_R, \sigma', c | D_S, D_R) \propto p(y_R | x_R, \omega'_R, \sigma', c)^{\alpha_R} \cdot p(y_S | x_S, \omega_S, \sigma', c)^{\alpha_S} \cdot p(\omega'_R, \sigma') p(c)$$
The scaling factors $\alpha_R$ and $\alpha_S$ are derived from $\beta$ to ensure that as $\beta$ approaches 0 or 1, the influence of one data source diminishes while the other remains fully utilized. The method employs a two-step MCMC procedure:

1. Joint Training: Estimate surrogate coefficients $c$ and intermediate parameters using the power-scaled likelihood.
2. Refined Inference: Re-estimate the unknown real-world parameters $(\omega_R, \sigma_R)$ using only the full likelihood of the real-world data $D_R$, conditioned on the posterior of $c$ from step 1. This avoids "double dipping" into the real data and ensures parameters are inferred based on real-world evidence.

Implementation

The framework is designed to be surrogate-agnostic, applicable to any probabilistic surrogate (e.g., Polynomial Chaos Expansions, Gaussian Processes, Neural Networks) implementable in probabilistic programming languages like Stan or PyMC. The authors utilize Hamiltonian Monte Carlo (HMC) for sampling.

Key Contributions

1. Novel Weighting Strategies: Introduction of two distinct methods (PW and PS) for combining heterogeneous data sources during surrogate training, offering a continuum between purely data-driven and purely simulation-based models.
2. Handling Asymmetry: The PS method specifically addresses the asymmetry of data sources (known vs. unknown inputs) through a two-step inference procedure that separates coefficient training from parameter refinement.
3. Surrogate Agnosticism: The proposed weighting strategies are independent of the specific surrogate family, allowing for broad applicability across different modeling contexts.
4. Diagnostic Capability: The framework provides a mechanism to diagnose structural limitations in simulation models by visualizing the competing information from simulation and real-world data.

Results

The authors evaluate the methods using two case studies:

Case Study 1: Synthetic Example

• Setup: A simulation model captures a global trend, while the "real" data includes an additional periodic term not present in the simulation.
• Findings:
  • Out-of-Sample (OOS): The data-driven surrogate ($\beta=0$) performs best as it fits the local data perfectly.
  • Out-of-Distribution (OOD): The simulation-based surrogate ($\beta=1$) performs best due to its ability to extrapolate based on physical laws.
  • Hybrid Performance: For intermediate $\beta$, the Power-Scaling (PS) method generally achieved lower RMSE (closer to the synthetic truth), while Posterior Predictive Weighting (PW) provided better ELPD (better uncertainty modeling).
  • Comparison with KOH: The PS method outperformed the standard Kennedy-O'Hagan (KOH) calibration framework in recovering the mean function for certain $\beta$ values, though KOH excelled in uncertainty quantification.

Case Study 2: Epidemiological Modeling (SIR Model)

• Synthetic Setup: A misspecified SIR model (fixed recovery rate) vs. real data with a different recovery rate.
  • Results: PS showed superior performance in OOD regions by leveraging simulation data as a regularizer to prevent extreme predictions from the data-driven model. PW provided a visual superposition of the conflicting trends (steeper decline in data vs. slower decline in simulation).
• Real-World Setup: COVID-19 data from Italy (first wave).
  • Results: The data-driven model failed to extrapolate (predicting infection increases), while the simulation-based model showed high uncertainty and predicted a too-strong decline. The Power-Scaling approach achieved an optimal ELPD at $\beta \approx 0.45$, balancing the two sources. The PW method showed monotonic improvement with $\beta$ but did not reach the peak performance of PS.

Significance and Claims

The paper claims that these hybrid modeling approaches offer a flexible way to improve predictive accuracy and coverage, particularly in challenging out-of-distribution regimes where single-source training fails.

• Diagnostic Insights: The authors emphasize that visualizing the posterior predictive distributions (especially via PW) serves as a diagnostic tool. It allows modelers to identify missing processes or misspecifications in the simulation model by observing where the simulation and real data diverge.
• Trade-offs: The paper modestly notes that the optimal method and weighting factor depend on the specific scenario and evaluation metric.
  • Power-Scaling is recommended for recovering the underlying truth and achieving robust mean predictions, as it uses simulation data as a regularizer during training.
  • Posterior Predictive Weighting is recommended for diagnostic purposes and intuitive interpretation of competing information sources, though it may be less robust if the data-driven component is poorly constrained.
• Future Directions: The authors suggest that while they focused on two sources, the framework can be extended to multiple sources. They also note that finding the optimal $\beta$ via line search can be computationally expensive for PS, suggesting future work on importance sampling or moment matching to reduce refitting costs.

In summary, the paper presents a framework that moves surrogate modeling from being a mere "replacement" for expensive simulations to becoming a component of a diagnostic tool for hybrid modeling, capable of leveraging both mechanistic knowledge and empirical evidence.