Technical Summary: Emulators for Large-scale Computer Experiments with Quantitative and Qualitative Inputs

Problem Statement

Computer experiments involving mixed inputs—comprising both quantitative variables and qualitative factors—are increasingly common in scientific and engineering domains, ranging from combustion and system biology to structural engineering. While statistical surrogate models (emulators) like Gaussian Processes (GPs) are standard tools for these experiments, constructing accurate and computationally efficient emulators at large scales remains a significant challenge.

Two primary hurdles exist:

1. Handling Mixed Inputs: Existing methods for mixed inputs, such as Easy-to-Interpret GPs (EzGP) and Latent Variable GPs (LvGP), often require a metric space for qualitative variables. As the number of qualitative factors or their levels increases, the number of covariance parameters grows rapidly, leading to over-parametrization and computational intractability.
2. Scalability: Conventional GP-based approaches scale as $O(n^3)$ for computation and $O(n^2)$ for memory, rendering them prohibitive for large datasets ($n$). While scalable approximations like the Vecchia approximation and localized GPs (e.g., laGP, TwinGP) exist for quantitative inputs, they have not been effectively extended to handle mixed inputs without sacrificing model flexibility or accuracy.

Methodology

The authors propose a novel, scalable framework termed SVA (Scaled Vecchia Approximation) that integrates a new covariance function with the Vecchia approximation to handle mixed inputs efficiently.

1. A New Covariance Function (SEzGP)

The core of the proposed method is a modified covariance function, SEzGP, which extends the EzGP framework.

• Structure: The model assumes an additive structure where the output $Y(x, z)$ is the sum of a global GP with quantitative inputs and local GPs conditioned on qualitative levels.
• Innovation: Unlike the standard EzGP or EEzGP, which assign distinct correlation parameters for every combination of quantitative factors and qualitative levels (leading to high dimensionality), the SEzGP model assumes that the correlation parameters for the quantitative inputs are invariant with respect to the qualitative variables.
• Formulation:
  $$K(w_i, w_j) = \sigma^2_0 \exp\left\{-\sum_{k=1}^p \theta^{(0)}_k (x_{ik} - x_{jk})^2\right\} + \sum_{h=1}^q \sum_{l_h=1}^{m_h} \mathbb{1}(z_{ih} = z_{jh} \equiv l_h) \sigma^2_h \exp\left\{-\sum_{k=1}^p \theta_k (x_{ik} - x_{jk})^2\right\}$$
  Here, the correlation parameters $\theta_k$ for the quantitative inputs are shared across all qualitative levels, significantly reducing the parameter count to $2 + 2p + q$.

2. Scaled Vecchia Approximation (SVA)

To achieve scalability, the authors adapt the scaled Vecchia approximation (Katzfuss et al., 2022) to the SEzGP model.

• Scaling: The quantitative inputs are scaled by the square root of their correlation parameters ($\sqrt{\theta_k}$), effectively transforming the input space to account for the relevance of each dimension.
• Approximation: The joint likelihood is approximated by a product of low-dimensional conditional distributions using a carefully selected conditioning set (nearest neighbors in the scaled space). This reduces computational complexity from $O(n^3)$ to approximately $O(nm^2)$, where $m \ll n$.
• Estimation: Parameters are estimated using the Fisher scoring algorithm to maximize the Vecchia likelihood.

3. Extensions of Benchmark Methods

To provide a comprehensive comparison, the authors extend two prominent large-scale GP methods to handle mixed inputs using the SEzGP covariance:

• La (Extended laGP): Adapts the Active Learning Cohn (ALC) criterion to sequentially select local design points for mixed inputs.
• Twin (Extended TwinGP): Adapts the "Twinning" approach, selecting global points independent of the prediction location and local points via nearest neighbors, utilizing one-hot encoding for qualitative variables during point selection.

Key Contributions

1. Novel Covariance Structure: Introduction of the SEzGP covariance function, which balances model flexibility with parameter parsimony by sharing quantitative correlation parameters across qualitative levels.
2. Scalable Framework: The successful integration of SEzGP with the scaled Vecchia approximation (SVA), enabling efficient prediction and full uncertainty quantification for massive datasets with mixed inputs.
3. Benchmark Extensions: Implementation of mixed-input versions of laGP and TwinGP to serve as rigorous baselines for comparison.
4. Empirical Validation: Comprehensive numerical studies and a real-world application demonstrating the superiority of the proposed approach.

Results

The paper evaluates the proposed SVA method against competitors (Twin, La, LEzGP, VA, and NN) across four numerical examples and one real-world application.

Numerical Experiments

Two scenarios were tested:

• Scenario 1: Sufficient data points per qualitative level combination.
• Scenario 2: Large number of level combinations with sparse data per combination.

Findings:

• Accuracy: SVA consistently achieved the lowest Root Mean Square Error (RMSE) across most scenarios, demonstrating superior predictive accuracy compared to non-scaled Vecchia (VA), localized methods (La, LE), and nearest-neighbor approaches (NN).
• Efficiency: While the Nearest Neighbor (NN) method was the fastest, it suffered from significantly lower accuracy. SVA offered a superior trade-off, maintaining competitive computation times (often comparable to NN) while delivering substantially better predictive performance.
• Scalability: In sparse data scenarios (Scenario 2), SVA maintained robust performance where other methods struggled with either accuracy or computational cost.

Real-World Application

A case study on beam deflection in structural engineering was conducted using synthesized data from the American Institute of Steel Construction (AISC) database.

• Setup: The model predicted maximum deflection based on continuous inputs (beam depth, area, span, load position) and qualitative inputs (material type, cross-section shape).
• Outcome: SVA outperformed all competing methods (La, Twin, NN, VA) in both predictive accuracy (log-RMSE) and computational efficiency, validating its utility in complex, physics-based engineering problems.

Significance and Claims

The paper claims that the proposed SVA framework effectively bridges the gap between scalable approximation techniques and the modeling of mixed-input computer experiments. By introducing a covariance function that reduces parameter complexity without ignoring qualitative effects, and coupling it with the Vecchia approximation, the authors demonstrate that:

1. Large-scale computer experiments with mixed inputs can be emulated with high accuracy and computational feasibility.
2. The scaling mechanism inherent in SVA is crucial for capturing the influence of both quantitative and qualitative factors effectively.
3. The method provides a practical solution for uncertainty quantification in large-scale settings where traditional GPs fail due to memory and time constraints.

The authors modestly note that while their extensions of laGP and TwinGP did not match the performance of SVA in these specific mixed-input settings, further sophisticated extensions of these methods may yet reveal potential. They also identify future research directions, including the application of the proposed covariance function to global optimization and contour estimation tasks.