Active Learning with Adaptive Non-Stationary Kernel for Continuous-Fidelity Surrogate Models

Technical Summary: Active Learning with Adaptive Non-Stationary Kernel for Continuous-Fidelity Surrogate Models

Problem Statement
Computer simulations of complex physical processes, particularly those utilizing Finite Element Methods (FEM), are often computationally prohibitive. High-fidelity simulations offer accuracy but require substantial resources, while low-fidelity simulations are cheaper but less accurate. Traditional multi-fidelity surrogate modeling often relies on discrete fidelity levels or stationary Gaussian Process (GP) assumptions. However, in many FEM applications, fidelity is determined by continuous parameters (e.g., mesh density or time-step size) that control numerical accuracy. Existing methods struggle to efficiently integrate these continuous fidelity parameters into a statistical framework that accounts for the specific convergence behavior of numerical errors (where error vanishes as the fidelity parameter approaches zero) and the varying correlation structures of simulation increments across the fidelity space. Furthermore, static "one-shot" experimental designs often fail to optimize the trade-off between information gain and computational cost compared to sequential strategies.

Methodology
The authors propose a framework combining a novel surrogate model with an active learning (sequential design) strategy.

1. Non-Stationary GP Surrogate Model:
   The framework models the computer output $y(x, t)$ as the sum of an exact solution $\phi(x)$ and an error function $\delta(x, t)$, where $x$ represents physical input parameters and $t$ represents a continuous fidelity parameter vector. Both components are modeled as Gaussian Processes.

   • Adaptive Kernel: The core innovation is a novel adaptive non-stationary kernel function, $K_\gamma(t_1, t_2)$, inspired by fractional Brownian motion and the Lifted Brownian (LB) kriging model. This kernel is designed to capture the convergence rate of the error function as $t \to 0$. It introduces a parameter $\gamma \in (0, 1)$ to control the short-range behavior and correlation of increments along the fidelity parameter space. This allows the model to adapt to diverse sample path behaviors (e.g., positively correlated, negatively correlated, or uncorrelated increments) observed in different FEM simulations.
   • Inference: Hyper-parameters are estimated using Maximum Likelihood Estimation (MLE) with a plug-in approach, offering a computationally efficient alternative to fully Bayesian methods. The model supports extrapolation to the exact solution at $t=0$.

2. Active Learning Strategy:
   The authors develop a sequential design strategy based on the Integrated Mean Squared Prediction Error (IMSPE).

   • Cost-Adjusted Criterion: To manage computational budgets, the selection of the next design point $(x_{n+1}, t_{n+1})$ maximizes the ratio of the IMSPE reduction to the computational cost $C(t)$.
   • Closed-Form Optimization: A key technical contribution is the derivation of a closed-form expression for the IMSPE reduction and its derivatives. This enables efficient numerical optimization (using gradient-descent with multi-starts) over the continuous fidelity space, avoiding the need for expensive Monte Carlo approximations or pre-defined discrete grids used in previous works.
   • Initial Designs: The study investigates the impact of initial designs (e.g., MaxPro, Multi-Mesh Experimental Design) on hyper-parameter estimation and subsequent active learning performance.

Key Contributions

• Novel Kernel Function: Introduction of an adaptive non-stationary kernel that generalizes previous models (including the model by Tuo et al., 2014) and better reflects the behavior of computer simulation outputs across continuous fidelity parameters, specifically accommodating varying correlation structures of increments.
• Efficient Sequential Design: Development of an active learning framework that utilizes a closed-form IMSPE criterion. This allows for the optimization of design points over a continuous fidelity domain, a capability not present in existing batch or discrete-level active learning methods.
• Comprehensive Evaluation: The methodology is validated through numerical experiments and three real-world FEM case studies: Poisson's equation, stress analysis of a jet engine turbine blade, and a time-dependent wave equation.

Results

• Numerical Studies: The proposed Active Learning (AL) method with the Lifted Brownian (LBM) kernel consistently outperformed single-fidelity models and one-shot (OS) designs across various parameter settings. The LBM kernel showed particular superiority when simulation increments were positively correlated ($\gamma = 0.95$).
• Case Studies:
  • Poisson's Equation: The LBM kernel significantly outperformed other methods for the "Average" response (where increments are positively correlated) and remained highly competitive for the "Maximum" response. AL methods consistently yielded lower Root Mean Squared Error (RMSE) than one-shot designs.
  • Jet Engine Turbine Blade: The multi-fidelity approach with AL and the LBM kernel provided robust predictive performance without requiring the selection of a single, potentially suboptimal, fine-mesh size. It outperformed both single-fidelity models and stacking design methods.
  • Wave Equation: The AL method demonstrated a pronounced advantage over one-shot designs by strategically selecting low-cost fidelity points that maximized predictive improvement, achieving lower RMSE within the same computational budget.
• Initial Design Sensitivity: While the choice of initial design significantly affected the robustness of hyper-parameter estimation, the final predictive performance of the active learning method was found to be relatively insensitive to the specific initial design choice, provided a reasonable budget was allocated.

Significance
The paper claims that this framework addresses critical gaps in multi-fidelity experimental design for continuous fidelity parameters. By integrating an adaptive kernel that aligns with numerical analysis convergence rates and an efficient active learning criterion, the method offers a computationally inexpensive alternative to costly FEM simulations. The ability to extrapolate to the exact solution ($t=0$) while providing uncertainty quantification distinguishes this approach from auto-regressive multi-fidelity models. The authors position their work as a step toward more efficient uncertainty quantification in complex simulations, noting that while their method improves predictive performance, future work is needed to address multi-dimensional outputs and transition costs between mesh configurations. An R package (MuFiMeshGP) implementing the methodology is provided to facilitate reproducibility.