📊 statistics

A general framework for modeling Gaussian process with qualitative and quantitative factors

This paper proposes a general framework for modeling Gaussian processes with mixed qualitative and quantitative factors by mapping qualitative inputs to a continuous latent space, enabling the use of standard kernel functions, natural incorporation of ordinal structures, and effective model selection through Bayesian criteria and cross-validation.

Original authors: Linsui Deng, C. F. Jeff Wu

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linsui Deng, C. F. Jeff Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, but your data is a messy mix of two very different types of information:

Quantitative factors: Things you can measure with a ruler or a thermometer (e.g., "It's 75°F," "The wind is 10 mph").
Qualitative factors: Things that are just names or categories (e.g., "It's a Tuesday," "The sky is cloudy," "The season is Summer").

For a long time, computer models struggled to handle this mix. They were great at math with numbers, but terrible at understanding that "Tuesday" is different from "Wednesday" in a way that isn't just a bigger or smaller number.

This paper introduces a new, flexible framework (a set of rules) to help these computer models understand both types of data at the same time. Here is how it works, explained through simple analogies:

1. The "Secret Language" Translator (Latent Variables)

The core idea of the paper is to translate the "names" (qualitative factors) into a "secret language" of numbers that the computer can understand.

The Problem: If you tell a computer "Red" and "Blue," it doesn't know if they are similar or far apart.
The Solution: The authors imagine a hidden, continuous map (a Latent Space). They place "Red" and "Blue" on this map as dots.
- If "Red" and "Orange" are similar, their dots are close together.
- If "Red" and "Blue" are very different, their dots are far apart.
The Magic: Once the computer sees these dots, it can use standard math (Gaussian Processes) to predict what happens in between them. It's like turning a list of ingredients into a smooth gradient of flavors.

2. The "Lego" Building Blocks (Kernels)

The paper doesn't just stop at translating the names; it offers different ways to connect the dots on that map. Think of these as different types of Lego connectors (called Kernels):

The Multiplicative Connector: Imagine building a tower where every block depends on the one below it. If one block is shaky, the whole tower wobbles. This is good when factors work together tightly.
The Additive Connector: Imagine a stack of independent plates. If one plate is wobbly, the others stay steady. This is good when factors act independently.
The Shape of the Connector: The paper tests different shapes for these connectors:
- Gaussian: Smooth, curved connections (like a hill).
- Linear: Straight-line connections (like a ramp).
- Exponential: Sharp, quick connections.

The authors show that by swapping these "connectors," they can recreate many old methods and even invent new, better ones.

3. The "Ordered" Staircase (Ordinal Variables)

Sometimes, the categories aren't just names; they have an order.

Example: "Small," "Medium," "Large."
The Old Way: Computers often treated "Small" and "Large" as just two random names, ignoring that "Medium" is right in the middle.
The New Way: The paper forces the "dots" on the secret map to line up like a staircase. "Small" is at the bottom, "Medium" in the middle, and "Large" at the top. This ensures the model respects the natural order, making predictions much more accurate.

4. The "Taste Test" (Model Selection)

Since there are so many different ways to build these models (different connectors, different map sizes), how do you know which one is best?

The authors propose two methods to pick the winner:

The "Leave-One-Out" Test: Imagine you have a deck of cards. You hide one card, try to guess it using the rest, and see how close you were. You do this for every card. The model that guesses best wins.
The "BIC" Score: This is like a "value for money" score. It checks if a model is accurate without being too complicated. A model that is 99% accurate but uses 1,000 rules might lose to a model that is 95% accurate but only uses 10 rules.

They also suggest Model Averaging: Instead of picking just one winner, why not let all the good models vote? The final prediction is a weighted average of the best guesses, making the result more robust and less likely to fail.

5. Real-World Results

The authors tested their framework on real engineering problems:

Building Embankments: Predicting how a dirt road holds up under different construction speeds and materials.
Designing Materials: Figuring out how strong a new metal alloy will be based on the types of atoms mixed in.

The Verdict: Their new framework consistently outperformed older methods. It was better at handling the mix of numbers and names, especially when the "names" had an order (like "Low," "Medium," "High").

Summary

Think of this paper as a universal adapter for computer experiments. Before, you needed a specific plug for every type of data (numbers vs. names). This new framework gives you a universal socket that can handle any mix, translates everything into a common language, and provides a smart system to choose the best tool for the job. It makes computer simulations more accurate, easier to interpret, and capable of handling the messy, real-world data we actually have.

1. Problem Statement

Computer experiments often involve inputs that are a mix of quantitative factors (continuous variables) and qualitative factors (categorical variables, often called QQ inputs). Examples include engineering designs involving material types (qualitative) and dimensions (quantitative).

Challenge: Standard Gaussian Process (GP) models are designed for continuous inputs. Applying them to qualitative factors requires specialized covariance structures.
Current Limitations: Existing methods for QQ inputs are fragmented. Some use specific correlation matrices (e.g., Qian et al., 2008), others use latent variable approaches (e.g., Zhang et al., 2020), and some use specific encodings (e.g., one-hot). There is a lack of a unified general framework that ties these approaches together, handles ordinal structures naturally, and provides a systematic way to select the best covariance structure.

2. Methodology

The authors propose a general latent-variable-based framework that maps qualitative factors into a continuous latent space, allowing the use of standard kernel functions.

Core Framework

Latent Mapping: Each qualitative factor $v_j$ with $a_j$ levels is mapped to a latent vector $z^{(j)}_{v_j} \in \mathbb{R}^{l_j}$ , where $1 \le l_j \le a_j$ .
Covariance Structure: The response $Y(U, V)$ $Y (U, V)$ is modeled as a GP where the covariance between two inputs $(U, V)$ $(U, V)$ and $(U', V')$ $(U^{'}, V^{'})$ is factorized:
$\text{Cov}\{Y(U, V), Y(U', V')\} = \sigma^2 K_U(U, U') K_Z(Z_V, Z'_V)$
- $K_U$ : Kernel for quantitative factors (e.g., Gaussian).
- $K_Z$ : Kernel for the concatenated latent vectors of qualitative factors.
Kernel Flexibility: The framework allows applying various kernels ( $K_Z$ $K_{Z}$ ) to the latent space, including:
- Multiplicative Structure: $K_Z = \prod K_j$ (captures interactions between factors).
- Additive Structure: $K_Z = \sum \psi_j K_j$ (captures main effects).
- Kernel Types: Gaussian, Exponential, and Linear kernels can be applied to the latent vectors.

Handling Ordinal Variables

For ordinal qualitative variables (where levels have a natural order), the framework imposes constraints on the latent vectors to preserve this order:

Isotropic Kernels: Latent vectors are constrained such that $0 = z_1 \le z_2 \le \dots \le z_{a_j}$ .
Linear Kernels: The framework uses hyperspherical coordinates, constraining the angles $\theta$ such that $0 = \theta_1 \le \theta_2 \le \dots \le \theta_{a_j} \le \pi$ .
Optimization: These constraints are transformed into non-negative increment parameters ( $\Delta$ ), allowing efficient optimization via algorithms like L-BFGS-B.

Identifiability

The paper establishes conditions to ensure the latent parameterization is unique (up to rotation/reflection/translation):

Linear Kernel: Requires specific constraints on the angles in hyperspherical coordinates to ensure uniqueness.
Isotropic Kernel: Requires fixing the origin and ensuring specific vector components are positive to handle translation invariance.
Theorem 1: Proves that any Mercer kernel structure on latent vectors can be equivalently represented by a linear kernel with a higher-dimensional latent space (via Cholesky decomposition), though lower-dimensional specific kernels may be more efficient.

Model Selection and Averaging

Given the variety of kernel choices and latent dimensions, the authors propose:

Model Selection: Using Leave-One-Out Cross-Validation (LOOCV) and Bayesian Information Criterion (BIC) to select the best kernel and dimension.
Model Averaging: A BIC-based model averaging (BICMAvr) strategy that combines predictions from multiple candidate models weighted by their posterior probabilities, enhancing robustness.

3. Key Contributions

Unified Framework: The paper unifies existing methods (e.g., Qian et al., Zhou et al., Deng et al., Zhang et al.) as special cases of the proposed latent variable framework.
Systematic Kernel Exploration: It introduces a systematic way to test different kernel functions (Gaussian, Exponential, Linear) on latent variables, revealing that the choice of kernel significantly impacts performance depending on the data structure.
Ordinal Integration: It provides a seamless, mathematically rigorous method to incorporate ordinal information by constraining latent vectors, outperforming methods that treat ordinal variables as nominal.
Identifiability Analysis: It provides rigorous proofs for the identifiability of latent parameters under linear and isotropic kernels, addressing gaps in previous literature.
Robust Prediction Strategy: The introduction of BIC-based model averaging offers a robust alternative to single-model selection, reducing the risk of selecting a suboptimal kernel.

4. Results

The authors evaluated the framework on four simulation examples (Beam Bending, Borehole, OTL Circuit, Piston) and two real-world case studies (3D Embankment Finite Element Model, Material Design).

Performance vs. Competitors: The proposed framework generally outperformed state-of-the-art competitors (EzGP, EEzGP, ctGP) in terms of Relative Root-Mean-Squared Error (RRMSE).
Kernel Sensitivity:
- Linear Kernels often performed best for specific datasets (e.g., Beam Bending), while Gaussian kernels were superior in others.
- Exponential kernels generally yielded the poorest performance.
- Multiplicative vs. Additive: Multiplicative structures generally outperformed additive ones, except when certain qualitative factors were inactive (irrelevant), where additive structures excelled.
Ordinal Benefits: Explicitly modeling ordinal structure (e.g., GauOrd) consistently improved accuracy compared to treating ordinal variables as nominal, particularly by regularizing parameter estimation and preventing overfitting.
Model Selection/Averaging:
- LOOCV (specifically log-likelihood based) provided stable selection performance.
- BICMAvr (Model Averaging) demonstrated the most stable performance across diverse scenarios, often outperforming single-model selection (BICMSel) in complex cases like the OTL circuit and Piston models.
Dimensionality: Selecting the correct latent dimension ( $l_j$ ) is critical. Too low leads to underfitting; too high leads to overfitting and increased computational cost.

5. Significance

Practical Utility: The framework offers a "one-stop" solution for practitioners dealing with mixed inputs, removing the need to choose between disparate existing methods.
Interpretability: By mapping qualitative levels to latent vectors, the model provides insights into the similarity between different levels (e.g., visualizing latent vectors reveals which levels are close in the response space).
Theoretical Advancement: The work bridges the gap between quantitative-only GP modeling and mixed-input modeling, extending standard kernel techniques to categorical data via latent spaces.
Future Directions: The framework lays the groundwork for developing more complex non-separable kernels and handling high-dimensional qualitative factors, though the authors note computational challenges remain for very large numbers of levels.

In summary, this paper establishes a flexible, theoretically grounded, and empirically superior framework for Gaussian Process modeling with mixed qualitative and quantitative inputs, emphasizing the importance of kernel selection, latent dimension tuning, and the integration of ordinal information.