📊 statistics

Gradient-enhancement and Gradient Predictions for Deep Gaussian Process Modeling of Expensive Computer Experiments

This paper proposes a novel Bayesian framework for Deep Gaussian Processes that integrates gradient information to enhance surrogate modeling and enable gradient predictions for expensive, nonstationary computer experiments, demonstrating superior performance over existing methods on both synthetic and real-world quantum mechanics datasets.

Original authors: Annie S. Booth

Published 2026-07-24

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Annie S. Booth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to learn the layout of a massive, shifting maze. You can't see the whole thing at once; you can only take a few steps, look around, and guess where the walls are. In the world of computer science, this is what scientists do when they try to understand complex simulations, like how a jet engine ignites or how a molecule vibrates. These simulations are like "black boxes": you put numbers in, and they spit out results, but running them takes so much time and power that you can only afford to peek inside a handful of times. To save time, scientists build "surrogates"—smart, fast mathematical shortcuts that guess what the big, slow machine would say if you asked it a question it hasn't answered yet.

Usually, these shortcuts work by drawing a smooth map based on the few points they've seen. But real-world problems are often messy and "nonstationary," meaning the rules of the maze change as you move. A smooth map might work in the flat hallway, but it fails miserably when the floor suddenly turns into a steep cliff. To fix this, scientists use a trick called a "Deep Gaussian Process" (DGP), which is like having a flexible, stretchy rubber sheet that warps the maze to make the tricky parts look flat and easy to draw. However, even with this stretchy sheet, if you only have a few points to look at, your guess might still be a bit wobbly.

This is where the paper by Annie S. Booth steps in with a clever upgrade. The author suggests that instead of just asking the computer, "What is the result here?", we should also ask, "How fast is the result changing right here?" In physics and engineering, computers can often tell you not just the answer, but also the "gradient"—the direction and speed of the slope. Think of it like a hiker who doesn't just know the altitude of a peak but also knows exactly how steep the path is under their boots. By feeding this extra "steepness" information into the stretchy rubber sheet, the paper shows we can build a much sharper, more accurate map, even when we have very few data points. The paper proves that this new method, which combines the flexible rubber sheet with the steepness clues, beats the old methods at predicting both the answers and the slopes, especially for tricky, changing landscapes.

The Paper's Story: Stretching the Map with Steepness Clues

The core of this work is about making better guesses for expensive computer experiments. The author, Annie S. Booth, proposes a new way to build these "surrogate models" by teaching them to use two types of superpowers at once: the ability to stretch the map (using Deep Gaussian Processes) and the ability to see the slope (using gradients).

The Problem with Old Maps
Imagine trying to draw a map of a mountain range using only five dots. If you use a standard, rigid map (a traditional Gaussian Process), you might get the general shape, but you'll likely miss the sharp cliffs or the flat valleys because the map tries to be smooth everywhere. If the mountain has a sudden drop-off (a nonstationary feature), a rigid map will either smooth it out too much or get confused.

To fix this, scientists started using "Deep Gaussian Processes" (DGPs). You can think of a DGP as a two-layered magic trick. The first layer takes your messy, real-world input (like the position of atoms in a molecule) and "warps" or stretches it into a new, cleaner space where the rules are simpler. The second layer then draws a smooth map on this new, stretched space. It's like taking a crumpled piece of paper, ironing it out flat, and then drawing your map on the smooth surface. This works great for complex shapes, but it still struggles when data is super scarce.

The New Superpower: Gradient-Enhancement
The paper introduces a game-changer: Gradient-Enhancement. In many physics simulations, the computer doesn't just give you a number (like "energy"); it also gives you the derivative (like "force"). In everyday terms, if the energy is the height of the hill, the force is the steepness of the hill at that exact spot.

The author realized that if you can tell the model not just where you are, but how steep the ground is, you can learn the shape of the mountain much faster. However, adding this steepness info to a simple map is easy, but adding it to a "stretchy" DGP map is incredibly hard. The math gets messy because you have to figure out how the stretching itself is changing the steepness.

The Solution: A Mathematical Chain Reaction
Booth's paper solves this by creating a new framework that treats the whole system as a chain reaction.

The Inner Layer: The model figures out how the input is being stretched (the warping).
The Outer Layer: The model figures out the result on that stretched map.
The Chain Rule: Using a classic math rule called the "multivariate chain rule," the model connects the dots. It calculates how the steepness of the final result is a combination of how steep the stretch is and how steep the result is on the stretched map.

By doing this, the model can use the observed "steepness" data to train the stretching layer itself. It's like if the hiker told the map-maker, "The ground is getting steeper here," and the map-maker used that clue to decide exactly how much to stretch the paper to make the hill look flat.

What the Paper Found
The author tested this new "Gradient-Enhanced DGP" (geDGP) against older methods using several tricky test cases:

The "Step" Function: A graph that stays flat, then suddenly jumps up. The old models struggled to find the exact spot of the jump. The new geDGP nailed it.
The "Squiggle" and "Plateau" Functions: These are wavy, bumpy surfaces with flat areas and steep drops. In simulations with only 25 to 30 data points, the geDGP was significantly more accurate than both the standard DGP and the gradient-enhanced simple map.
Real-World Quantum Mechanics: The author applied this to real data from the SPICE dataset, which simulates the energy and forces of molecules (like potassium bromide and sodium iodide). These are complex, nonstationary systems. The geDGP outperformed all other methods in predicting both the energy and the forces, even with very few observations (as few as 11 for some molecules).

The Speed Bump and the Shortcut
There is a catch. Adding all this gradient information makes the math much heavier. If you have 100 data points and 5 dimensions, the computer has to do calculations for 600 points (100 results + 500 gradients). This can slow things down to a crawl.

To fix this, the paper introduces an optional "Vecchia approximation." Think of this as a smart shortcut. Instead of trying to compare every single point to every other point (which is slow), the model only looks at the nearest neighbors. This speeds up the process dramatically without losing much accuracy, making the method usable even for larger datasets.

The Verdict
The paper concludes that combining the flexibility of Deep Gaussian Processes with the extra information from gradients creates a superior tool for expensive computer experiments. It suggests that for problems where data is scarce and the behavior is complex (like molecular simulations), this new method offers the best accuracy and the most reliable uncertainty estimates. The author has even released the code as an open-source package called deepgp, so others can use this "stretchy map with steepness clues" to solve their own tough problems.

Technical Summary: Gradient-Enhancement and Gradient Predictions for Deep Gaussian Process Modeling

Problem Statement
Deep Gaussian Processes (DGPs) have emerged as powerful surrogate models for complex, nonstationary computer experiments, offering greater flexibility than traditional stationary Gaussian Processes (GPs) by warping the input space through latent layers. However, standard DGPs face two primary limitations in the context of expensive, deterministic simulations:

Underutilization of Gradient Data: Many physics and engineering simulations (e.g., via adjoint solvers) provide gradient information alongside function values. While gradient-enhanced GPs (geGPs) are well-established, integrating gradients into the DGP framework remains unexplored.
Lack of Gradient Predictions: Downstream tasks such as active learning and Bayesian optimization often require gradient predictions at unobserved inputs. While GP gradient predictions are straightforward, obtaining them for DGPs is challenging due to the functional nature of the latent warping and the complexity of posterior inference.

Furthermore, incorporating gradients exacerbates existing computational challenges, including ill-conditioned covariance matrices and the cubic computational cost ( $O(n^3)$ ) of matrix inversions, which become even more severe when gradients increase the effective data dimension.

Methodology
The authors propose a comprehensive Bayesian framework for DGPs that integrates gradients on every Gaussian layer. The core methodology involves:

Model Formulation: The DGP is defined as a functional composition where an inner GP layer warps the input space $X$ to a latent space $W$ , and an outer GP layer maps $W$ to the response $y$ . The authors upgrade this prior to include gradients on both layers. The inner layer models the latent nodes $w_d$ and their partial derivatives $\partial w_d / \partial x$ , while the outer layer models the response $y$ and its derivatives with respect to the latent space $\partial y / \partial w$ .
Chain Rule Integration: The gradient of the response with respect to the original inputs ( $\nabla_x y$ ) is not modeled directly but is derived via the multivariate chain rule: $\nabla_x y = (\nabla_x W) \cdot (\nabla_w y)$ . This allows the model to combine the inferred gradients of the latent warping and the outer regression function.
Inference via Elliptical Slice Sampling (ESS): To handle the intractable posterior of the latent warping, the authors employ ESS. They extend the ESS algorithm to sample the latent nodes and their gradients simultaneously ( $W_{all}$ ). This involves solving a linear system at each iteration to update the outer layer's gradient observations based on the proposed latent gradients.
Gradient-Enhanced Training: When training data includes observed gradients ( $\nabla_x y$ ), the likelihood function is modified to condition on these observations. The ESS acceptance ratio utilizes a likelihood that incorporates both the response and the gradient data, effectively treating the gradient observations as additional constraints on the latent warping.
Vecchia Approximation: To address computational bottlenecks caused by the increased dimensionality of gradient-enhanced covariance matrices (size $N \times N$ where $N = n + nD$), the authors implement an optional Vecchia approximation. They propose a specific ordering strategy where response observations are ordered first, followed by their corresponding gradients, with conditioning sets selected based on nearest neighbors. This reduces the computational complexity from cubic to linear in the number of conditioning points.
Hyperparameter Inference: The framework uses a fully Bayesian approach, integrating out the outer layer variance via a reference prior and sampling length scales via Metropolis-Hastings within a Gibbs loop alongside the ESS.

Key Contributions

Gradient-Enhanced DGPs (geDGP): The first comprehensive Bayesian framework for DGPs that conditions on observed gradients. This allows the model to leverage derivative information to improve predictive accuracy, particularly in data-scarce regimes.
DGP Gradient Predictions: A method to obtain posterior predictive distributions for the gradients of the DGP response at unobserved inputs. This is achieved by inferring the gradients of the latent warping and the outer layer, then combining them via the chain rule.
Computational Scalability: The integration of Vecchia approximation specifically tailored for gradient-enhanced DGPs, enabling the application of these models to datasets with hundreds of observations and gradients, which would otherwise be computationally prohibitive.
Open-Source Implementation: The release of the deepgp R-package on CRAN, which includes these new gradient capabilities and the Vecchia approximation, along with code to reproduce all experiments.

Results
The authors benchmark their methodology against stationary GPs, gradient-enhanced GPs (geGP), and non-enhanced DGPs on three nonstationary test functions ("squiggle," "plateau," and "ignition") and a real-world quantum mechanics application (SPICE dataset).

Performance: On nonstationary surfaces, the gradient-enhanced DGP (geDGP) consistently outperformed both the geGP and the non-enhanced DGP in terms of Root Mean Squared Error (RMSE) and Continuous Ranked Probability Score (CRPS).
Gradient Prediction: The DGP's ability to predict gradients was superior to that of the standard GP, even when the GP was gradient-enhanced. This highlights the importance of the nonstationary warping in accurately capturing gradient behavior.
Real-World Application: In the quantum mechanics application (predicting molecular energy and forces), the geDGP provided the most accurate predictions for both energy and forces across various molecules, demonstrating the utility of combining nonstationary modeling with gradient information.
Computational Efficiency: While gradient-enhancement increases computational cost, the Vecchia approximation successfully mitigated these costs, allowing for feasible training and prediction times even with $N=700$ (as in the ignition function example).

Significance and Claims
The paper claims that this work advances the state of surrogate modeling for expensive computer experiments by bridging the gap between the flexibility of DGPs and the information-rich nature of gradient data. The authors assert that:

Gradient-enhancement is most impactful when data is limited, a common scenario in expensive simulations.
The ability to predict gradients from a DGP is crucial for downstream tasks like optimization and active learning.
The proposed framework offers a robust, fully Bayesian alternative to variational inference methods, which may sacrifice performance for speed.
The combination of nonstationary warping and gradient information yields superior uncertainty quantification and predictive accuracy compared to existing stationary or non-enhanced approaches.

The authors remain modest regarding limitations, noting that the current implementation assumes deterministic, noise-free observations and that future work is needed to address stochastic experiments, missing gradient data, and the optimization of Vecchia ordering strategies for gradient models.

The Paper's Story: Stretching the Map with Steepness Clues

Technical Summary: Gradient-Enhancement and Gradient Predictions for Deep Gaussian Process Modeling

More like this