Bayesian neural networks with interpretable priors from Mercer kernels

This paper introduces "Mercer priors," a new class of interpretable priors for Bayesian neural networks derived from Mercer representations of covariance kernels, which enable the networks to approximate Gaussian process samples and thereby combine the scalability of neural networks with the uncertainty quantification interpretability of Gaussian processes.

The Gist

Imagine you are trying to teach a robot to predict the weather. You give it data, and it learns. But here's the problem: How confident is the robot?

If the robot says, "It will rain tomorrow," but it's actually just guessing wildly, that's dangerous. In science and engineering, we need to know not just the answer, but how sure the model is. This is called Uncertainty Quantification.

For a long time, there have been two main ways to do this:

1. The "Gaussian Process" (GP): Think of this as a master craftsman. It's incredibly precise and can tell you exactly how "wiggly" or "smooth" its predictions should be. It's very interpretable (we understand why it thinks what it thinks). But, it's slow. If you have a huge dataset (like millions of weather readings), the craftsman gets overwhelmed and stops working.
2. The "Bayesian Neural Network" (BNN): Think of this as a super-fast assembly line robot. It can handle massive amounts of data and learn complex patterns quickly. But, it's a bit of a "black box." We don't really know what rules it's following. Usually, we just tell it, "Guess randomly," which isn't very helpful for safety-critical decisions.

The Problem

Scientists want the speed of the robot but the wisdom of the craftsman.

• If you try to make the robot act like the craftsman, it usually breaks or becomes too slow.
• If you just let the robot guess randomly, it might make dangerous mistakes because it doesn't understand the "rules of the game" (like how smooth a temperature curve should be).

The Solution: "Mercer Priors"

The authors of this paper invented a new way to train the robot. They call it Mercer Priors.

Here is the analogy:
Imagine the "Craftsman" (GP) has a secret recipe book that defines exactly how a perfect weather curve should look. This recipe is written in a very complex mathematical language called a Mercer Kernel.

• Old Way: To make the robot follow the recipe, you had to rebuild the robot's entire brain (architecture) to match the recipe. This was hard and often didn't work.
• The New Way (Mercer Priors): Instead of rebuilding the robot's brain, you just change the "ingredients" you feed it.

The authors figured out how to take the Craftsman's secret recipe and translate it directly into the initial "mood" or "personality" of the robot's parameters (its weights and biases).

How It Works (The Metaphor)

Think of the Neural Network as a musical instrument (like a guitar).

• Standard Training: You just pluck the strings randomly and hope it sounds good.
• Mercer Prior Training: Before you even play a note, you tune the strings based on a specific piece of music (the "Mercer representation"). You aren't changing the guitar; you are just ensuring that any note you play from now on will naturally sound like that specific song.

By doing this, the robot (BNN) starts with a "personality" that already knows how to behave like the master craftsman (GP). It knows how to be smooth, how to be wiggly, or how to handle periodic patterns (like seasons), just by how it was initialized.

Why Is This a Big Deal?

1. Speed: The robot is still fast. It can handle millions of data points because it's still a neural network.
2. Interpretability: Because we tuned it using the "Craftsman's recipe," we know exactly what kind of behavior to expect. If we want the temperature to be smooth, we tune it to be smooth.
3. Real-World Power: The paper shows this working on real, messy problems:
   • Motorcycle Crashes: Predicting how a helmet absorbs impact with varying levels of noise.
   • CO2 Levels: Predicting atmospheric carbon dioxide, which goes up and down in a yearly cycle.
   • Spacecraft Heat: Figuring out how heat shields work on a spaceship, which involves solving complex physics equations.

The "Brownian Motion" Test

To prove their method works, they tried to make the robot mimic Brownian Motion (the random jittery movement of particles in water). This is a notoriously difficult, "jagged" pattern.

• They showed that with their new "ingredients," the robot could mimic this jagged movement almost perfectly, even though the robot itself is made of smooth, mathematical functions.
• It's like teaching a smooth marble to roll in a way that looks exactly like a jagged rock, just by changing how you push it initially.

Summary

The paper introduces a clever trick: Don't change the machine; change the starting conditions.

By using a mathematical tool called the Mercer representation, they can inject the "wisdom" of slow, precise Gaussian Processes directly into the "speed" of fast Neural Networks. This gives us AI models that are both fast enough for big data and smart enough to be trusted in critical scientific applications.

Technical Summary

Here is a detailed technical summary of the paper "Bayesian neural networks with interpretable priors from Mercer kernels" by Alex Alberts and Ilias Bilionis.

1. Problem Statement

Bayesian Neural Networks (BNNs) are essential for uncertainty quantification (UQ) in scientific and engineering applications where decisions rely on noisy or limited data. However, standard BNNs suffer from a critical limitation: the lack of interpretable priors.

• The Issue: In standard BNNs, priors are typically independent and identically distributed (i.i.d.) Gaussian distributions over weights. Due to the complex non-linear mapping of neural networks, it is difficult to understand how these weight priors constrain the function space (the output). Consequently, the resulting uncertainty estimates often lack physical or mathematical meaning.
• The Alternative & Its Drawback: Gaussian Processes (GPs) offer highly interpretable priors defined by covariance kernels, allowing users to encode specific structural knowledge (e.g., smoothness, periodicity). However, GPs do not scale well to large datasets or high-dimensional inverse problems due to the $O(N^3)$ computational cost of inverting covariance matrices.
• The Gap: While it is known that wide BNNs converge to GPs in the infinite-width limit, the specific activation function required to match a desired GP kernel is often unknown or impractical to derive. Existing methods to force BNNs to mimic GPs (e.g., Ridgelet priors) often suffer from the curse of dimensionality or high computational costs.

2. Methodology: The Mercer Prior

The authors propose a new class of priors called Mercer Priors. The core idea is to construct a prior distribution directly over the BNN parameters ($\theta$) such that the resulting network samples approximate draws from a target Gaussian Process defined by a specific covariance kernel $k$.

Theoretical Foundation

1. Gaussian Measures: The method treats the sample paths of a target GP as draws from a Gaussian measure $\mathcal{N}(0, S)$ on a function space $L^2(\Omega)$, where $S$ is the covariance operator associated with the kernel $k$.
2. Mercer's Theorem: The authors utilize Mercer's theorem to express the inverse covariance operator $S^{-1}$ (the precision operator) via its spectral decomposition:
   $$k^{-1}(s, t) = \sum_{n=1}^{\infty} \lambda_n^{-1} \phi_n(s) \phi_n(t)$$
   where $\lambda_n$ and $\phi_n$ are the eigenvalues and eigenfunctions of the covariance operator.
3. Prior Definition: Instead of defining a prior on weights, they define a prior density over the network parameters $\theta$ based on the energy of the network output $u_\theta$ in the RKHS induced by the kernel:
   $$p(\theta) \propto \exp\left( -\frac{1}{2} \langle u_\theta, S^{-1} u_\theta \rangle \right)$$
   This effectively penalizes network outputs that deviate from the desired covariance structure.

Scalable Sampling Scheme (SGLD)

Evaluating the inner product $\langle u_\theta, S^{-1} u_\theta \rangle$ involves infinite sums and integrals, which are computationally intractable. The authors derive an unbiased Monte Carlo estimator to enable sampling using Stochastic Gradient Langevin Dynamics (SGLD):

• Truncation: The infinite sum is truncated to $K$ terms (eigenpairs).
• Importance Sampling: The integrals over the domain $\Omega$ are approximated by sampling points uniformly (or via importance sampling).
• Unbiased Estimator: The log-density gradient is estimated using minibatches of domain points and a minibatch of eigenvalues/eigenfunctions.
  $$\nabla_\theta \log p(\theta) \approx \text{Stochastic Estimate using } \{t_\beta\}, \{t_\gamma\}, \{n_\alpha\}$$
• Advantage: This approach avoids inverting large covariance matrices. The computational cost scales linearly with the number of network parameters and the number of sampled eigenpairs, allowing for massive parallelization and super-resolution evaluation.

3. Key Contributions

1. Mercer Priors: Introduction of a novel prior class that bridges the interpretability of GPs with the scalability of BNNs. It allows users to specify any GP kernel (via its eigen-decomposition) and have the BNN inherit that structure.
2. Scalable Sampling Algorithm: Development of an SGLD-based sampling scheme that uses unbiased estimators of the prior density. This bypasses the need for explicit integration or matrix inversion, making the method applicable to large datasets and high-dimensional problems.
3. Flexible Kernel Construction: The method allows for the construction of custom kernels by defining eigenvalues and eigenfunctions directly, bypassing the need for closed-form analytical kernels (which are often unavailable for complex structures).
4. Theoretical Analysis: Detailed investigation into the convergence of the method with respect to:
   • Spectral Truncation ($K$): More eigenvalues capture higher frequencies.
   • Network Width ($N_s$): Wider networks better approximate the target GP.
   • The authors show that convergence to the target GP is expected as $K, N_s \to \infty$.

4. Results and Validation

The paper validates the method through a case study on Brownian motion and three real-world application examples.

Case Study: Brownian Motion

• Goal: Generate BNNs that mimic the non-differentiable paths of Brownian motion (Wiener measure).
• Findings:
  • With sufficient eigenvalues ($K=1000$) and network width, the empirical covariance of the BNN samples closely matches the true Brownian motion covariance $k(s,t) = \min(s,t)$.
  • Statistical tests (Kolmogorov-Smirnov) confirm that the marginal distributions of the BNN samples match the theoretical $N(0, t^2)$ distribution.
  • Limitation: Finite-width networks with smooth activation functions (e.g., sigmoid) produce smooth approximations of Brownian motion, failing to capture the extreme high-frequency "roughness" unless $K$ and width are very large.

Application Examples

1. Hierarchical GP Regression (Heteroscedastic Noise):
   • Task: Modeling motorcycle crash acceleration data with time-varying noise.
   • Result: The Mercer prior enabled a hierarchical model where both the mean and variance functions were modeled as BNNs with specific smoothness constraints. The method successfully captured the heteroscedastic noise structure and scaled to a super-fine mesh (1M points) where standard GP inference would be impossible.
2. Periodic Time Series (CO2 Concentration):
   • Task: Predicting Mauna Loa CO2 levels with a linear trend and strong periodicity.
   • Result: By engineering a custom kernel using periodic eigenfunctions, the BNN with a Mercer prior successfully enforced the periodic structure in future predictions. It outperformed a standard BNN with i.i.d. Gaussian priors, which failed to maintain the periodic trend in the extrapolation region.
3. Nonlinear PDE Inverse Problem (Thermal Conductivity):
   • Task: Inferring the temperature-dependent thermal conductivity of spacecraft insulation from temperature measurements, governed by a nonlinear heat equation.
   • Result: Replacing the standard Gaussian prior with a Mercer prior (enforcing smoothness via $\Delta^{-2}$) allowed for scalable posterior sampling via SGLD. The method successfully reconstructed the thermal conductivity profile with uncertainty quantification, a task computationally prohibitive for standard GPs due to the repeated need to solve nonlinear PDEs and invert large matrices.

5. Significance

• Bridging the Gap: The work successfully unifies the interpretability of Gaussian Processes with the computational scalability of Neural Networks. It allows scientists to use "meaningful" priors (based on physical laws or known structures) without sacrificing the ability to handle large-scale data.
• Scalability in Inverse Problems: It provides a practical solution for Bayesian inverse problems governed by nonlinear PDEs, where traditional GP-based methods fail due to cubic scaling.
• Design Flexibility: By decoupling the prior from the network architecture and anchoring it in the spectral properties of the kernel, the method offers a principled way to design BNNs for specific scientific tasks without requiring complex feature engineering or custom activation functions.
• Future Directions: The paper highlights that while empirical convergence is strong, rigorous theoretical convergence proofs for finite-width networks remain an open challenge. Additionally, the joint learning of kernel hyperparameters (e.g., length scales) alongside network weights is identified as a future area for development.