📊 statistics

On the Brittleness of Maximum Likelihood Estimation for Gaussian Process Hyperparameter Optimization

This paper investigates the brittleness of Maximum Likelihood Estimation (MLE) in training Gaussian Processes, demonstrating that when underlying assumptions are violated, MLE leads to poor generalization, and proposes theoretically grounded, practical solutions that outperform existing methods in accuracy, uncertainty quantification, and inference cost for downstream engineering tasks.

Original authors: Tyler R. Johnson, Kian Ben-Jacob, Christopher P. Muller, Ramin Bostanabad

Published 2026-08-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tyler R. Johnson, Kian Ben-Jacob, Christopher P. Muller, Ramin Bostanabad

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, you are looking at a messy pile of data points scattered on a table. Your goal is to draw a smooth line (or a complex shape) through these dots to predict what happens next. In the world of engineering and science, this is called "machine learning," and it's the secret sauce behind everything from designing safer bridges to predicting how a new drug will behave.

To draw that line, the detective needs a rulebook, or a "loss function," to tell them how close their guess is to the truth. For decades, the most popular rulebook has been called Maximum Likelihood Estimation (MLE). Think of MLE as a very strict teacher who says, "Your job is to make your model explain the data you have perfectly." If the data looks like a smooth curve, MLE tries to fit a smooth curve. If the data looks noisy, MLE tries to explain the noise. It's intuitive and powerful, but it has a secret weakness: it assumes the world is exactly the way the teacher thinks it is. If the data is messy, the teacher is confused, and the model might look perfect in class (on the training data) but fail miserably on the final exam (real-world predictions).

Recently, a new kind of detective has arrived: the "Foundation Model." These are super-smart, pre-trained AI systems (like TabPFN) that have already seen millions of made-up problems. They don't need to learn from scratch; they just look at your data and instantly guess the answer. They are fast and often very good. But here is the big question: Is the old, strict teacher (MLE) actually broken, or can we just teach it better? And is the new super-detective actually better, or just lucky?

This paper, written by a team of engineers at the University of California, Irvine, dives deep into this debate. They treat the "brittleness" of the old teacher (MLE) like a fragile glass vase. They ask: "If we shake the vase a little (add noise) or look at it from a weird angle (high dimensions), does it shatter?"

The authors ran a massive experiment involving over 3,000 different simulations. They tested the old teacher (Gaussian Processes trained with MLE) against the new super-detective (TabPFN) on all sorts of puzzles, from simple curves to complex, multi-dimensional engineering problems.

Here is what they found. First, they confirmed that the old teacher is indeed brittle. When the data is scarce or the problem is complicated, MLE often gets stuck in a "local optimum." Imagine a hiker trying to find the highest peak in a foggy mountain range. MLE is like a hiker who only looks at the ground immediately around their feet. If they start in a small valley, they might think they've found the top because they can't see the higher peaks nearby. This leads to models that look great on the data they were trained on but are terrible at predicting new things.

However, the paper doesn't say "throw MLE in the trash." Instead, they found that if you give the hiker a better map and a flashlight, they can still win. By changing how the teacher looks at the numbers (using a "log-scale" transformation) and by letting the teacher try starting from many different spots on the mountain (multiple initializations), the old teacher becomes much more robust. With these simple tweaks, the MLE-trained models became incredibly accurate, often beating the new super-detective in predicting not just the answer, but also how sure they were about that answer.

The study also compared these models in a real-world scenario called "Bayesian Optimization," which is like trying to find the best recipe for a cake by tasting only a few bites at a time. Here, the old teacher (when tuned correctly) was often faster and more efficient than the new super-detective. The super-detective is great at guessing without training, but it gets slow and expensive when you have to ask it a million questions. The old teacher, once trained, is lightning fast.

In the end, the paper suggests that we shouldn't blindly abandon the old, trusted methods for the shiny new ones. The "brittleness" of Maximum Likelihood Estimation isn't a fatal flaw; it's a bug that can be fixed with better tools and a little more patience. While the new foundation models are impressive, a carefully tuned Gaussian Process can still be the champion of accuracy, uncertainty, and speed for many engineering tasks. The key isn't to switch teams, but to make sure your coach knows how to train the player properly.

Technical Summary: On the Brittleness of Maximum Likelihood Estimation for Gaussian Process Hyperparameter Optimization

Problem Statement

Machine learning (ML) has become integral to engineering design workflows, often serving as surrogates for expensive simulations or experiments. A critical step in training these models is the selection of a loss function, with Maximum Likelihood Estimation (MLE) being the most widely adopted approach due to its statistical foundations and computational tractability. However, MLE is inherently brittle: its theoretical guarantees rely on assumptions such as correct model specification, independent observations, and sufficiently large datasets—conditions frequently violated in practical engineering design problems.

When these assumptions fail, MLE-optimized models may fit training data well but generalize poorly, a mismatch particularly detrimental in workflows requiring uncertainty quantification (UQ), such as Bayesian Optimization (BO). This brittleness affects even Gaussian Processes (GPs), which are often incorrectly presumed to be robust against overfitting. Furthermore, the emergence of foundation models (FMs) like TabPFN, which utilize pre-trained inference mechanisms to bypass per-problem training, raises questions about the comparative efficacy of traditional MLE-trained GPs versus modern FMs.

Methodology

The authors systematically evaluate the brittleness of MLE in training GPs for regression and classification tasks, comparing them against TabPFN (versions 2.0 and 2.5). The study encompasses 3,170 independent simulations across varying noise levels, dataset sizes ( $N = 5D_x$ and $N = 20D_x$ ), and input dimensionalities ( $D_x \in \{4, 10, 20, 40, 80\}$ ).

Proposed Mitigations for MLE

Rather than abandoning MLE, the paper proposes practical strategies to mitigate its brittleness:

Reparameterization: Transforming hyperparameters into log-scale (e.g., representing lengthscales as $10^\omega$ ) to ensure large gradients and remove constraints, facilitating better convergence.
Soft Clamping: Introducing a soft clamp function to bound hyperparameters during optimization, preventing numerical instability without causing non-smooth behavior at boundaries.
Multi-Initialization: Employing space-filling designs for multiple random initializations (defaulting to 16) to reduce the risk of converging to poor local optima.
Alternative Loss Functions: The study evaluates alternatives such as Leave-One-Out (LOO) log-pseudo-likelihood, Kernel Flows, and proper scoring rules (NIS, CRPS), though it notes that LOO can be sensitive and Kernel Flows lack native UQ capabilities.

Experimental Setup

Regression: Nine benchmark problems (e.g., Ackley, Rosenbrock, Buckling) were tested using RRMSE for accuracy and Normalized Interval Score (NIS) for UQ.
Bayesian Optimization: BO trials were conducted on the same benchmarks, comparing GP-based BO (retraining via MLE at each step) against TabPFN-based BO (in-context learning).
Classification: Four datasets (Electrical Grid, Truss 6D, Stellar, Steel Plates) were used to evaluate accuracy and Expected Calibration Error (ECE), utilizing a Dirichlet target construction to cast classification as a regression problem for GPs.
Baselines: Comparisons included GPyTorch (standard MLE), GP+ (the authors' implementation with proposed mitigations), and TabPFN versions.

Key Results

Regression Performance

MLE vs. Alternatives: GPs trained with the proposed MLE mitigations (GP+) generally performed on par with or better than TabPFN 2.5 in terms of prediction accuracy (RRMSE) and UQ (NIS).
Kernel and Loss Sensitivity: While GP+ (PE) (Power Exponential kernel) and GP+ (LOO) showed superior performance in specific high-dimensional or low-data regimes, they were inconsistent. GP+ (LOO) often suffered from poor UQ in low-data regimes due to overconfidence (narrow prediction intervals).
Data Regime: In low-data, high-noise scenarios, standard GPyTorch often failed (learning constant values), whereas GP+ with mitigations remained robust. In high-data regimes, GP+ variants consistently outperformed TabPFN in most benchmarks, except for the Zakharov function where TabPFN excelled due to better handling of observation scale.
Preprocessing: Log-scale target preprocessing significantly improved GP performance on the Buckling problem, whereas TabPFN remained robust to such scaling choices.

Bayesian Optimization

Efficiency vs. Accuracy: GP-based BO generally approached optima faster than TabPFN-based BO in most benchmarks (e.g., Buckling, Borehole).
Computational Trade-off: TabPFN eliminates per-iteration retraining costs but incurs high inference costs when evaluating acquisition functions over large candidate pools (as gradient-based optimization is infeasible). Conversely, GPs require retraining but support analytic acquisition functions and fast predictions. In small-data regimes where GP retraining is cheap, GPs proved computationally competitive or superior.

Classification Performance

Kernel Selection: The relationship between training loss and test performance was not strict. For instance, the Matérn kernel ( $\nu=0.5$ ) had the highest training loss but achieved the lowest ECE and highest accuracy on the Steel Plates Faults dataset.
TabPFN vs. GP+: TabPFN 2.5 maintained a slight advantage in accuracy and calibration across most classification tasks, particularly those involving categorical variables and multi-class settings. However, this advantage diminished as training set size ( $N$ ) increased, with GP+ variants overtaking TabPFN in accuracy on the Electrical Grid Stability dataset for $N \ge 200$ .

Significance and Contributions

The paper claims that while MLE is the dominant paradigm for GP training, its effectiveness is highly dependent on implementation details often overlooked in practice. The primary contributions are:

Demonstration of Brittleness: The study provides extensive empirical evidence that MLE-trained GPs can fail systematically in engineering contexts due to non-convex optimization landscapes and model misspecification, particularly in small-data, high-dimensional settings.
Practical Remedies: The authors demonstrate that simple, practical interventions (log-scale reparameterization, soft clamping, multi-initialization) can substantially mitigate MLE failures without abandoning the framework.
Comparative Analysis with Foundation Models: The work offers a rigorous comparison between classical probabilistic methods (GPs) and emerging foundation models (TabPFN). It concludes that carefully implemented MLE-trained GPs remain highly competitive, often outperforming TabPFN in prediction accuracy, UQ quality, and inference cost, particularly in the dimensionality ranges explored ( $D_x < 100$ ).
Blueprint for Practitioners: The paper provides a blueprint for building accurate and robust GPs that can rival or exceed foundation models, emphasizing that investing in proper MLE implementation is often more effective than switching loss functions or models entirely.

The authors note that their findings are based on analytic benchmarks with Gaussian noise and suggest future work should extend to non-Gaussian noise, real-world datasets, and downstream tasks like multi-fidelity modeling. They also highlight the potential value of meta-learning GP hyperpriors inspired by TabPFN's training paradigm to further reduce sensitivity to initialization.