⚡ electrical engineering

Efficient Gaussian process learning via subspace projections

This paper introduces a novel projected likelihood training objective for Gaussian processes that utilizes lower-dimensional linear projections to achieve superior accuracy and computational efficiency compared to exact and variational sparse GP methods on moderately large datasets.

Original authors: Elsa Cazelles, Felipe Tobar

Published 2026-01-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elsa Cazelles, Felipe Tobar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but incredibly slow, robot how to predict the future based on a massive history book. This robot is a Gaussian Process (GP). It's famous for being incredibly accurate and for telling you how confident it is in its predictions (uncertainty quantification). However, there's a catch: the more data you give it, the slower it gets. If you have a few thousand pages of history, the robot takes forever to read them all. If you have a million pages, it simply gives up.

This paper introduces a new trick to make the robot faster without making it "dumber." They call this trick Projected Likelihood (PL).

Here is how it works, using some everyday analogies:

1. The Problem: The "Perfect Memory" Bottleneck

Normally, to learn from data, the robot tries to memorize the relationship between every single pair of data points. If you have 1,000 data points, it has to check 1,000,000 connections. This is like trying to understand a crowd of people by shaking hands with everyone and asking them what they think of everyone else. It's thorough, but it takes a lifetime.

2. The Old Shortcut: The "Representative Group"

Scientists previously tried to speed this up by picking a small group of "representatives" (called inducing variables) from the crowd. The robot only talks to these representatives and assumes everyone else is just like them.

The Flaw: Sometimes the robot gets the wrong idea about the crowd because the representatives aren't perfect. It might think the noise level is higher or lower than it really is. Also, the robot still has to do a lot of extra math to figure out who the best representatives are, which eats up time.

3. The New Solution: The "Shadow Projection"

The authors propose a different approach. Instead of picking specific people to talk to, they shine a light on the whole crowd from a few different angles to cast shadows on the wall.

The Analogy: Imagine you have a complex 3D sculpture (your data). Instead of studying every curve of the sculpture, you project its shadow onto a flat wall from a few random directions.
The Magic: The authors found that if you choose these shadow-casting angles randomly (specifically, pointing in random directions on a sphere), the shadow retains almost all the important information needed to learn the shape.
The Result: The robot only needs to study the 2D shadows (which are much smaller and simpler) instead of the 3D sculpture. This is the Projected Likelihood.

4. Why It's Better (The "Sweet Spot")

The paper tested this new method against the old "Representative Group" method on datasets ranging from 500 to 8,000 data points.

Accuracy: The "Shadow" method (PL) learned the shape of the data much more accurately than the "Representative" method. It didn't get confused about the noise levels or the patterns.
Speed: Even though the math for the shadows looks complicated on paper, in practice, it was faster. Why? Because the "Representative" method had to take many more steps to "learn" the right representatives, whereas the "Shadow" method just needed a few steps to get it right.
The "Random" Surprise: You might think you need to carefully pick the best angles to cast the shadow. The paper shows you don't! Just picking random angles works surprisingly well and captures the most important details of the data.

5. The Bottom Line

The authors are saying: "We found a way to compress the data into a lower-dimensional 'shadow' that keeps the most important information."

For small to medium datasets (up to about 8,000 points): This new method is the clear winner. It is faster and more accurate than the current standard.
The Trade-off: It's not magic; it does lose a tiny bit of information (just like a shadow loses the depth of the object), but the paper proves this loss is so small that the robot learns almost exactly as well as if it had seen the whole object, but in a fraction of the time.

In short, instead of trying to read the whole encyclopedia, the robot now reads a very cleverly summarized version that fits on a single sheet of paper, and it learns the story just as well.

Technical Summary: Efficient Gaussian Process Learning via Subspace Projections

Problem Statement
Gaussian Processes (GPs) are a standard approach for time series analysis, particularly in scenarios involving uneven sampling, missing data, and uncertainty quantification. However, exact GP training via maximum likelihood (ML) incurs a computational cost of $O(n^3)$ , rendering it infeasible for datasets exceeding a few thousand points. While sparse GP methods, such as the Variational Free Energy (VFE) approach using inducing variables, reduce complexity to $O(nm^2)$ , they introduce specific drawbacks:

Bias: They often produce biased hyperparameter estimates (e.g., VFE tends to overestimate noise variance, while FITC underestimates it) due to the restricted choice of the approximating variational distribution.
Asymptotic Efficiency: Their computational superiority is only realized asymptotically (for sufficiently large $n$ ), and the training loss involves complex operations (multiple Cholesky decompositions) that may require more optimization steps to converge.

The authors identify a need for a training method that balances computational efficiency with accuracy for moderately large datasets, avoiding the biases inherent in current sparse approximations.

Methodology: Projected Likelihood (PL)
The paper proposes a novel training objective termed Projected Likelihood (PL). Instead of approximating the posterior via inducing variables, PL constructs a surrogate GP over a lower-dimensional linear projection of the data.

Projection Mechanism: The $n$ -dimensional observation vector $y$ is projected onto $k$ directions ( $k \ll n$ ) defined by vectors $\omega_1, \dots, \omega_k$ on the unit sphere $S^{n-1}$ . This yields a compressed $k$ -dimensional vector $z = \Omega^\top y$ .
Training Objective: The training minimizes the negative log-likelihood (NLL) of this projected data. The resulting objective function, $L_{PL}(\theta)$ , has a computational cost of $O(kn^2)$ .
Information Loss Analysis: The authors derive a closed-form expression for the Fisher information loss incurred by using PL instead of exact ML. They prove that this loss is quadratic in the conditional covariance $\Sigma_{Y|Z}$ $Σ_{Y ∣ Z}$ .
- Theoretical Insight: The optimal projection directions correspond to the top $k$ eigenvectors of the covariance matrix $K_x$ . Since these are unknown prior to training, the authors experimentally evaluate random projection strategies.
- Empirical Finding: Among various projection structures (One-hot, Localized, Repulsive, and Spherical), random projections uniformly distributed on the unit sphere (Spherical) were found to effectively minimize the trace of the residual covariance $\Sigma_{Y|Z}$ , thereby reducing information loss.

Key Contributions

Novel Objective: Introduction of the Projected Likelihood (PL) as a training objective based on low-dimensional linear projections rather than inducing variables.
Theoretical Quantification: A derivation of the information loss associated with PL, demonstrating that it can be minimized via random projections on the unit sphere.
Efficiency vs. Accuracy Trade-off: The paper demonstrates that while PL has a theoretical complexity of $O(kn^2)$ (which appears higher than the $O(nm^2)$ of sparse GPs for small $m$ ), it is empirically more efficient. This is because PL requires significantly fewer optimization steps to converge compared to VFE, which often struggles with a more complex loss landscape.
Bias Reduction: Unlike VFE, which is known to bias hyperparameter estimates (specifically noise variance), PL achieves solutions closer to the exact ML baseline.

Experimental Results
The authors validated PL against VFE and exact ML across synthetic and real-world datasets (Sunspots and Neonatal EEG) using various kernels (SE, Laplace, RQ, LocPer) and optimizers (BFGS, Adam).

Optimization Speed: In experiments with 1,000 samples, PL achieved an NLL closer to the exact ML baseline than VFE while requiring fewer iterations. For instance, with the Adam optimizer, PL took 0.574 seconds compared to 1.57 seconds for VFE.
Scalability: For datasets ranging from 500 to 4,000 points, PL reached the target NLL faster than VFE for $n < 3000$ . As $n$ increased, the performance gap narrowed, but PL maintained superior or comparable accuracy.
Real-World Performance: On the Sunspot dataset (3,319 points) and EEG dataset (8,000 training points), PL consistently outperformed VFE in terms of accuracy (lower NLL or RMSE) across all tested kernels. While computational times became similar for very large datasets, PL maintained a clear accuracy advantage.
Variance Estimation: Posterior variance estimates from PL remained closer to the ML estimate, whereas VFE tended to overestimate the variance.

Significance and Claims
The paper claims that the PL method offers a competitive alternative to sparse GPs for moderately large datasets (up to 8,000 points). Its primary significance lies in demonstrating that compressed data representations can be leveraged in GP learning to discard natural redundancies in time series without significantly sacrificing learning accuracy.

Crucially, the authors argue that the evaluation of approximate-likelihood methods should not rely solely on the computational cost of a single loss evaluation (e.g., $O(nm^2)$ vs $O(kn^2)$ ) but must also consider the complexity of the loss landscape. PL's ability to converge in fewer steps makes it practically more efficient than VFE, despite the theoretical complexity of its objective function. The study validates that random projections on the unit sphere provide a robust, low-overhead mechanism to approximate the exact GP likelihood.