Stationary Kernels and Gaussian Processes on Lie Groups and their Homogeneous Spaces I: the compact case

This paper presents constructive and practical techniques for building, calculating, and sampling stationary Gaussian processes on compact Lie groups and their homogeneous spaces, thereby extending standard non-Euclidean modeling capabilities to practitioners.

The Gist

Imagine you are trying to predict the weather, but instead of looking at a flat map of the world, you are trying to predict it on a spinning globe, a twisted donut, or even a complex 4D shape that exists in a physics simulation.

This is the challenge the authors of this paper tackle. They are building a new toolkit for Gaussian Processes (GPs).

What is a Gaussian Process? (The "Smart Guessing Machine")

Think of a Gaussian Process as a super-smart guessing machine. If you give it a few data points (like "it's raining in London" and "it's sunny in Paris"), it doesn't just draw a straight line between them. It draws a smooth, wiggly curve that represents all the possible ways the weather could be in between, while also telling you how confident it is in its guess.

In standard machine learning, these machines work great on flat, 2D maps (Euclidean space). But the real world is often curved, twisted, or has symmetries (like a sphere or a rotating robot arm). Standard GPs break down on these shapes because they don't understand the geometry.

The Problem: The "Flat Earth" Trap

The paper starts by pointing out a common mistake. If you try to use a standard "distance" formula (like measuring the straight line through the Earth) on a sphere, your math breaks. It's like trying to measure the distance between New York and London by drilling a tunnel through the Earth's core; the math says they are close, but in reality, you have to fly over the surface.

Previous attempts to fix this were either too messy (using "heuristic" tricks that worked sometimes but weren't reliable) or too slow (requiring supercomputers to solve complex physics equations).

The Solution: The "Symmetry" Superpower

The authors' breakthrough is realizing that many of these weird shapes (like spheres, rotating groups, and projective planes) have symmetries.

• Symmetry means if you rotate a sphere, it looks the same. If you shift a pattern on a donut, it looks the same.

The paper introduces a new way to build these "Smart Guessing Machines" that respects these symmetries. They call these Stationary Kernels.

The Analogy:
Imagine you are painting a pattern on a spinning globe.

• Old Way: You try to paint a pattern on a flat sheet of paper, then wrap it around the globe. The pattern stretches and tears at the poles.
• New Way (This Paper): You design the pattern specifically for the globe. You use the globe's own "language" (mathematics called Representation Theory) to ensure that no matter how you spin the globe, the pattern looks consistent and smooth.

How They Did It: The "Musical" Approach

The authors use a branch of math called Representation Theory, which is essentially the study of how groups (like rotations) act on things.

Think of a complex shape like a symphony orchestra.

• The Old Way: Trying to describe the sound of the orchestra by listing every single note every musician is playing at once. It's chaotic and hard to compute.
• The New Way: The authors realize the orchestra is made of distinct "families" of notes (harmonics). They break the problem down into these fundamental "notes" (called characters and spherical functions).
  • Just as a musical chord is a combination of specific notes, the "smoothness" of their new Gaussian Process is a combination of these mathematical "notes."
  • By knowing the "notes" (the math of the shape), they can write a recipe to generate the pattern without needing to solve the whole physics problem from scratch.

What Can You Do With This?

The paper provides two main tools for practitioners:

1. The Calculator: A way to instantly calculate how "similar" two points are on these weird shapes. This is the "kernel" that tells the machine how to connect the dots.
2. The Sampler: A way to generate random, realistic examples of data on these shapes.
   • Example: Imagine you are training a robot to walk. The robot's leg movements happen on a complex curved space. This paper gives you the math to generate thousands of realistic, smooth walking patterns to train the robot, ensuring it doesn't fall over.

The "Heat" and "Matérn" Kernels

The authors didn't just invent new math; they adapted the two most popular types of kernels used in machine learning:

• The Heat Kernel: Think of this as dropping a drop of ink in water and watching it spread. It creates very smooth, gentle curves.
• The Matérn Kernel: Think of this as a slightly rougher, more jagged pattern (like a mountain range). It allows for more "bumpy" data.

They showed how to calculate these specific patterns on spheres, rotating groups, and other complex shapes using their new "symmetry" method.

Why This Matters

Before this, if you wanted to use Gaussian Processes on a sphere or a complex robot joint, you were stuck with slow, approximate, or broken methods.

This paper is like handing a carpenter a new set of tools specifically designed for curved wood.

• It makes the math fast (you can compute it on a laptop).
• It makes the math reliable (it's guaranteed to work correctly).
• It makes the math accessible (they even provided software code so engineers can use it immediately).

In short, they took a very abstract, high-level mathematical theory and turned it into a practical, everyday tool for engineers and scientists working with the complex, non-flat geometry of the real world.

Technical Summary

1. Problem Statement

Gaussian Processes (GPs) are a cornerstone of machine learning for modeling unknown functions and quantifying uncertainty. However, standard GP models are typically defined on Euclidean spaces ($\mathbb{R}^n$) using stationary kernels (e.g., Matérn, Squared Exponential) that rely on translation invariance.

Many real-world applications in physics, engineering, robotics, and neuroscience require modeling functions on non-Euclidean spaces, specifically:

• Lie Groups: Such as the Special Orthogonal Group $SO(n)$ or Special Unitary Group $SU(n)$.
• Homogeneous Spaces: Such as spheres ($S^n$), Stiefel manifolds ($V(k,n)$), and projective spaces ($RP^n$).

The Core Challenge:
Defining "stationarity" on these curved spaces is non-trivial. Naively replacing Euclidean distance with geodesic distance in standard kernels (e.g., $e^{-d(x,x')^2}$) often fails to produce positive semi-definite (PSD) kernels, rendering them invalid for GPs. Existing approaches often rely on:

1. Stochastic Partial Differential Equations (SPDEs): Which require solving difficult eigenvalue problems for the Laplace-Beltrami operator.
2. Ad-hoc heuristics: Lacking a principled theoretical foundation.

The paper aims to develop constructive, practical, and theoretically sound techniques to build stationary GPs on compact Lie groups and their homogeneous spaces, enabling exact or approximate Bayesian learning with guaranteed PSD kernels.

2. Methodology

The authors bridge Representation Theory (specifically of compact Lie groups) with Gaussian Process theory. They generalize Bochner's Theorem (which links stationary kernels to spectral measures in Euclidean space) to non-Euclidean settings using group symmetries.

A. Theoretical Framework

• Stationarity Definition: A kernel $k$ is stationary if it is invariant under the group action: $k(g \triangleright x, g \triangleright x') = k(x, x')$.
• Lie Groups ($G$): The authors show that a stationary kernel on a compact Lie group is a linear combination of characters ($\chi_\lambda$) of irreducible unitary representations:
  $$k(g_1, g_2) = \sum_{\lambda \in \Lambda} a(\lambda) \text{Re}(\chi_\lambda(g_2^{-1} g_1))$$
  where $\Lambda$ is the set of irreducible representations, and $a(\lambda) \geq 0$ are spectral coefficients.
• Homogeneous Spaces ($G/H$): For spaces like spheres or projective planes, the kernel is expressed using zonal spherical functions ($\pi_{jk}^{(\lambda)}$), which are specific matrix coefficients of the group representations that remain constant on cosets:
  $$k(g_1 H, g_2 H) = \sum_{\lambda \in \Lambda} \sum_{j,k=1}^{r_\lambda} a_{jk}^{(\lambda)} \text{Re}(\pi_{jk}^{(\lambda)}(g_2^{-1} g_1))$$

B. Computational Techniques

To make these abstract formulas practical, the authors develop three key algorithmic primitives:

1. Enumerating Representations:

   • They utilize the Highest Weight Theory to map irreducible representations to integer tuples called signatures ($\Lambda \subset \mathbb{Z}^m$).
   • This allows for the systematic enumeration and truncation of the infinite series required for kernel evaluation.

2. Pointwise Kernel Evaluation:

   • Lie Groups: They use the Weyl Character Formula to compute characters $\chi_\lambda(g)$ as ratios of polynomials (or determinants) of the eigenvalues of $g$. This allows for efficient, differentiable evaluation.
   • Homogeneous Spaces: They introduce Generalized Periodic Summation. If the kernel on the group $G$ is known, the kernel on $G/H$ is obtained by averaging the group kernel over the subgroup $H$ (using Haar measure). This avoids deriving complex spherical functions for every specific manifold.

3. Efficient Sampling (Generalized Random Phase Fourier Features):

   • Instead of solving SPDEs or computing full eigen-decompositions, they propose a Monte Carlo sampling method.
   • They construct random feature maps using the characters (for groups) or zonal spherical functions (for homogeneous spaces) combined with random phases and random samples from the Haar measure.
   • This enables sampling from the prior and posterior GPs in $O(L \cdot S)$ time (where $L$ is the truncation order and $S$ is the number of Monte Carlo samples), bypassing the $O(N^3)$ cost of standard GP inference for large datasets.

C. Specific Kernel Classes

The authors derive explicit spectral coefficients $a(\lambda)$ for two major kernel families:

• Heat Kernels: Defined as the fundamental solution to the heat equation on the manifold. They correspond to the squared exponential kernel in Euclidean space.
• Matérn Kernels: Defined via an integral transform of the heat kernel (analogous to the Euclidean case). They derive closed-form expressions for the coefficients $a(\lambda)$ based on the eigenvalues of the Laplace-Beltrami operator, which are explicitly calculable via representation theory (Freudenthal's formula).

3. Key Contributions

1. Constructive Theory: Provided a rigorous, constructive characterization of stationary GPs on compact Lie groups and homogeneous spaces using representation theory, generalizing Bochner's theorem.
2. Algorithmic Primitives: Developed practical algorithms for:
   • Evaluating kernels pointwise with automatic differentiation support.
   • Efficiently sampling prior and posterior paths without solving SPDEs.
   • Computing Karhunen-Loève bases.
3. Explicit Kernel Formulas: Derived exact spectral coefficients for Heat and Matérn kernels on these spaces, ensuring they are positive semi-definite by construction.
4. Generalized Periodic Summation: Introduced a technique to compute kernels on homogeneous spaces by averaging group kernels, simplifying the implementation for complex manifolds.
5. Software Implementation: Contributed these methods to the GeometricKernels library, making these models accessible to practitioners.

4. Results

The paper validates the methodology through theoretical analysis and empirical experiments:

• Convergence Analysis:
  • Truncation Error: The error decreases rapidly (polynomially for Matérn, super-exponentially for Heat kernels) as the number of representations (signatures) increases. The convergence is faster than standard manifold Fourier feature expansions due to the grouping of eigenfunctions into characters.
  • Sampling Error: The Monte Carlo sampling (Generalized Random Phase) converges at the standard $O(1/\sqrt{S})$ rate.
• Empirical Evaluation:
  • Experiments were conducted on $SO(3)$, $SO(5)$, and Stiefel manifolds $V(k, 5)$.
  • Results show that a small number of terms (dozens of signatures) are sufficient to accurately approximate the limiting kernel.
  • The proposed methods successfully generated valid GP samples and performed regression tasks (e.g., on the real projective plane $RP^2$ and spheres) with appropriate uncertainty quantification.
• Differentiability: Proved that the constructed Matérn kernels possess the same smoothness properties (Sobolev space regularity) as their Euclidean counterparts.

5. Significance

• Bridging Theory and Practice: This work moves non-Euclidean GPs from theoretical abstractions to practical tools. It replaces "black box" heuristics with mathematically grounded, computable methods.
• Scalability: By avoiding the need to compute full Laplace-Beltrami eigenpairs (which is often intractable for complex manifolds) and instead using representation-theoretic characters, the method scales efficiently.
• Broad Applicability: The framework covers a vast class of spaces relevant to modern AI and science, including rotation groups (robotics), spheres (cosmology), and projective spaces (computer vision).
• Foundation for Future Work: This is Part I of a two-part series. Part II extends these techniques to non-compact symmetric spaces (e.g., hyperbolic spaces, SPD matrices), further expanding the horizon of geometric machine learning.

In summary, the paper provides a unified, computationally efficient, and theoretically rigorous framework for deploying stationary Gaussian processes on compact symmetric spaces, leveraging the deep connection between group representation theory and spectral analysis of manifolds.