🤖 machine learning

Learning Orthonormal Bases for Function Spaces

This paper proposes a neural network-based framework that parameterizes and optimizes orthonormal bases in infinite-dimensional function spaces by modeling them as continuous paths on the orthogonal Lie manifold driven by finite-rank skew-adjoint generators, a method proven to be universal and demonstrated to effectively adapt fixed bases like Fourier to specific datasets and physical structures.

Original authors: Hamidreza Kamkari, Mohammad Sina Nabizadeh, Justin Solomon

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hamidreza Kamkari, Mohammad Sina Nabizadeh, Justin Solomon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to describe a complex piece of music or a detailed painting. To do this, you usually break it down into basic building blocks. In math and computer science, these building blocks are called bases.

Think of a standard basis (like the Fourier basis used in this paper) as a set of standard Lego bricks. They are perfect, uniform, and come in a fixed set of shapes (sine waves, for example). You can build almost anything with them, but if you want to build a very specific, weird shape, you might need thousands of bricks, and the result might look a bit "blocky" or inefficient.

This paper proposes a new way to think about these building blocks. Instead of using fixed, pre-made Lego bricks, the authors suggest 3D-printing custom bricks that are perfectly shaped for the specific object you are trying to build.

Here is how they do it, broken down into simple concepts:

1. The Problem: Fixed Bricks vs. Custom Shapes

In many fields (like analyzing faces, simulating fluids, or processing signals), we use these standard "Lego bricks" (bases) to represent data.

The Issue: Standard bricks are great for general use, but they aren't optimized for specific tasks. If you are trying to describe a face, standard bricks might need hundreds of pieces to capture the curve of a nose, whereas a custom "nose-shaped" brick would do it in one piece.
The Goal: The authors want to find the perfect set of bricks for a specific dataset, but they need to ensure these new bricks still have the nice mathematical properties of the old ones (like being able to be added together easily without breaking the math).

2. The Solution: The "Lie Manifold" Slide

The authors realized that you can't just randomly invent new bricks; if you do, the math breaks. Instead, they treat the space of all possible "perfect brick sets" as a giant, smooth slide (mathematically called a "Lie manifold").

The Analogy: Imagine you start at the top of a slide with your standard, boring Lego bricks. Your goal is to slide down to a spot where the bricks have morphed into the perfect shapes for your specific data (like faces or fluid waves).
The Trick: You can't just jump off the slide. You have to slide down a continuous path. The paper uses a mathematical tool called an ODE (Ordinary Differential Equation) to describe this path. Think of the ODE as the track of the slide. As you move along the track, the bricks slowly, smoothly transform from standard shapes into custom shapes.

3. The Secret Sauce: The "Rank-2" Generator

Here is the most surprising part. To control this slide and morph the bricks, you might think you need a massive, complex machine.

The Claim: The authors prove that you don't need a giant machine. You only need a tiny, simple engine (mathematically, a "rank-2 generator") to control the entire transformation.
The Metaphor: Imagine you want to rotate a massive, heavy globe. You might think you need a huge crane. But the authors show that if you push the globe with a tiny, specific force in a specific pattern over time, you can rotate it to any angle you want. Even though the engine is small, the time it spends pushing allows it to achieve massive, complex changes.
Neural Networks: They use a Neural Network (a type of AI) to act as this tiny engine. The AI learns exactly how to push the "bricks" so that by the end of the slide, they are perfectly shaped for the data.

4. Why This Matters: "Discretization-Free"

Most computer methods turn smooth things (like a curve or a fluid) into a grid of pixels or dots to solve them. This is called "discretization."

The Problem with Grids: If you zoom in too close, the grid looks jagged. You lose the smoothness of the original shape.
The Paper's Advantage: Because their method uses continuous functions (math formulas) rather than grids, the resulting "custom bricks" are smooth and infinite. You can zoom in as much as you want, and the shape remains perfect. It's like having a vector image instead of a pixelated photo.

5. What They Actually Did (The Experiments)

The paper doesn't just talk theory; they tested this "custom brick" idea on three specific things:

Face Recognition (CelebA & MNIST): They took a dataset of faces and digits. Instead of using standard waves to describe them, they learned a new set of "eigenfaces" (custom bricks) that captured the most important features of the faces with far fewer pieces than standard methods. The result was a clearer, more efficient description of the faces.
Fluid Dynamics (Koopman Operator): They simulated swirling water (vortices). Standard methods often lose energy or get messy over time. Their method learned a set of "fluid modes" that preserved the energy of the water perfectly, even as the simulation ran for a long time.
Neural Network Analysis (NTK): They looked at how a neural network learns. They used their method to find the "natural directions" in which the network learns, revealing details about the learning process that grid-based methods missed.

Summary

In short, this paper introduces a way to use AI to morph standard mathematical building blocks into custom shapes that fit specific data perfectly. They do this by sliding along a mathematical path (using a tiny AI engine) that guarantees the new shapes remain mathematically perfect and smooth, avoiding the "pixelation" errors common in other methods.

Technical Summary: Learning Orthonormal Bases for Function Spaces

Problem Statement

Infinite-dimensional orthonormal basis (ONB) expansions are fundamental to representing and computing with function spaces, offering favorable linear algebraic properties. However, standard fixed bases (e.g., Fourier, wavelets) often fail to adapt to the specific structure of a given problem or dataset. While data-driven methods like Principal Component Analysis (PCA) or spectral methods can identify optimal bases, they typically rely on discretization (grids or meshes), introducing approximation errors and limiting the representation to finite dimensions. Conversely, neural network-based approaches that learn continuous functions often sacrifice the linear algebraic structure of the function space, making operations like addition or differentiation unpredictable in the weight space.

The core challenge addressed by this paper is how to parameterize and optimize an infinite, continuous, and orthonormal basis that adapts to specific tasks while strictly preserving the linear structure of the function space, without incurring discretization errors.

Methodology

Lie Group Formulation

The authors propose viewing the space of orthonormal bases not as a static set, but as a path on the Lie manifold of the infinite-dimensional orthogonal group ( $SO(\mathcal{H})$ ). Instead of optimizing an orthogonal operator $Q$ directly, the method models $Q$ as the solution to an Ordinary Differential Equation (ODE) driven by a time-dependent, skew-adjoint infinitesimal generator $K(t)$ :

$\frac{d}{dt}Q(t) = K(t)Q(t), \quad Q(0) = I$

Because $K(t)$ is skew-adjoint ( $K^* = -K$ ), the solution $Q(t)$ remains orthogonal for all $t$ , ensuring the resulting basis is always orthonormal.

Finite-Rank Generators and Universality

A key theoretical contribution is the proof that the infinite-dimensional space of orthogonal operators can be approximated arbitrarily well using finite-rank generators. Specifically, the authors prove a universality theorem (Theorem 1) stating that any rotation in the Hilbert-Schmidt orthogonal group can be approximated by the flow of an ODE governed by a rank-2 skew-adjoint operator.

The generator $K_\theta(t)$ is parameterized by neural networks. For a rank- $r$ generator, the kernel is defined as:
$K_\theta(t)[x, y] = \sum_{k=1}^r \left( u_k(t, x) v_k(t, y) - v_k(t, x) u_k(t, y) \right)$
where $u_k$ and $v_k$ are learned functions (implemented as MLPs) dependent on time $t$ and spatial coordinates. This reduces the optimization of an infinite-dimensional basis to training the parameters $\theta$ of these finite-rank neural networks.

Structure-Preserving Integration

To solve the ODE numerically while preserving the orthogonality constraint, the authors employ the Cayley transform (a Lie group integrator). Unlike standard solvers (e.g., Forward/Backward Euler) which accumulate error and violate orthogonality, the Cayley method uses a $(1,1)$ -Padé approximant:
$\tilde{Q}_\ell = \left(I - \frac{\Delta t}{2} \tilde{K}_\ell\right)^{-1} \left(I + \frac{\Delta t}{2} \tilde{K}_\ell\right)$
This ensures that the discrete update preserves inner products and norms exactly at every step. The authors utilize the Woodbury identity to efficiently compute the inverse action, reducing the computational complexity to be linear in the number of quadrature points $D$ and cubic in the rank $r$ ( $O(r^3 + r^2D)$ ).

Variational Objectives

The framework optimizes the basis by minimizing specific variational objectives:

Eigenproblems (PCA): Maximizing the explained variance of a stochastic process by diagonalizing the covariance operator.
Neural Tangent Kernel (NTK): Diagonalizing the NTK operator to analyze training dynamics.
Koopman Operators: Learning a basis that approximates the Koopman operator for dynamical systems, ensuring energy preservation even when the underlying flow map is not strictly volume-preserving.

Key Results

The paper demonstrates the framework's efficacy across several domains:

1D Synthetic PCA: On a dataset of signals with sharp discontinuities, the learned basis outperforms both the Fourier basis and finite-dimensional PCA. It avoids the Gibbs phenomenon and the Nyquist limit inherent to grid-based methods, achieving order-of-magnitude lower reconstruction errors with roughly 100 basis elements.
Real Data PCA (MNIST & CelebA): Using Implicit Neural Representations (INRs) for MNIST and CelebA, the method learns "eigenfaces" and digit representations that are continuous and discretization-free. The learned bases capture more cumulative energy and provide higher PSNR reconstructions than Fourier baselines with fewer elements.
NTK Diagonalization: When applied to a two-moon classifier, the method resolves fine spatial details near the decision boundary that grid-based eigensolvers miss due to fixed resolution.
Koopman Operator Learning: In fluid dynamics (Taylor-Green vortices), the learned basis generates future state snapshots that strictly conserve energy (Hilbert norm), whereas standard numerical solvers (RK methods) fail to preserve energy over long trajectories.

Significance and Claims

The paper claims to offer a principled framework for learning continuous, orthonormal bases that bridges the gap between the adaptability of neural networks and the structural guarantees of linear algebra.

Theoretical Guarantee: The universality theorem ensures that even low-rank (rank-2) generators are sufficient to approximate any target orthonormal basis, providing a strong theoretical foundation for the parameterization.
Discretization-Free: Unlike spectral methods that require grids, this approach operates directly in function space, allowing evaluation, differentiation, and integration at any point in the domain.
Structure Preservation: By modeling the basis as a path on the orthogonal Lie manifold, the method guarantees exact orthonormality and energy conservation, properties often lost in standard neural approximations.
Versatility: The framework is shown to be applicable to diverse tasks, from statistical dimensionality reduction (PCA) to analyzing neural network training dynamics (NTK) and simulating physical systems (Koopman operators).

The authors acknowledge that optimizing through ODE trajectories can be computationally challenging and that reaching theoretical limits may require long integration horizons. However, they posit that this approach provides a powerful new lens for modeling complex systems involving function spaces, particularly where preserving linear structure and continuity is paramount.