Neurally-plausible radial basis kernels using distributed Fourier embeddings

Technical Summary: Neurally-plausible radial basis kernels using distributed Fourier embeddings

Problem Statement
Cognitive systems require cohesive, continuous representations of multidimensional spaces where equidistant points maintain their sense of equidistance regardless of direction. Traditional Fourier-based embeddings, such as Random Fourier Features (RFFs) and standard Spatial Semantic Pointers (SSPs), typically realize single-dimensioned kernels. When these are combined to form higher-dimensional representations, the resulting inner-product similarity becomes a product of individual kernels. Consequently, the similarity between equidistant points depends on both the magnitude and the direction of the vector connecting them, failing to achieve true radial symmetry. While Hexagonal Spatial Semantic Pointers (HexSSPs) have been observed to exhibit radial basis function (RBF)-like behavior, the exact analytical nature of their induced kernel has remained uncharacterized. Furthermore, it is unclear how alternative sampling methods using grid cell-like representations can approximate arbitrary radial basis kernels.

Methodology
The paper utilizes Bochner's theorem to establish a mathematical bridge between probability distributions in Fourier space and their corresponding shift-invariant kernels. The core approach involves characterizing the kernel $K(\mathbf{x})$ as the inverse Fourier transform of a probability density function $p(\boldsymbol{\omega})$ over the phase space.

To achieve a radial basis kernel (isotropic similarity independent of direction), the authors propose sampling rows of a phase matrix $\mathbf{A}$ from a joint probability distribution that is rotation-invariant (isotropic). This requires the distribution of phase vectors to be separable into independent magnitude ($r = \|\boldsymbol{\omega}\|_2$) and direction ($\hat{\boldsymbol{\omega}}$) components, where the directional component is uniformly distributed over the surface of an $n$-ball.

The paper derives a general integral form for the similarity kernel in an $n$-dimensional feature space:
$$K(\mathbf{x}) = \Gamma\left(\frac{n}{2}\right) \int_0^\infty p(r) \left( \frac{2}{r\|\mathbf{x}\|_2} \right)^{\frac{n}{2}-1} J_{\frac{n}{2}-1}(r\|\mathbf{x}\|_2) \, dr$$
where $J$ is the Bessel function of the first kind.

The authors then apply this framework to three specific sampling distributions for the magnitude $r$:

1. Uniform Distribution ($r \sim U(0, \lambda)$): Corresponds to the standard HexSSP implementation where simplex scales are uniformly sampled.
2. Chi Distribution ($r \sim \chi(n)$): Corresponds to sampling from an isotropic Gaussian distribution.
3. Scaled Beta Distribution ($r \sim \beta_\lambda(n, 1)$): Corresponds to uniform sampling within an $n$-ball.

Key Contributions and Results

1. Analytical Characterization of HexSSP Kernels:
   The paper provides the first analytical derivation of the kernel induced by standard HexSSPs (where magnitudes are uniformly distributed). The resulting kernel is identified as an integrated hypergeometric kernel (or simply "hypergeometric kernel").

   • The kernel is expressed as a power series involving the Pochhammer symbol.
   • Analysis shows that as the dimensionality $n$ increases, the kernel widens and oscillations decrease, approaching a constant value of 1. This implies that for sufficiently high feature dimensions, the embedding space loses the ability to distinguish between points, a limitation not present in Gaussian kernels.

2. Derivation of Alternative Kernels:
   The authors demonstrate that by altering the magnitude sampling distribution within the HexSSP framework, different radial basis kernels can be realized:

   • Gaussian Kernel: By sampling magnitudes from a Chi distribution ($r \sim \chi(n)$), the framework realizes a standard Gaussian kernel $K(\mathbf{x}) = e^{-\|\mathbf{x}\|^2/2}$. Unlike the hypergeometric kernel, this form is robust to changes in dimensionality $n$ regarding its shape, provided the distribution of $r$ scales appropriately.
   • $n$-jinc Kernel: By sampling magnitudes from a scaled Beta distribution ($r \sim \beta_\lambda(n, 1)$), the framework realizes a decaying Bessel function kernel, termed the "$n$-jinc kernel." This kernel behaves like a jinc function ($x^{-1}J_0(x)$) when $n=2$. Like the hypergeometric kernel, its shape is dependent on the dimensionality $n$.

3. Optimality of Grid Cell-like Representations:
   The paper argues that the geometric structure of HexSSPs—constructed from centred simplices (analogous to grid cell firing fields)—provides an optimal primitive for approximating radial basis functions. The simplex shape ensures equal contribution of all points in all directions while minimizing the number of points required to cover the $n$-dimensional feature space. The combination of multiple grid cell orientations and scales allows the system to approximate any symmetric, centred probability distribution $p(r)$, and thus any radial basis kernel.

Significance and Claims
The paper claims to demonstrate that neurally-plausible representations with grid cell-like phase structures are sufficient ("all you need") to approximate any radial basis function. By analytically characterizing the hypergeometric kernel and deriving Gaussian and $n$-jinc alternatives, the work bridges the gap between abstract distributed representations (SSPs) and biological plausibility (grid cells).

The authors state that this work aims to motivate increased consideration of how neurally-plausible representations are constructed for tasks involving high-dimensional optimization and cognitive modeling. The paper emphasizes that grid cell-like representations are not only capable of realizing radial basis kernels but are optimal for doing so, offering a unified framework for synthesizing physical and perceptual phenomena into a single representational space.