🧬 biology

On periodic distributed representations using Fourier embeddings

This paper proposes using high-dimensional, real-valued periodic embeddings to overcome the limitations of scalar angular measures in representing periodic signals, specifically demonstrating how to construct Dirichlet and periodic Gaussian kernels within the Spatial Semantic Pointers framework.

Original authors: Jakeb Chouinard

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jakeb Chouinard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine you are trying to teach a computer how to understand things that go in circles, like the hands of a clock, the direction a person is facing, or the pitch of a musical note.

In the digital world, computers usually treat numbers as a straight line. If you tell a computer that "1 degree" and "359 degrees" are almost the same, it gets confused. To the computer, 1 and 359 are far apart on a number line, even though on a clock face, they are right next to each other. This creates a "sharp cliff" in the data where things that should be smooth suddenly break.

This paper proposes a clever way to fix that problem using periodic distributed representations. Here is how it works, explained through simple analogies:

1. The Problem: The Straight Line vs. The Circle

Think of a standard computer number line as a long, straight road. If you are driving near the end of the road (say, mile 359) and you want to go to mile 1, you have to drive all the way back to the start. The computer sees a huge distance between them.

But in the real world, many things are circular. If you turn your head slightly from facing North to facing North-North-West, that's a tiny change. But if you represent North as "0" and North-North-West as "359," the computer thinks you spun almost all the way around the world.

2. The Solution: The "Orchestra" of Signals

Instead of using a single number to represent an angle (like "0" or "359"), the authors suggest using a high-dimensional embedding.

Imagine you don't use a single number to describe a direction. Instead, you use a whole choir of 100 different singers (a high-dimensional vector). Each singer hums a specific note (a wave) that repeats over and over.

When you are at "0 degrees," the choir sings a specific harmony.
When you are at "359 degrees," the choir sings a harmony that is almost identical to the one at "0 degrees."
When you are at "180 degrees" (the opposite side), the choir sings a completely different, dissonant harmony.

By using this "choir" of repeating waves (specifically, Fourier embeddings), the computer can instantly recognize that 0 and 359 are neighbors because their "songs" sound very similar. There is no sharp cliff; the transition is smooth.

3. Tuning the "Songs" (The Kernels)

The paper explains that you can control exactly how these "songs" sound by choosing how you pick the notes for your choir. This is called defining the kernel shape.

The authors show two main ways to tune this:

The "Dirichlet" Method (Uniform Sampling): Imagine picking notes for your choir completely at random from a specific range. This creates a pattern that looks like a repeating "sinc" wave (a wave with a big bump in the middle and smaller ripples on the sides). It works well, but the ripples can get a bit messy and negative, which can be confusing for the computer when comparing complex, multi-dimensional directions.
The "Gaussian" Method (Normal Sampling): Imagine picking notes for your choir based on a bell curve, where the middle notes are chosen most often and the extreme notes are chosen rarely. This creates a Periodic Gaussian Kernel.
- The Analogy: Think of this like a soft, fuzzy spotlight. The center is bright (high similarity), and it fades out smoothly in all directions without the messy, negative ripples of the first method. It is "smoother" and more reliable for representing complex spaces.

4. Why This Matters

The authors argue that by using these specific "choir" representations (which they call Spatial Semantic Pointers), we can build computer models that handle circular data much better than before.

They specifically mention that this could help models understand:

Direction and Orientation: Like a robot knowing which way is "up" or "forward" without getting confused at the 360-degree mark.
Circadian Rhythms: Understanding time cycles (day/night).
Musical Patterns: Recognizing notes that wrap around an octave.
Stimulus Perception: Even things like color, which often cycle (red wraps around to red again).

Summary

In short, the paper says: "Stop using straight lines to measure circles." Instead, use a high-dimensional set of repeating waves (like a choir of singers) to represent angles. By carefully choosing how we tune these waves, we can create a smooth, continuous map where the beginning and the end of a cycle connect perfectly, allowing computers to understand the world's natural cycles just like we do.

Technical Summary: On Periodic Distributed Representations Using Fourier Embeddings

Problem Statement
Periodic signals, such as orientation angles, circadian rhythms, and musical patterns, are fundamental to physical and perceptual phenomena. However, representing these signals using scalar, real angular measures (e.g., radians or degrees) introduces significant challenges. Specifically, scalar representations struggle to distinguish or process nearby angles when their absolute difference exceeds $\pi$ , resulting in sharp discontinuities at the limits of the signal's period. For instance, the angular distance between $\epsilon$ and $2\pi - \epsilon$ is physically small, yet their scalar difference is large ( $|2\pi - 2\epsilon|$ ). Existing distributed embedding approaches, such as Random Fourier Features (RFFs), typically focus on continuous feature representations mapping infinite number lines to high-dimensional trajectories, leaving the characterization and analysis of periodic versions of these embeddings and their induced kernels underexplored.

Methodology
The authors propose constructing real-valued, periodic embeddings in high-dimensional space to resolve scalar discontinuities. The core representation is defined as $\phi(\mathbf{x}) = \mathcal{F}^{-1}\{e^{j\mathbf{A}\mathbf{x}}\}$ , where $\mathbf{x} \in \mathbb{R}^n$ , $j$ is the imaginary unit, and $\mathbf{A} \in \mathbb{R}^{d \times n}$ is a conjugate symmetric phase matrix. These embeddings are identified as Spatial Semantic Pointers (SSPs), chosen for their neural feasibility and compositional structure.

To ensure the embeddings are periodic with a specific period $t^\circ \in \mathbb{R}^+$ , all entries of the matrix $\mathbf{A}$ must be integer multiples of period-scaled Fourier roots ( $2\pi n t^{\circ -1}$ ). The set of possible entries, $\Omega$ , is sampled from a discrete set $S^\circ(t^\circ)$ .

The similarity between two points $\mathbf{x}$ and $\mathbf{y}$ is measured via the dot product of their embeddings, which approximates a shift-invariant kernel $K(\mathbf{x} - \mathbf{y})$ . Since the columns of $\mathbf{A}$ are independent and identically distributed (i.i.d.), the $n$ -dimensional kernel is the product of individual 1D kernels: $K(\mathbf{x}) = \prod_{i=1}^n k(x_i)$ . The shape of the individual kernel $k(x)$ is determined by the inverse Fourier transform of the probability mass function (PMF) used to sample the entries of $\mathbf{A}$ .

The paper formalizes two specific kernel shapes based on different sampling distributions for $\mathbf{A}$ :

Dirichlet Kernel: Generated by uniformly sampling integer multiples of the Fourier roots within a frequency band $B$ . The resulting kernel is a normalized Dirichlet kernel (equivalent to a repeating sinc function).
Periodic Gaussian Kernel: Generated by sampling entries from a centered, pseudo-normal distribution over the integers within the same frequency band. The resulting kernel approximates a ratio of theta functions, yielding a periodic Gaussian similarity kernel.

Key Contributions

Formalization of Periodic SSPs: The work provides a formal framework for constructing high-dimensional periodic embeddings using Spatial Semantic Pointers, ensuring that the representations are real-valued and cyclic.
Kernel Characterization: The authors derive the specific mathematical forms of the induced kernels for periodic embeddings. They demonstrate that uniform sampling of phase matrices yields a product of Dirichlet kernels, while normal-like sampling yields a product of periodic Gaussian kernels.
Control over Kernel Shapes: The paper highlights that by controlling the sampling distribution of the phase matrix $\mathbf{A}$ , one can construct a variety of different kernel shapes (e.g., Dirichlet vs. Gaussian) to suit different modeling needs.
Addressing Dimensionality Issues: The authors note that while the product-of-Dirichlets kernel is highly oscillatory and not radially symmetric (making it less ideal for equidistant feature spaces), the product-of-Gaussians kernel offers a smoother, non-negative alternative, though it remains radially asymmetric.

Results
The paper presents theoretical derivations and visualizations (Figure 1) comparing the normalized periodic Gaussian and Dirichlet kernels. The visualizations confirm that:

The Dirichlet kernel (from uniform sampling) exhibits pronounced negative lobes and high oscillation near zero, which can be suboptimal for representing continuous, multidimensional feature spaces where equidistant features should be equally similar.
The Periodic Gaussian kernel (from normal sampling) is smoother and non-negative everywhere, providing a more stable similarity measure, although it retains radial asymmetry similar to its Dirichlet counterpart.

Significance and Claims
The paper claims to offer a method for constructing high-dimensional representations that are inherently cyclic, thereby avoiding the discontinuities associated with scalar angular representations. By characterizing the repeating kernels for high-dimensional feature spaces, the work provides a mechanism to control the nature of dot-product similarities. The authors state that these techniques are intended to be applied to cognitive models involving direction, orientation, and stimulus perception (such as color) to create models capable of exhibiting human-like behaviors. The work serves as a foundational step in formalizing Dirichlet and periodic Gaussian kernels within the context of neurally plausible representation schemes.