🤖 machine learning

Kolmogorov-Arnold Fourier Networks

The paper introduces Kolmogorov-Arnold Fourier Networks (KAF), a novel architecture that overcomes the parameter explosion and high-frequency limitations of traditional KANs by replacing local B-splines with a global, adaptive spectral representation using trainable Random Fourier Features and a hybrid activation mechanism, thereby achieving state-of-the-art performance and efficiency across diverse tasks.

Original authors: Jusheng Zhang, Yijia Fan, Kaitong Cai, Keze Wang, Wenhao Wang

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jusheng Zhang, Yijia Fan, Kaitong Cai, Keze Wang, Wenhao Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Engineered" Machine

Imagine you are trying to teach a robot to recognize a cat in a photo.

Old Way (MLP): You give the robot a standard toolkit. It's fast and simple, but sometimes it misses fine details like the texture of fur or the sharp edge of an ear.
The "New" Way (KAN): A few years ago, researchers invented a new toolkit called KAN (Kolmogorov-Arnold Network). Theoretically, this toolkit is a genius. It can describe any shape or pattern perfectly. However, in practice, it's like trying to build a house with a million tiny, custom-made bricks.
- The Issue 1 (Too Heavy): It requires so many parameters (bricks) that it becomes incredibly heavy and slow to train. It's like trying to carry a backpack full of gold bricks; it's too much weight for the robot to handle efficiently.
- The Issue 2 (Blurry Vision): Because KAN uses "smooth" curves (called B-splines) to build its shapes, it struggles to see sharp, jagged, or "high-frequency" details. It's like looking at a photo through a soft-focus filter; the cat looks nice, but the whiskers are blurry.

The Solution: KAF (The "Spectral" Upgrade)

The authors propose a new architecture called KAF (Kolmogorov-Arnold Fourier Network). They didn't just tweak the old KAN; they fundamentally changed how it "thinks" about data.

Think of KAF as taking that heavy, blurry KAN toolkit and swapping the bricks for musical notes.

1. Swapping Bricks for Music (Spectral Reparameterization)

Instead of building shapes out of millions of tiny, local pieces (B-splines), KAF builds them using Random Fourier Features (RFF).

The Analogy: Imagine you want to draw a complex wave.
- KAN tries to draw it by connecting thousands of tiny, smooth dots. It takes forever and uses too much ink.
- KAF realizes that any wave is just a combination of simple sine waves (like musical notes). Instead of drawing dots, it just tunes a few "knobs" to adjust the pitch and volume of these notes.
The Result: This drastically reduces the number of "bricks" (parameters) needed. The robot becomes much lighter and faster, but it can still build complex shapes.

2. The "Hybrid" Engine (GELU-Fourier Activation)

KAF introduces a special engine that runs on two types of fuel at the same time:

Fuel A (GELU): This is the "smooth" fuel. It handles the big, easy parts of the picture (like the general shape of the cat).
Fuel B (Fourier/RFF): This is the "sharp" fuel. It handles the tricky, high-frequency details (like the whiskers and fur texture).
The Magic: The robot has a smart switch (learnable coefficients) that decides how much of each fuel to use. At the start of training, it uses mostly the smooth fuel to get the basics right. As it learns, it gradually turns up the "sharp" fuel to capture the fine details. It's like a chef who starts with a basic broth and slowly adds spices to perfect the flavor.

3. Tuning the Radio (Initialization Strategy)

The paper mentions a specific number, 1.64, used to set up the "radio stations" (frequencies) the robot listens to.

The Analogy: If you tune a radio too far to the left, you only hear bass (low frequencies). If you tune too far right, you only hear static (high frequencies).
The Fix: The authors calculated that setting the "volume" of the frequencies to 1.64 is the perfect balance. It ensures the robot listens to the right mix of sounds from the very beginning, so it doesn't get confused or miss important signals.

What Did They Prove?

The authors tested this new "Musical Robot" (KAF) against the old robots (MLP and KAN) in several arenas:

Vision (Seeing): On tasks like recognizing handwritten numbers or complex images (CIFAR, ImageNet), KAF was more accurate and used fewer resources than the others.
Language (Reading): When used inside a language model (GPT-2), KAF learned to predict text better and faster than the standard version.
Science (Solving Equations): They asked the robot to solve complex physics equations (like how heat spreads or how waves move). KAF was much better at capturing the sharp, sudden changes in these equations compared to the blurry KAN.
Fitting Shapes: When asked to draw a perfect sine wave (a smooth curve that goes up and down), KAF drew it almost perfectly, while the others made mistakes at the edges.

The Bottom Line

The paper claims that KAF solves the two biggest problems of the previous "genius" network (KAN):

It is lightweight (fewer parameters, faster training).
It has sharp vision (it can see high-frequency details that others miss).

By replacing the heavy, smooth "bricks" of the past with a flexible, musical "Fourier" approach, KAF manages to be both efficient and powerful, making it a strong candidate for the next generation of AI.

Technical Summary: Kolmogorov-Arnold Fourier Networks (KAF)

Problem Statement
While Kolmogorov-Arnold Networks (KANs) offer strong theoretical expressiveness by replacing fixed activation functions with learnable B-spline basis functions, they face two critical barriers to practical adoption in high-dimensional tasks:

Parameter Explosion: The dual-matrix architecture (activation function matrix and B-spline coefficient matrix) causes parameter counts to scale significantly higher than traditional Multi-Layer Perceptrons (MLPs), leading to high storage and computational costs.
Spectral Limitations: The inherent smoothness of B-spline basis functions limits the ability to capture high-frequency features, resulting in suboptimal performance on data with rich spectral characteristics, such as natural images, audio, and complex differential equations.

Methodology
The authors propose the Kolmogorov-Arnold Fourier Network (KAF), which fundamentally redefines the KAN paradigm through spectral reparameterization. The core architecture replaces the local, grid-based B-spline representation with a global, adaptive spectral representation.

Key methodological components include:

Spectral Reparameterization: KAF merges the large matrices of KAN using the associative property of matrix multiplication, reducing parameter complexity from $O(d_{in} \times d_{out} \times (G + K + 3))$ to $O(d_{in} \times d_{out})$ .
Trainable Random Fourier Features (RFF): Instead of B-splines, KAF employs learnable RFFs to map inputs into a Reproducing Kernel Hilbert Space (RKHS). The frequency matrix $W$ is initialized via a spectral alignment strategy (specifically $\sigma = 1.64$ ) derived from the Central Limit Theorem and Fourier analysis of the GELU activation function. This initialization aligns the RFF spectrum with the prior knowledge of natural signals, mitigating spectral leakage.
Hybrid GELU-Fourier Activation: The network utilizes a dynamic activation mechanism defined as $H(x) = \alpha \odot \text{GELU}(x) + \beta \odot \psi_{\text{RFF}}(x)$ . The coefficients $\alpha$ and $\beta$ are learnable scaling parameters. During training, $\alpha$ prioritizes low-frequency smoothness initially, while $\beta$ (initialized to a small value, e.g., $10^{-2}$ ) progressively grows to capture high-frequency components, enabling an automatic transition from fixed activation to hybrid Fourier-symbolic representations.

Key Contributions

Basis Transformation: A shift from local B-splines to global, adaptive spectral representations, reducing parameter complexity while preserving expressiveness.
Spectral Alignment Initialization: An RFF initialization strategy ( $\sigma = 1.64$ ) that breaks the smoothness limitations of fixed kernels, allowing accurate capture of high-frequency components.
Adaptive Hybrid Activation: A mechanism that dynamically balances low-frequency smoothness and high-frequency representation during training, ensuring stability and efficiency.

Experimental Results
Comprehensive experiments were conducted across Computer Vision (CV), Natural Language Processing (NLP), audio, and scientific computing tasks:

Benchmark Performance: On vision datasets (MNIST, CIFAR-10/100, SVHN), KAF consistently achieved higher accuracy than MLPs, KANs, FAN, and GPKAN under comparable parameter budgets.
Large-Scale Models: When integrated into ResNet-18, ViT-Tiny, and MLP-Mixer, KAF improved Top-1 accuracy (e.g., 91.72% on CIFAR-10 for ResNet-18 vs. 91.19% for MLP) while maintaining reasonable computational costs.
Language Modeling: In GPT-2 architectures, replacing the Feed-Forward Network (FFN) with KAF reduced Perplexity (PPL) on WikiText (180.85 vs. 184.53 for MLP) and OpenWebText, while significantly reducing training time compared to KAN, which suffered from instability and massive parameter overhead.
Function Approximation & PDEs: KAF demonstrated superior fitting capabilities on complex functions (including high-frequency sums and discontinuous functions) and achieved lower errors in solving Partial Differential Equations (Poisson, Heat, Burgers) compared to MLPs and other baselines.
Ablation Studies: Experiments confirmed the necessity of the hybrid GELU-RFF activation and the specific initialization of $\sigma = 1.64$ , showing that removing either component degrades performance.

Significance
The paper claims that KAF successfully bridges the gap between the theoretical interpretability of Kolmogorov-Arnold networks and the parameter efficiency required for scalable deep learning. By addressing the "paradox between theory and practice" inherent in KANs, KAF offers a practical alternative that retains the function-approximation perspective of KANs while achieving state-of-the-art performance with improved efficiency. The authors position KAF as a viable replacement for both KAN and MLP components in modern neural architectures, particularly for tasks requiring high-frequency feature extraction.