FEKAN: Feature-Enriched Kolmogorov-Arnold Networks

This paper introduces Feature-Enriched Kolmogorov-Arnold Networks (FEKAN), a parameter-efficient extension that enhances the computational efficiency, convergence speed, and predictive accuracy of existing KAN architectures through feature enrichment while maintaining their interpretability.

The Gist

Imagine you are trying to teach a robot to understand the complex, chaotic world around it—like predicting the weather, simulating how a bridge vibrates, or modeling how a virus spreads.

For a long time, we used a standard type of robot brain called an MLP (Multilayer Perceptron). It's like a general-purpose worker: it's fast and good at many things, but it sometimes misses the tiny, high-pitched details (like a sudden gust of wind) because it's used to thinking in broad, smooth strokes.

Then, a new type of brain called KAN (Kolmogorov-Arnold Network) arrived. Think of KAN as a master craftsman. Instead of using a blunt hammer (fixed math rules), it uses a set of flexible, learnable tools (curved lines) to sculpt the data. This makes KAN incredibly accurate and easy for humans to understand (interpretable).

But there's a catch: The master craftsman is slow. It takes a long time to learn, and sometimes it gets stuck or confused when the data is too complex or "noisy."

Enter the authors of this paper, who introduce FEKAN (Feature-Enriched KAN).

The Core Idea: Giving the Craftsman a "Cheat Sheet"

Imagine you are trying to solve a difficult puzzle.

• The Old Way (Standard KAN): You are handed a box of plain, square tiles. You have to figure out how to arrange them to make a picture of a swirling galaxy. It's possible, but it takes forever, and you might get the colors wrong.
• The FEKAN Way: Before you start, someone hands you a specialized toolkit. They say, "Here are some pre-cut tiles that already look like swirls, and some that look like stars. Just use these to build your galaxy."

In technical terms, this "specialized toolkit" is called Feature Enrichment.
FEKAN takes the raw input (like time and space coordinates) and instantly transforms it into a richer, more detailed version before it even reaches the main brain. It adds things like sine waves, cosine waves, or polynomial curves right at the door.

Why is this a Big Deal?

The paper shows that FEKAN is like giving the master craftsman a superpower without making them slower or more expensive to hire. Here's what happens when you use FEKAN:

1. It Sees the "High-Frequency" Details:
   Standard KANs often ignore rapid changes (like high-pitched sounds or sharp spikes in a graph). This is called "spectral bias." FEKAN fixes this. By pre-adding the "swirl" tiles, the brain doesn't have to struggle to learn them from scratch. It can focus on the big picture while the details are already there.

   • Analogy: It's like trying to draw a jagged mountain range. A standard KAN tries to draw it with smooth, curved lines and fails. FEKAN is given a jagged pencil right away, so it draws the mountain perfectly in half the time.

2. It's Faster and More Stable:
   Because the brain doesn't have to work as hard to figure out the basics, it learns faster. The paper tested this on many difficult math problems (Partial Differential Equations) and found that FEKAN converged (finished learning) much quicker than the original KAN.

   • Analogy: If you are trying to run a marathon, a standard KAN is running in heavy boots. FEKAN is running in lightweight, aerodynamic shoes. They are the same runner, but FEKAN gets to the finish line much sooner.

3. It Doesn't Forget (Continual Learning):
   One of the biggest problems with AI is "catastrophic forgetting"—learning a new task makes the AI forget the old one. KANs are already good at remembering, but FEKAN is even better.

   • Analogy: Imagine a library. When you add a new book (new data), a standard KAN might accidentally knock over the old books. FEKAN has a special shelf system (the feature enrichment) that keeps the new book organized without disturbing the old ones.

4. It Handles "Messy" Math Better:
   The paper tested FEKAN on some math problems that usually make other AI models crash or produce errors (like the Chebyshev basis functions). FEKAN acted as a stabilizer, keeping the training smooth and error-free.

The Bottom Line

The authors didn't just build a bigger, heavier brain. They built a smarter interface.

They showed that by simply changing how the data enters the system (adding a "feature enrichment" layer), you can get:

• Higher Accuracy: Better predictions.
• Faster Speed: Less training time.
• Better Stability: Fewer crashes and errors.
• Same Cost: It doesn't require more computing power or memory.

In short: FEKAN takes the already impressive "master craftsman" (KAN) and gives it a set of pre-made, high-quality tools. The result is a system that solves complex scientific problems faster, more accurately, and more reliably than ever before, all while remaining easy for humans to understand.

Technical Summary

1. Problem Statement

Kolmogorov-Arnold Networks (KANs) have emerged as a promising alternative to Multilayer Perceptrons (MLPs) due to their interpretability (via functional decomposition) and parameter efficiency. However, vanilla KAN architectures suffer from significant limitations:

• High Computational Cost & Slow Convergence: Training KANs is substantially more expensive and slower than training MLPs.
• Spectral Bias: Like MLPs, KANs struggle to learn high-frequency components and fine-scale structures, often failing to capture complex oscillatory behaviors or discontinuities.
• Instability: Certain KAN variants (e.g., those using Chebyshev or Wavelet bases) frequently exhibit training divergence or numerical instability.
• Scalability: These issues limit the practical applicability of KANs in scientific machine learning (SciML), particularly for solving Partial Differential Equations (PDEs) and learning neural operators.

The paper posits that while feature enrichment is a standard technique in classical machine learning (e.g., kernel methods, Fourier feature mappings for MLPs), it has not been systematically explored within the KAN framework to address these specific bottlenecks.

2. Methodology: Feature-Enriched KAN (FEKAN)

The authors propose FEKAN, a simple yet effective extension of the vanilla KAN architecture that incorporates feature enrichment without increasing the number of trainable parameters in the core network.

Core Architecture

• Feature Map ($\gamma$): Before the input $x$ enters the KAN backbone, it is transformed into an enriched feature space $\mathcal{F} \subset \mathbb{R}^{n+m}$ via a mapping $\gamma: \mathbb{R}^n \to \mathbb{R}^{n+m}$.
• Enrichment Strategy: The mapping typically appends nonlinear basis functions (e.g., Fourier terms like $\sin(ax), \cos(ax)$, polynomials, or interaction terms) to the original input vector.
  • Example: $\gamma(x) = [x, \sin(a_1 x), \cos(a_1 x), \dots]^\top$.
• Integration: The enriched features are fed into the standard KAN layers. The underlying KAN structure (learnable univariate activation functions on edges) remains unchanged, preserving interpretability.
• Physics-Informed Extension (PI-FEKAN): For PDE solving, the feature map is applied to spatiotemporal coordinates $(x, y, z, t)$ before the loss function (residual, boundary, and initial conditions) is computed.

Theoretical Foundations

The paper establishes rigorous theoretical backing for FEKAN:

1. Feature-Enriched Kolmogorov Superposition Theorem: The authors extend the classical theorem to show that any continuous function of inputs and additional continuous features can be represented as a composition of univariate functions. This implies FEKAN can approximate high-dimensional functions with error bounds independent of dimension, mitigating the "curse of dimensionality."
2. Representation Capacity: Theorem 2 proves that feature enrichment strictly enlarges the family of representable functions ($\mathcal{R}_n \subsetneq \mathcal{R}_{n,m}$) and reduces the complexity required to approximate certain targets (e.g., high-frequency functions).
3. Neural Tangent Kernel (NTK) Analysis: The authors analyze NTK drift to explain training dynamics. They find that vanilla KANs exhibit rapid eigenvalue decay in the NTK, leading to spectral bias (ignoring high frequencies). FEKAN slows this decay, allowing the model to prioritize and learn high-frequency components effectively.

3. Key Contributions

1. Novel Architecture: Introduction of FEKAN, which enhances KANs via input feature enrichment while maintaining the original architecture's interpretability and parameter efficiency.
2. Theoretical Framework: Formal proof of the Feature-Enriched Kolmogorov Superposition Theorem and analysis of representation capacity gains and Rademacher complexity.
3. Comprehensive Benchmarking: Extensive evaluation across:
   • Function Approximation: High-frequency and discontinuous test functions.
   • Dynamical Systems: Lorenz attractor (chaotic ODEs).
   • PDE Solving: Helmholtz, Allen-Cahn, and Klein-Gordon equations (steady and time-dependent).
   • Neural Operators: Learning mappings for high-frequency bubble dynamics (DeepOKAN vs. DeepOFEKAN).
4. Stability Mechanism: Demonstration that feature enrichment stabilizes training for unstable bases (like Chebyshev) without requiring manual scaling or architectural overhauls.
5. Continual Learning: Evidence that FEKAN suppresses catastrophic forgetting in boundary value problems better than vanilla KANs.

4. Key Results

The paper presents empirical evidence across multiple domains showing FEKAN consistently outperforms vanilla KAN and its variants (FastKAN, WavKAN, ReLUKAN, ChebyKAN, etc.):

• Accuracy & Convergence:

  • Function Approximation: FEKAN achieves order-of-magnitude reductions in relative $L_2$ error compared to KAN for high-frequency and discontinuous functions. It converges significantly faster.
  • PDE Solving: In PI-FEKAN experiments, the method reduced errors by 50% to 99% compared to PI-KAN. For example, on the Helmholtz equation with Chebyshev bases, FEKAN achieved stable training where PI-KAN diverged or produced high errors.
  • Neural Operators: In learning high-frequency bubble dynamics, enriching the branch network in DeepOFEKAN reduced errors by an order of magnitude compared to DeepOKAN.

• Computational Efficiency:

  • FEKAN maintains comparable computational costs (seconds per iteration) to vanilla KAN despite the added feature map, as the enrichment layer is a fixed, non-trainable transformation.
  • It achieves higher accuracy with fewer trainable parameters or fewer training epochs, leading to better overall efficiency.

• Stability & Robustness:

  • Chebyshev Instability: While ChebyKAN often diverges (NaNs) during training, FEKAN with Chebyshev bases converges stably.
  • Spectral Bias: FEKAN overcomes the inherent spectral bias of KANs, successfully capturing high-frequency structures that vanilla KANs miss.
  • Continual Learning: In phase-wise boundary data introduction, FEKAN retained learned boundary conditions across phases with minimal forgetting, whereas PI-KAN exhibited significant fluctuations and loss of prior knowledge.

• Random Feature Enrichment: The study showed that sampling feature frequencies from a Gaussian distribution (Random Fourier Features) is often superior to deterministic selection, reducing the hyperparameter search space to a single variance parameter ($\sigma$).

5. Significance and Implications

• Bridging the Gap: FEKAN makes KANs a viable option for large-scale scientific computing by addressing their primary weakness: computational inefficiency and slow convergence.
• General Principle: The paper establishes "feature enrichment" as a universal principle for improving KAN-based models, applicable regardless of the specific basis function (spline, wavelet, Chebyshev, etc.) used in the network.
• Scientific Foundation Models: By improving scalability and stability, FEKAN positions KANs as a potential core component for Scientific Foundation Models (SciFMs), offering an interpretable alternative to black-box MLPs.
• Practical Applicability: The method requires no changes to the training algorithm (backpropagation remains the same) and no increase in trainable parameters, making it a "drop-in" upgrade for existing KAN implementations.

In conclusion, FEKAN successfully enhances the Kolmogorov-Arnold Network framework, transforming it from a theoretically interesting but computationally heavy architecture into a robust, efficient, and highly accurate tool for scientific machine learning.