💻 computer science

Scaling of Gaussian Kolmogorov--Arnold Networks

This paper establishes that the Gaussian scale parameter $\epsilon$ in Gaussian Kolmogorov--Arnold Networks is primarily governed by the first layer's feature geometry, proposing a practical operating interval of $\epsilon \in [\frac{1}{G-1}, \frac{2}{G-1}]$ as a robust design rule that ensures stability and competitiveness across various architectures and applications.

Original authors: Amir Noorizadegan, Sifan Wang

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amir Noorizadegan, Sifan Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize patterns in a complex painting. You give the robot a set of "lenses" (mathematical tools) to look at the painting. Some lenses are sharp and focused on tiny details, while others are blurry and see the whole picture at once.

This paper is about Gaussian Kolmogorov–Arnold Networks (Gaussian KANs), a type of AI that uses these "lenses" (called Gaussian Radial Basis Functions) to learn. The big question the authors asked was: "How blurry or sharp should these lenses be?"

In technical terms, this "blurriness" is controlled by a number called $\epsilon$ (epsilon). If $\epsilon$ is too small, the lenses are too sharp and the robot gets confused by noise. If $\epsilon$ is too big, the lenses are so blurry that the robot can't see any details at all.

Here is the simple breakdown of their discovery:

1. The "First Impression" Rule

The most important finding of this paper is that the first layer of the network is the boss.

Think of the AI network as a relay race. The first runner (the first layer) receives the raw data (the painting). If the first runner drops the baton or runs the wrong way, it doesn't matter how fast the second or third runners are; the team will lose.

The Discovery: The authors found that the "blurriness" ( $\epsilon$ ) of the first layer determines the entire team's success. If the first layer gets confused because the lenses are too blurry, the later layers can never fix it.
The Analogy: Imagine trying to read a book through a foggy window. If the window is too foggy (bad $\epsilon$ ), you can't see the words. No matter how much you squint or how good your eyesight is later, you still can't read. You have to clean the window first.

2. The "Goldilocks" Zone

The authors spent a lot of time testing different levels of blurriness. They found that there isn't just one perfect number; there is a safe range where the AI works best.

They discovered a simple rule based on how many "lenses" (centers) you have:

Too Sharp (Small $\epsilon$ ): The lenses are so focused they only see tiny specks. The AI misses the big picture.
Too Blurry (Large $\epsilon$ ): The lenses are so wide that everything looks like a gray smear. The AI loses all detail and the math becomes unstable (like trying to balance a tower of Jenga blocks that are all the same color).
Just Right: The sweet spot is when the "blur" of one lens overlaps nicely with its neighbor, but not so much that they merge into a blob.

The Magic Formula:
They found that the perfect range is roughly between $1/(G-1)$ and $2/(G-1)$ , where $G$ is the number of lenses.

Simple translation: If you have 20 lenses, your "blur" setting should be somewhere between 0.05 and 0.1. It's a simple recipe that works almost every time, no matter what problem you are solving.

3. Why This Matters (The "Feature, Not a Bug" Idea)

In the past, scientists thought that how sensitive these AI models were to the "blur" setting was a flaw (a bug). They thought, "If the model breaks when I change one number, the model is bad."

The authors argue the opposite: This sensitivity is a superpower.

The Analogy: Think of a radio tuner. If the radio is too sensitive, it picks up static. If it's not sensitive enough, it picks up nothing. But if you tune it to the exact right frequency, the music is crystal clear.
The authors say: "Don't fear the sensitivity. Use it!" By understanding exactly how the "blur" affects the first layer, we can tune the radio to the perfect station without guessing.

4. Practical Benefits

This paper gives engineers and scientists a "cheat sheet" so they don't have to guess:

No more guessing: You don't need to try 100 different settings. Just use the formula based on how many lenses you have.
Faster training: You can find the right setting very quickly by looking at the error in the first few seconds of training, rather than waiting hours.
Works everywhere: They tested this on everything from simple math problems to complex physics equations (like sound waves in a room), and the rule held up every time.

Summary

This paper is like finding the perfect focus ring for a camera lens. Before, photographers (AI researchers) had to guess and check to get a sharp photo. Now, the authors say: "Here is a simple rule. Set your focus ring to this specific range based on your lens count, and you will get a perfect picture every time, whether you are taking a photo of a flower or a landscape."

It turns a mysterious, difficult math problem into a simple, reliable design rule.

1. Problem Statement

Kolmogorov–Arnold Networks (KANs) are a structured alternative to Multilayer Perceptrons (MLPs) that replace fixed node activations with learnable univariate functions on edges. While Gaussian Radial Basis Functions (RBFs) have been proposed as a basis for KANs (e.g., in FastKAN) due to their smoothness and computational efficiency, a critical gap remains: how to systematically select the Gaussian scale parameter ( $\epsilon$ ).

In existing literature, $\epsilon$ is often chosen heuristically, via trial-and-error, or made trainable without theoretical grounding. In classical RBF theory, the scale parameter governs a trade-off between approximation power and numerical stability (conditioning). Poor choices lead to either weak approximation (if $\epsilon$ is too small) or severe ill-conditioning and representation collapse (if $\epsilon$ is too large). The paper addresses the lack of a principled method for selecting $\epsilon$ specifically within the context of deep, edge-based Gaussian KAN architectures.

2. Methodology

The authors approach the problem by analyzing the first layer of the Gaussian KAN, arguing it is the decisive bottleneck for the entire network's behavior.

Theoretical Framework:
- The paper models the Gaussian KAN layer as a feature map $\Phi$ followed by a linear coefficient map.
- It establishes that the first layer acts directly on the input domain. If the first layer loses the distinguishability of inputs (due to poor scaling), subsequent layers cannot recover this information.
- The authors analyze the feature matrix $\Phi$ (rather than the Gram matrix $\Phi^\top\Phi$ ) as the primary object for stability analysis. They derive that the condition number of the Gram matrix is the square of the feature matrix's condition number ( $\kappa(\Phi^\top\Phi) = \kappa(\Phi)^2$ ).
- They define a "collapse" regime where large $\epsilon$ causes the feature matrix to approach a rank-one limit, rendering the network unable to distinguish inputs.
Experimental Setup:
- Benchmarks: Four regression targets ( $F_1$ to $F_4$ ) covering smooth, periodic, sharp, and discontinuous functions, plus a Physics-Informed Neural Network (PINN) Helmholtz equation.
- Architecture: Standard KAN structures (e.g., $[2, 12, 12, 1]$ ) with varying depths and widths.
- Search Strategy: A brute-force sweep of $\epsilon$ over a logarithmic range to establish a ground truth for error behavior, compared against conditioning-based diagnostics.
- Diagnostics: The study tracks Validation RMSE, the spectral condition number $\kappa(\Phi)$ , and the onset of numerical rank loss.

3. Key Contributions

A. Identification of the First-Layer Bottleneck

The paper proves theoretically and demonstrates experimentally that the first layer is the dominant factor in scale selection.

Proposition: If two inputs become indistinguishable after the first-layer feature map (due to large $\epsilon$ ), the entire network output becomes identical for those inputs.
Evidence: Varying $\epsilon$ only in the first layer replicates the error curve of varying $\epsilon$ across all layers. Conversely, collapsing deeper layers has a negligible effect compared to collapsing the first layer.

B. Derivation of a Practical Operating Interval

Based on the conditioning of the first-layer feature matrix, the authors derive a simple, parameter-free rule for selecting $\epsilon$ .

Lower Bound: Determined by the spacing of Gaussian centers. If $\epsilon$ $ϵ$ is smaller than the center spacing, the basis is too localized.
- $\epsilon_{min} = \frac{1}{G-1}$ (where $G$ is the number of centers).
Upper Bound: Determined by the onset of numerical rank loss and ill-conditioning. The authors find that the condition number threshold ( $\kappa \approx 3 \times 10^3$ $κ \approx 3 \times 1 0^{3}$ ) and the onset of rank loss occur simultaneously.
- Empirical fitting shows the upper bound scales as $\epsilon_{max} \approx \frac{2}{G-1}$ .
The Rule: The recommended operating interval is:
$\epsilon \in \left[ \frac{1}{G-1}, \frac{2}{G-1} \right]$
This rule depends primarily on the grid resolution ( $G$ ) and is largely independent of the number of collocation points ( $N$ ) or input dimension ( $d$ ).

C. Validation Across Domains

The proposed interval is validated across:

Different Target Functions: Smooth, oscillatory, sharp, and discontinuous functions.
Collocation Density ( $N$ ): Increasing $N$ improves accuracy but does not shift the optimal $\epsilon$ window significantly.
Network Architecture: Deeper or wider networks do not shift the left boundary of the optimal $\epsilon$ (controlled by the first layer), though they may broaden the right boundary.
Input Dimension: The U-shaped error curve and optimal $\epsilon$ range remain consistent from 1D to 4D.
Physics-Informed Learning: The rule holds for the Helmholtz equation, even with varying frequency parameters ( $\lambda$ ).

D. Practical Applications

Variable Scales: The paper demonstrates that assigning random variable scales to centers within the derived interval can outperform a single fixed scale.
Efficient Search: The training Mean Squared Error (MSE) at early epochs (e.g., 500) correlates strongly with the final validation error, allowing for rapid scale selection without full training or ground truth labels.
Chebyshev Comparison: A properly scaled Gaussian KAN achieves accuracy competitive with, and sometimes superior to, a matched Chebyshev KAN (same number of trainable coefficients), without the rigid stability issues often associated with high-degree polynomials.

4. Results

Error Landscape: The validation error exhibits a distinct U-shape with respect to $\epsilon$ . Errors are high for very small $\epsilon$ (localization) and very large $\epsilon$ (collapse/ill-conditioning).
Optimal Range: The minimum error consistently falls within the interval $[\frac{1}{G-1}, \frac{2}{G-1}]$ .
Conditioning: The condition number of the first-layer feature matrix rises sharply as $\epsilon$ exceeds $\frac{2}{G-1}$ , coinciding with the degradation of model accuracy.
Robustness: The rule holds regardless of whether the scale is shared across layers or fixed per layer, though shared scales are recommended for simplicity.

5. Significance

This paper fundamentally shifts the perspective on Gaussian KANs from treating $\epsilon$ as an arbitrary hyperparameter to viewing it as a design principle governed by the geometry of the input domain and the first layer.

Stability vs. Accuracy: It operationalizes the "conditioning-accuracy trade-off" in deep learning, showing that pushing the conditioning to the limit of numerical stability (within the safe regime) yields the best approximation.
Practical Utility: The derived formula $\epsilon \in [\frac{1}{G-1}, \frac{2}{G-1}]$ provides an immediate, robust starting point for practitioners, eliminating the need for expensive hyperparameter sweeps.
Theoretical Extension: It extends classical RBF scaling theory (specifically the work of Larsson, Schaback, and Fasshauer) into the neural network setting, confirming that the strong dependence on scaling is a "feature" that, when understood, allows for precise control over model behavior.

In summary, the paper provides a rigorous, experimentally validated framework for scaling Gaussian KANs, ensuring that these networks are both numerically stable and highly accurate across a wide range of scientific machine learning tasks.