Adaptive RBF-KAN: A Comparative Evaluation of Dynamic Shape Parameters in Kolmogorov-Arnold Networks

Technical Summary: Adaptive RBF-KAN

Problem Statement
Kolmogorov-Arnold Networks (KANs) offer an alternative to Multi-Layer Perceptrons (MLPs) by placing learnable univariate functions on network edges rather than using fixed activation functions at nodes. While the original KAN implementation utilizes B-spline bases, this approach incurs significant computational costs during training. FastKAN addresses this by replacing splines with Gaussian Radial Basis Functions (RBFs), significantly improving efficiency. However, standard FastKAN relies on a fixed kernel shape parameter ($h$). The choice of this parameter is critical: small values may cause overfitting, while large values lead to oversmoothing. A fixed parameter limits the model's adaptability across diverse function types (e.g., smooth surfaces, discontinuities, oscillatory patterns, and singularities). Furthermore, the standard FastKAN formulation is restricted to Gaussian kernels, potentially limiting representational flexibility for non-smooth or highly localized functions.

Methodology
The authors propose an Adaptive RBF-KAN (also referred to as adaptive FastKAN) that introduces two primary enhancements to the existing framework:

1. LOOCV-Based Initialization: Instead of using a fixed or randomly initialized shape parameter, the authors estimate the initial kernel scale ($h_{opt}$) using Leave-One-Out Cross-Validation (LOOCV). This is performed on an auxiliary one-dimensional RBF interpolation model constructed from the input data. To ensure numerical stability, a small regularization term ($\lambda = 10^{-9}$) is added to the interpolation matrix. The optimal $h$ is found via a two-stage search (coarse and fine) that minimizes the maximum LOOCV error.
2. Adaptive Shape Parameter Learning: To allow the network to refine the kernel scale during training, the shape parameter is reparameterized in log-space ($\theta = \ln(h)$). This ensures $h = e^\theta$ remains strictly positive, preventing numerical instability (division by zero or exponential growth) during gradient-based optimization. The parameter $\theta$ is updated alongside the spline coefficients via backpropagation.
3. Extended Kernel Family: The framework is extended beyond the Gaussian kernel to include a broader family of radial basis functions. Specifically, the authors introduce Matérn (M2, M4, M6) and Wendland (W2, W4, W6) kernels into the KAN architecture for the first time. These kernels offer varying degrees of smoothness and compact support, allowing the network to better adapt to functions with different structural characteristics.

Key Contributions

• First Integration of LOOCV and Deep KANs: To the best of the authors' knowledge, this is the first study to integrate LOOCV-based kernel scale estimation with the training of deep KAN architectures.
• Novel Kernel Integration: The paper introduces Matérn and Wendland kernels into the KAN framework, moving beyond the Gaussian-only approach of standard FastKAN.
• Data-Driven Initialization Strategy: The proposed method provides a principled, data-driven initialization for the kernel scale, reducing the reliance on expensive hyperparameter searches while allowing for further refinement during training.
• Comprehensive Benchmarking: The study evaluates the proposed method against MLPs, standard KANs, EfficientKAN, Chebyshev KAN, KAF, and standard FastKAN across four distinct benchmark functions representing smooth, discontinuous, oscillatory, and singular behaviors.

Results
Numerical experiments on the unit square domain $[0, 1]^2$ reveal the following:

• Kernel Selection Matters: No single kernel is optimal for all function types.
  • Smooth Functions (Franke): The Gaussian (GA) kernel performs best, consistent with its infinite differentiability.
  • Discontinuities: The Matérn C2 (M2) kernel significantly outperforms GA, capturing sharp jumps with lower error and less blurring.
  • Oscillatory Patterns: The Gaussian kernel fails to capture high-frequency oscillations, whereas the Wendland C2 (W2) kernel reconstructs the pattern accurately.
  • Singularities: The Wendland C6 (W6) kernel reduces error by approximately 52% compared to the GA baseline for localized peaks.
• Adaptivity Improves Performance: The adaptive RBF-KAN consistently outperforms the standard FastKAN with a fixed shape parameter. For instance, on the oscillatory function, the error dropped from $4.54 \times 10^{-1}$ (fixed GA) to $6.24 \times 10^{-2}$ (adaptive W2).
• Efficiency vs. Accuracy Trade-off: While the standard KAN (spline-based) offers high accuracy, it is computationally expensive (training times ~60s vs. ~3-5s for RBF variants). The Adaptive RBF-KAN maintains the computational efficiency of FastKAN while achieving competitive or superior accuracy across all test cases. MLPs were found to be computationally efficient but struggled with accuracy on complex patterns, often over-smoothing high-frequency features.

Significance and Claims
The paper claims that combining LOOCV-based initialization with adaptive kernel learning provides a practical and flexible strategy for improving RBF-based KAN models. The authors emphasize that their work represents the first attempt to connect LOOCV-based scale estimation with deep KAN training. They posit that the proposed method addresses the limitations of fixed kernel choices and fixed shape parameters, enabling the network to better adapt its basis representation to the specific structural characteristics of the target function (smoothness, discontinuity, oscillation, or singularity). The study concludes that while different kernels show advantages for different classes of functions, the adaptive framework offers a robust solution that balances the computational efficiency of FastKAN with the accuracy required for diverse approximation challenges. Future work is suggested to investigate applications in higher-dimensional problems, noisy real-world datasets, and scientific computing tasks such as solving partial differential equations.