💻 computer science

Dictionary-KAN: Resolving the Optimization Paradox of Kolmogorov-Arnold Networks via Complex RKHS, Machine-Verified Theory, and Discrete Hierarchical Refinement

This paper introduces Dictionary-KAN (DKAN), a machine-verified architecture that resolves the optimization paradox of Kolmogorov-Arnold Networks by employing complex-coefficient RBF dictionaries and discrete hierarchical refinement to achieve superior multivariate regression, PDE coefficient recovery, and hardware-efficient interpretability while avoiding the memory and convergence issues of continuous spline-based KANs.

Original authors: Kiarash Mohammadi

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kiarash Mohammadi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern artificial intelligence, a persistent challenge has long haunted researchers: how to build machines that learn complex patterns without losing their way in a sea of mathematical confusion. For decades, the standard approach has relied on massive, dense grids of connections, where every part of the system talks to every other part. While powerful, these systems often struggle to find the most efficient path to a solution, getting stuck in local traps or requiring so much memory that they simply cannot run on available hardware. A newer idea, known as the Kolmogorov-Arnold representation, offered a different path. It suggested that any complex, multi-dimensional relationship could be broken down into a series of simpler, one-dimensional steps added together. This concept promised a more elegant and interpretable way to model the world, but when scientists tried to build it, they ran into a fundamental paradox. The mathematical tools they used to make these networks flexible were too unstable, causing the learning process to collapse or become impossibly expensive to compute.

A researcher named Kiarash Mohammadi has now proposed a solution to this paradox with a new architecture called Dictionary-KAN. Instead of trying to stretch and warp a continuous grid of data points, which often leads to the instability that plagued previous attempts, this new system uses a fixed, pre-defined set of building blocks. Imagine a dictionary of words that never changes; the network simply learns how to mix these words together to form sentences, rather than trying to invent new letters on the fly. By anchoring every connection in this stable dictionary, the researcher ensures that the mathematical problem the computer solves is always smooth and predictable, avoiding the sudden crashes and lost progress that occur in older models. This approach allows the system to grow larger and more detailed without forgetting what it has already learned, a feat that was previously impossible with these types of networks.

The innovation goes deeper than just stability. The researcher lifted the entire system into a complex mathematical space, allowing the network to naturally understand how different variables multiply and interact with one another. In previous versions, the system had to be forced to learn these interactions through heavy, inefficient calculations. Here, the structure itself handles the multiplication, making the learning process far more efficient. This design also includes a unique method for refinement. When the network needs to become more precise, it can insert new layers of detail between existing ones without disturbing the work already done. The new parts start with zero influence, ensuring that the network's output remains exactly the same before and after the expansion, effectively eliminating the problem of "catastrophic forgetting" where learning new things erases old knowledge.

To ensure that these claims were not just theoretical hopes, the researcher subjected the core logic to a rigorous, machine-verified check. Using a specialized computer program designed to prove mathematical truths, every step of the optimization theory was verified to be correct under specific conditions. The computer confirmed that the system has a single, unique best solution and that the method used to find it will always converge to that solution without getting stuck. This level of certainty is rare in the field, where many theories rely on assumptions that are difficult to prove. The results of this verification were then put to the test in a series of real-world simulations. On tasks involving the interaction of multiple variables, the new system outperformed the standard dense models by a factor of twenty, achieving much higher accuracy with far fewer resources.

The system also demonstrated remarkable capabilities in scientific discovery. When asked to identify the laws governing a swinging pendulum with air resistance, the network successfully isolated the correct physical variables, including the subtle damping force that other models missed. Similarly, when tasked with uncovering the equations behind a fluid dynamics problem known as the Burgers equation, it reconstructed the correct mathematical relationships with less than one percent error, even though it had to invent the necessary multiplication terms on its own. In a practical application, the researchers trained the network to model the way light reflects off a rough metal surface, a task critical for realistic computer graphics. The resulting model was so clean and structured that it could be translated directly into a short, human-readable computer program, removing the need for the heavy neural network software entirely.

However, the research is not without its limitations, and the author is careful to report them with the same clarity as the successes. When the system was tested on data with sharp, sudden jumps, such as a step function, it exhibited a known weakness called ringing, where the output oscillates slightly around the sharp edge. While the new system handled this better than previous versions, it could not match the precision of simpler, older models in these specific cases. Furthermore, the mathematical guarantees provided by the machine verification rely on certain conditions being met, such as the data having a specific structure, which means the theory is not a universal guarantee for every possible dataset. The researcher explicitly notes that the system is not designed to be a perfect memory for every detail, but rather a tool for understanding and refining complex relationships.

The work represents a significant step forward in making artificial intelligence more reliable and efficient. By replacing unstable, continuous grids with a fixed dictionary of functions, the researcher has resolved a long-standing optimization paradox that has hindered the development of these networks. The ability to grow the network without forgetting, to discover physical laws with high precision, and to compile the final result into simple code suggests a future where these systems are not only powerful but also understandable and practical. The findings are presented not as a final answer to all problems, but as a stable foundation upon which more advanced versions can be built, with the core theory already verified by a machine to be mathematically sound.

Technical Summary: Dictionary-KAN

Problem Statement

The paper addresses the "optimization paradox" inherent in continuous Kolmogorov-Arnold Networks (KANs). While the Kolmogorov-Arnold representation theorem guarantees that any multivariate continuous function can be decomposed into 1D functions and additions, standard continuous KAN implementations (using B-splines, Chebyshev polynomials, or learnable Gaussian centers) encounter four critical failures in practice:

Basis Function Pathologies: Bounded rational functions cause vanishing gradients, while high-degree polynomials create corrugated loss landscapes with erratic zero-crossings.
Grid Instability: Dynamic re-gridding of spline knots leads to non-convex sorting collapses and domain-warping, causing catastrophic forgetting during localized updates.
Violation of KA Constraints: Modeling non-linear variable interactions (e.g., $x_1x_2$ ) often requires dense linear projections or gating, effectively reintroducing the dense weight matrices of Multi-Layer Perceptrons (MLPs) and breaking the 1D summation structure.
Regularization Intractability: Standard Sobolev $L_2$ penalties blur sharp transitions, while strict Lipschitz bounds overly constrain expressivity.

Furthermore, continuous KANs suffer from a "memory curse of dimensionality" where dense grid activations scale poorly with network width, often leading to out-of-memory (OOM) failures.

Methodology: Dictionary-KAN (DKAN)

The author proposes Dictionary-KAN, an architecture that replaces continuous, learnable grid warping with a fixed, discrete dictionary approach grounded in functional analysis. The core design commitments are:

1. Fixed RKHS Dictionaries with Complex Coefficients

Instead of learnable centers, each edge function is defined as a fixed Gaussian (magnitude-RBF) dictionary in a Reproducing Kernel Hilbert Space (RKHS). The coefficients are learnable complex numbers.

Edge Function: $\phi(z) = w_{b1}z + w_{b2}z^2 + \sum_{m=1}^M w_m \exp(-\gamma |z - c_m|^2)$ , where $c_m$ are fixed real centers and $w$ are complex weights.
Complex Routing: Inputs are lifted into the complex plane ( $x_p \mapsto x_p + 0i$ ). Analytic base terms ( $z, z^2$ ) naturally isolate multiplicative cross-products (e.g., $z^2 = (x+iy)^2 = x^2 - y^2 + 2ixy$ ) without requiring dense matrix projections.

2. Discrete Hierarchical Refinement (DHR)

To increase capacity without retraining or disrupting existing representations, DKAN employs DHR. This process inserts midpoint centers between existing dictionary centers. Crucially, new weights are initialized to exactly zero, ensuring the network output remains bit-identical before and after refinement. This guarantees "zero catastrophic forgetting."

3. Discrete Total-Variation (TV) Regularization

To enforce sparsity and piecewise-linear weight profiles, an $\ell_1$ penalty is applied to the second discrete differences of the weight sequence: $R_{TV}(w) = \lambda_{TV} \sum |w_{m+1} - 2w_m + w_{m-1}|$ .

4. Machine-Verified Theory

A distinguishing feature of the work is that the core optimization theory is formally verified in Lean 4 / Mathlib. The author does not rely on informal asymptotic arguments; instead, they provide machine-checked proofs for:

Strict Convexity: The loss is strictly convex under full-rank dictionary assumptions.
Stationarity: Gradient descent stationarity is equivalent to the normal equations ( $K^*(Kw-y)=0$ ).
Convergence: Geometric convergence of gradient descent is proven under a derived Polyak–Lojasiewicz (PL) condition.
Zero-Forgetting: The DHR identity is proven to preserve the function pointwise.

Key Results

1. Optimization Stability and Accuracy

On a coupled multivariate regression task ( $f(x,y) = \sin(x)\cos(y) + xy$ ), DKAN outperforms dense RBF-KAN baselines by approximately 20× in Test MSE.

Complex vs. Real: Complex-valued DKAN (C-DKAN) converges smoothly and monotonically. In contrast, real-valued DKAN (R-DKAN) exhibits severe loss instability and gradient shocks, validating the complex phase as a geometric stabilizer.
Zero-Forgetting: Upon DHR expansion, the network output change is exactly 0.0, confirming the theoretical guarantee.

2. Limitations on Discontinuities

The paper reports negative results regarding discontinuities. On a step-function approximation task, smooth magnitude-RBF dictionaries exhibit Gibbs-like ringing (overshoot/undershoot).

While C-DKAN rings less than dense KANs, it cannot match the near-exact step approximation of ReLU MLPs.
The TV regularizer, which penalizes weight-space curvature, fails to suppress function-space overshoot because broad overshoot bumps have small discrete second derivatives. The author explicitly states that weight-space TV does not control function-space ringing.

3. Hardware Scaling

DKAN resolves the memory bottleneck of continuous KANs.

Dense KANs require $O(BN^2G)$ memory for edge-activation tensors, leading to OOM at width $N=256$ .
DKAN scales with $O(BNM)$ memory. At $N=128$ , DKAN uses 18.6× less VRAM than the dense baseline. At $N=256$ , the dense model fails, while DKAN completes within 2.66 GB.

4. Scientific Discovery (ODEs and PDEs)

Damped Pendulum: DKAN successfully recovers the coupled state-space structure $\{\theta, \dot{\theta}\}$ and the nonlinearity $\sin(\theta)$ . A real-valued baseline failed to retain the damping coordinate $\dot{\theta}$ , substituting it with a monotonic proxy.
Burgers' Equation: DKAN discovers the viscous Burgers' PDE coefficients ( $u_t = -1.0024 u u_x + 0.0095 u_{xx}$ ) with <1% error. Crucially, it synthesizes the non-linear product $u u_x$ natively via the imaginary part of $z^2$ (where $z = u + i u_x$ ), whereas the baseline required this term as an explicit input feature.

5. Symbolic Compilation

DKAN can be compiled into a ~50-line closed-form GLSL shader for BRDF energy compensation. By pruning small weights and using fixed centers, the network converts directly to a symbolic expression tree, enabling real-time deployment without a neural runtime.

Significance and Claims

The paper positions Dictionary-KAN not as a universal interpolator (where unstructured MLPs may still achieve lower raw MSE on smooth grids), but as a stable, interpretable, and hardware-efficient alternative to both MLPs and continuous KANs.

Its primary significance lies in:

Resolving the Optimization Paradox: By fixing the dictionary and using complex routing, it eliminates grid instability and vanishing gradients while maintaining the topological constraints of the KA theorem.
Machine-Verified Guarantees: It provides the first machine-checked proofs for KAN convergence, convexity, and zero-forgetting refinement, moving beyond informal theoretical claims.
Native Multiplicative Coupling: It achieves variable interaction modeling through complex analytic properties rather than dense projections.
Scalability: It removes the memory curse of dimensionality associated with dense grid-based KANs.

The author is modest in their claims, explicitly acknowledging that smooth dictionaries struggle with discontinuities and that the current theory relies on conditional premises (e.g., full-rank dictionaries) that require data-dependent certification in future work (V2). The work is framed as a "V1" foundation, with future versions targeting kernel-theoretic unification and deep system scaling.