📊 statistics

Geometry-Constrained Kolmogorov-Arnold Networks: Learning Edge Geometry via Banach Duality

This paper introduces Geometry-Constrained Kolmogorov-Arnold Networks (Banach-KANs), which replace fixed edge activations with learnable functions derived from Banach duality maps controlled by a scalar exponent $p$ , achieving superior or competitive performance in symbolic regression and demonstrating enhanced robustness to noise and small sample sizes compared to traditional fixed-basis KANs.

Original authors: K S Sesh Kumar

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: K S Sesh Kumar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of scientific discovery, many natural laws describe how one quantity changes in response to another. The swing of a pendulum depends on its angle; the speed of a car depends on traffic density; the brightness of a star depends on its temperature. For decades, scientists have used mathematical models to capture these relationships, but a newer approach has emerged that treats the model itself as a flexible, learning entity. This approach, known as a Kolmogorov–Arnold Network, works by breaking a complex problem into many small, simple pieces. Instead of forcing the entire system to learn a single, rigid rule, it builds a network where every connection between two points is governed by its own unique, learnable function. The central challenge for these networks has always been deciding what shape those functions should take. Traditionally, researchers have had to pick a specific shape in advance—like a smooth curve or a repeating wave—and stick with it for the whole problem. This is a bit like trying to fix a broken machine with only one type of wrench; it works well for some bolts but fails miserably on others.

A researcher at Imperial College London has proposed a different way to think about this problem. They realized that the real issue is not the shape of the function itself, but the underlying "geometry" or space in which that function lives. In mathematics, geometry determines how distances are measured and how sharp or smooth a curve can be. The researcher developed a new type of network where the geometry is not fixed before the learning begins. Instead, the network learns the best geometry for each individual connection directly from the data. They achieved this by introducing a single, adjustable number for every connection in the network. This number acts as a dial that shifts the behavior of the connection from being sharp and threshold-like to being smooth and linear, or even flat and saturated. By letting the data decide where to set this dial, the network can adapt its own internal structure to match the specific quirks of the problem it is solving.

The researcher tested this idea on fifty different mathematical problems, ranging from standard physics equations to synthetic challenges designed to stress-test the system. They compared their new, geometry-adaptive networks against older models that relied on fixed shapes, such as splines (which are like flexible rulers used to draw smooth curves) or polynomials. The results showed that the fixed-shape models struggled when the data contained sudden jumps or sharp corners, often producing wobbly, inaccurate results because their rigid shapes could not bend enough to fit the data. In contrast, the new networks, which could adjust their own geometry, matched or beat the performance of every fixed-shape baseline. On a core set of eighteen difficult equations, the new method achieved the best average ranking, and on the full set of fifty, it performed just as well as the strongest traditional methods.

Perhaps the most significant finding was how these new networks handled noise. In the real world, measurements are rarely perfect; they often contain random errors or "static." When the researcher added increasing amounts of noise to their data, the traditional models fell apart quickly. Their error rates grew by a factor of twenty-one or more as the noise increased. The new geometry-adaptive networks were far more robust. Even as the noise level rose, their error rates increased by a much smaller factor, often less than four times the original error. This suggests that by learning the right geometry, the network can ignore the random static and focus on the true signal, a capability that fixed-shape models lack.

The study also revealed that the network does not just learn a single, uniform geometry for everything. Instead, different connections within the same network learned different settings for their geometry dial. Some connections learned to be very sharp to handle sudden changes, while others learned to be smooth for gradual trends. This specialization happened consistently across different types of equations and input dimensions. For instance, when the problem involved more variables, the network tended to learn sharper geometries more often. This behavior provides a kind of interpretability: by looking at the settings the network chose, researchers can see a signal that reflects the underlying structure of the problem. The network essentially tells us, "This part of the problem needs a sharp edge, while that part needs a smooth curve."

The researcher also explored what happens when there is very little data available. In these small-sample scenarios, the new networks again outperformed the fixed-shape models, which typically require large amounts of data to learn effectively. The ability to adapt the geometry allowed the new networks to approximate the correct answers with far fewer examples. However, the study also noted limits to this approach. While the new method excels at low-to-mid-dimensional problems, it is not a replacement for deep, massive neural networks used in tasks like image recognition. In fact, when tested on image datasets, the new method only matched the performance of standard networks if it was given twenty to thirty times more parameters, suggesting that its strength lies in efficiency and adaptability for specific types of regression problems rather than raw scale.

Ultimately, this work shifts the focus from choosing the right tool to building a tool that can change its own shape. The researcher demonstrated that the key to solving complex regression problems is not the specific mathematical basis used to represent the data, but the geometric space in which that representation lives. By making that space a learnable parameter, they created a system that is more robust to noise, more efficient with small datasets, and more capable of capturing the true nature of the relationships it is trying to model. The findings suggest that in the future, the most effective models may not be those with the most complex fixed architectures, but those that can learn the very geometry of the problem they are solving.

Technical Summary: Geometry-Constrained Kolmogorov–Arnold Networks

Problem Statement

Kolmogorov–Arnold Networks (KANs) replace fixed activation functions in deep architectures with learnable univariate edge functions. While existing KAN variants (e.g., those using B-splines, Chebyshev polynomials, or Fourier features) offer flexibility, they rely on fixed basis expansions. This imposes a specific function-space geometry (e.g., smoothness, periodicity, or locality) a priori, before data is observed.

The paper argues that this fixed geometry is a fundamental limitation. As demonstrated in the introduction, different fixed bases fail in distinct ways when fitting targets with mismatched structures (e.g., smooth bases struggling with discontinuities like Heaviside steps). The core problem is not representational capacity but geometric mismatch: the imposed geometry of the basis does not align with the intrinsic geometry of the target function.

Methodology: Geometry-Constrained KANs

The author proposes geometry-constrained KANs, where the edge activation is not selected from a discrete set of bases but is derived from the duality map of a learnable Banach space geometry.

1. Theoretical Foundation

The approach leverages the correspondence between a norm, a convex potential, and its gradient (the activation):
$\text{norm} \implies \text{geometry} \implies \text{convex potential } \Psi \implies \text{activation } \phi = \nabla \Psi$
Instead of fixing the basis, the author parameterizes the geometry using the $\ell_p$ family. The exponent $p$ controls the qualitative behavior of the edge:

$p \to 1$ : Produces sharp, threshold-like behavior (reminiscent of $\ell_1$ geometry).
$p = 2$ : Recovers the linear/Euclidean regime.
$p > 2$ : Produces flatter responses near the origin and power-law growth in the tails.

2. Architecture Variants

The paper defines a controlled family of three architectures to isolate the effects of expressivity and geometric adaptation:

Tanh-KAN: Uses the standard $\tanh$ activation (derived from Orlicz duality, $\Psi(z) = \log \cosh z$ ). It offers bounded expressivity but no geometric adaptation.
$\ell_p$ -KAN: Uses the duality map $J_p(z) = \text{sign}(z)(|z| + \epsilon)^{p-1}$ . It allows geometric adaptation via a learned exponent $p$ but lacks bounded saturation.
Banach-KAN: The primary model, which combines both branches:
$\phi(x) = a_1 \tanh(w_1 x + b_1) + a_2 J_p(w_2 x + b_2) + d$
This architecture unifies bounded saturation (from $\tanh$ ) and adaptive power-law geometry (from $J_p$ ), allowing edges to span diverse response regimes.

3. Implementation Details

Learnable Exponent: The exponent is parameterized as $p = 1 + \exp(\theta)$ , allowing unconstrained optimization of $\theta$ .
Numerical Stability: The exact duality map $J_p$ has a singular derivative at $z=0$ for $p < 2$ . The author implements an everywhere- $C^1$ smoothed map: $J_p^\epsilon(z) = z(z^2 + \epsilon^2)^{(p-2)/2}$ , which preserves the geometric interpretation in the tails while ensuring differentiability at the origin.
Trainable Depth Condition: Theoretical analysis (Proposition 1) suggests that for a feedforward stack with Gaussian pre-activations, a critical weight scale exists only if $p > 3/2$ . Below this threshold, signal propagation may become unstable without residual connections.

Key Contributions

Geometry-Constrained Parameterization: Introducing KAN edge activations derived from Banach duality maps where the geometry (controlled by $p$ ) is learned directly from data, rather than fixed by a basis choice.
Unified Interpretation: Reframing KAN edge functions not as basis expansions but as geometry-controlled mappings.
Empirical Robustness: Demonstrating that learning geometry improves robustness to measurement noise and approximation in small-sample regimes compared to fixed-basis KANs and explicit spline regularization.
Interpretable Signal: Showing that learned exponents provide a consistent, target-dependent geometric ordering (e.g., sharper targets yield lower $p$ ) that is stable across noise levels and input dimensions.
Ablation of Flexibility: Isolating the "geometry" effect from generic parametric flexibility by comparing against matched-budget controls (e.g., slope-adaptive activations, rational functions, and mixture models).

Experimental Results

The models were evaluated on 50 symbolic regression targets (40 from the AI Feynman benchmark, 10 synthetic stress tests) and 12 PMLB tabular datasets.

Performance on Clean Data

Median NRMSE: Banach-KAN achieved a median NRMSE of 0.030 on the full 50-equation benchmark, statistically tying with Chebyshev-KAN (0.030) and outperforming Spline-KAN (0.037).
Ranking: Banach-KAN achieved the best average rank on the "Core 18" subset (2.00) and tied for the best on the full set (2.32 vs. 2.34 for Spline).
Small-Sample Regime: Banach-KAN won the most equations in the small-sample regime ( $n=50$ and $n=100$ ), outperforming fixed-basis models that require more data to converge.

Robustness to Noise

The paper highlights significant gains under measurement noise ( $\sigma$ increasing from 0 to 1):

Degradation Ratio: As noise increased, fixed-basis models degraded significantly (Spline G=3 degraded 21.6 $\times$ , Cheby-KAN 14.9 $\times$ ).
Geometry-Constrained Stability: The geometry-constrained variants degraded far less. The $\ell_p$ -KAN degraded only 3.7 $\times$ , and Banach-KAN degraded 8.8 $\times$ . Notably, even a cross-validated spline degraded $\approx$ 11 $\times$ , making the learned geometry more robust than tuned regularization.
Noise Regime Shift: At high noise ( $\sigma=1.0$ ), the $\ell_p$ variant outperformed Banach-KAN, suggesting a trade-off where pure power-law geometry is more robust to extreme noise than the combined architecture.

Extrapolation and Tabular Data

Extrapolation: On out-of-range testing (training on inner 60% of input space), Banach-KAN and $\ell_p$ -KAN achieved the lowest NRMSE (0.35 and 0.42), significantly outperforming local bases like Splines (0.73) and RBFs (0.86).
Tabular Data: On 12 PMLB datasets, the geometry-constrained family won 10/12 datasets. Tanh-KAN was the strongest single model (5 wins), while Banach-KAN won 1. Spline G=3 won zero.

Significance and Claims

The paper positions its contribution as a design principle rather than a state-of-the-art benchmark record. The author explicitly states:

Modest Scope: The method is designed for low-to-mid-dimensional regression. It does not compete with deep architectures (like ResNets) on their native ground (e.g., image classification) where MLPs match performance only at 20–30 $\times$ the parameter count.
The "Right Tool" for the Regime: The claim is that at comparable per-edge budgets, under measurement noise, and in small-sample regimes, the induced geometry (learned via $p$ ) governs learning success more than raw expressivity.
Interpretability: The learned exponent $p$ serves as a "geometric readout" of the target function's structure, providing an interpretable signal about the required regularity (sharp vs. smooth) without manual tuning.
Limitations: The author acknowledges that the per-edge form has a capacity ceiling as data size grows and that the evaluation is currently limited to symbolic regression and tabular data, with future work needed for high-dimensional modalities (sequences, images).

In summary, the paper argues that geometry is the primary modeling choice in KANs. By making the geometry adaptive through a learned exponent $p$ , the model can align its inductive bias with the data, resulting in superior robustness and approximation in regimes where fixed bases fail.