Technical Summary: Action at a Distance: A Universal Reproducing Kernel Hilbert Space from Polynomial Alignment and IMQ Distance

Problem Statement

Standard Multi-Layer Perceptron (MLP) layers lack a canonical Hilbert-space norm. Generalization bounds for trained networks typically rely on Lipschitz or spectral surrogates of the weight matrix, which are function-level objects loosely constructed and disconnected from the specific units. This structural gap arises because standard scalar activations (e.g., ReLU, GELU, sigmoid) are functions of an inner product $\sigma(w^\top x)$, not symmetric kernel sections of the joint variable $(w, x)$. Consequently, a trained layer's read-out is not a finite element of any Reproducing Kernel Hilbert Space (RKHS), depriving the model of tools like closed-form norms, fixed-kernel universality, and characteristicness.

Existing Mercer-section primitives suffer from a trade-off:

1. Radial kernels (Gaussian RBF, Inverse Multiquadric/IMQ) possess locality and Hilbert structure but lack "alignment" (the kernel diagonal is constant, and the polynomial alignment term $(w^\top x + b)^2$ plays no role).
2. Aligned kernels (universal dot-product kernels) possess alignment and Hilbert structure but lack distance locality (no $\|x-w\|$ decay).

The paper identifies the need for a finite, trainable hidden-unit primitive that simultaneously combines locality, alignment, and Hilbert structure.

Methodology

The authors introduce the Yat kernel ($k_\Upsilon$), a rational hidden-unit primitive defined as:
$$k_{b,\varepsilon}(w, x) = \frac{(w^\top x + b)^2}{\|x - w\|^2 + \varepsilon}, \quad b \ge 0, \varepsilon > 0$$
This kernel is the Schur product of a polynomial alignment numerator $p_b(x, w) = (x^\top w + b)^2$ and an Inverse Multiquadric (IMQ) denominator $h_\varepsilon(x, w) = (\|x - w\|^2 + \varepsilon)^{-1}$.

The methodology proceeds through four main theoretical pillars:

1. Mercer Structure and Universality: The paper establishes that for $b \ge 0$, the kernel is Positive Semidefinite (PSD) and thus a Mercer kernel. Crucially, for $b > 0$, the Yat kernel Loewner-dominates a scaled IMQ kernel ($b^2 h_\varepsilon \preceq k_{b,\varepsilon}$). By Aronszajn's inclusion theorem, this implies the Yat RKHS ($\mathcal{H}_{b,\varepsilon}$) contains the IMQ RKHS, inheriting its universality and characteristicness properties.
2. Nonradial Alignment Structure: Unlike pure radial kernels, Yat atoms retain a "directional quadratic shadow" at infinity. The asymptotic trace along a ray $ru$ is $T_\infty g_\varepsilon(u) = (u^\top w)^2$, independent of $b$ and $\varepsilon$. This opens nonradial alignment channels absent in finite IMQ expansions, which decay to zero at infinity.
3. Algebraic IMQ Embedding: The paper provides a constructive proof that any IMQ atom can be exactly recovered from three Yat atoms via a second-order finite difference in the bias parameter $b$:
   $$g_\varepsilon(\cdot; w, 3h) - 2g_\varepsilon(\cdot; w, 2h) + g_\varepsilon(\cdot; w, h) = 2h^2 k_\varepsilon^{\text{IMQ}}(\cdot, w)$$
   This identity is sharp; two atoms are insufficient to reproduce an IMQ section.
4. Layer-Local RKHS Capacity: For a trained layer with shared $(b, \varepsilon)$, the squared RKHS norm of a finite expansion $f = \sum \alpha_j k_{b,\varepsilon}(w_j, \cdot)$ admits a closed-form expression: $\|f\|^2 = \alpha^\top K \alpha$, where $K$ is the Gram matrix of the trained weights. This allows for a direct Rademacher generalization bound driven explicitly by the kernel diagonal $k_{b,\varepsilon}(x, x) = (\|x\|^2 + b)^2 / \varepsilon$.

Key Contributions

1. The Yat Kernel: A rational, finite, trainable primitive that unifies polynomial alignment and IMQ locality.
2. Fixed-Kernel Universality: Proof that a single fixed Yat kernel (with $b>0$) is universal on every compact domain, not merely a member of a universal family.
3. Directional Far-Field Trace: Identification of a structural invariant where Yat atoms maintain a non-zero quadratic trace at infinity, distinguishing them from all finite radial combinations.
4. Exact IMQ Recovery: An algebraic identity showing that IMQ atoms are exactly constructible from three Yat atoms, transferring IMQ approximation rates to Yat with a factor-3 width overhead.
5. Closed-Form Norm and Bounds: Derivation of an exact RKHS norm ($\alpha^\top K \alpha$) and a single-layer Rademacher bound that explicitly depends on the hyperparameters $b$ and $\varepsilon$ and the input norm $\|x\|$, offering transparency absent in standard MLPs.

Results

• Theoretical Properties: The paper establishes that Yat kernels are PSD, characteristic, and strictly positive definite for $b > 0$. They satisfy a strict RKHS containment $\mathcal{H}_{\text{IMQ}} \subsetneq \mathcal{H}_{b,\varepsilon}$.
• Approximation Rates: Through the finite-difference identity, any IMQ approximation rate transfers to Yat. Conversely, the paper demonstrates a "width-complexity gap": a single Yat atom can approximate a quadratic ridge function to arbitrary accuracy, whereas a bounded-variation IMQ expansion requires exponentially many atoms in the ambient dimension.
• Generalization Bounds: The derived Rademacher bound scales as $B(R^2 + b)/(\sqrt{n}\sqrt{\varepsilon})$. Unlike Gaussian RBF (constant diagonal) or IMQ (constant diagonal $\varepsilon^{-1}$), the Yat diagonal depends polynomially on $\|x\|$, making $b$ and $\varepsilon$ explicit regularization targets.
• Empirical Diagnostics:
  • CLIP Probe: On ImageNet-1k with frozen CLIP features, a single-layer Yat classifier (trained centers) achieved 73.9% accuracy, outperforming a pure polynomial kernel (72.5%) and a Yat variant with frozen random centers (69.5%), isolating the benefits of the IMQ denominator and center optimization.
  • Causal Language Model: A 261M parameter causal LM replacing GELU MLP blocks with Yat primitives reached the GELU baseline accuracy regime (matching seed variance) while achieving lower perplexity on WikiText-103 and long-range tasks. The closed-form RKHS norm remained applicable per layer.

Significance and Claims

The paper claims that the Yat kernel is the first finite, trainable hidden-unit primitive that simultaneously possesses:

1. PSD/Mercer structure for $b \ge 0$.
2. Fixed-kernel universality for $b > 0$ via Loewner domination.
3. Nonradial alignment with a non-zero directional far-field trace.
4. Exact algebraic embedding of IMQ atoms (sharp at three atoms).
5. A closed-form, layer-local RKHS norm with explicit dependence on hyperparameters.

The authors emphasize that while these properties hold for a single layer, they do not extend to a global theory for deep networks. There is no parameter-independent Mercer kernel that captures the Gram structure of a deep Yat stack. The deep-layer theory is limited to:

• Prefix-conditioned pullbacks: Valid RKHS structures exist if prior layers are fixed.
• Ambient Sobolev containment: Bounded-depth stacks lie in a fixed Sobolev RKHS, but the capacity bounds are qualitative and suffer from super-exponential growth in depth.

The work positions Yat not as a replacement for standard MLPs in all contexts, but as a theoretically grounded alternative that restores the connection between neural units and RKHS theory, providing explicit norms and generalization bounds that are otherwise unavailable for standard scalar activations.