🤖 machine learning

Learning Expressive Random Feature Models via Parametrized Activations

This paper introduces the Random Feature Model with Learnable Activation Functions (RFLAF), which parameterizes activation functions using weighted sums of basis functions to significantly expand the model's expressivity and achieve superior performance and efficiency compared to traditional fixed-activation random feature models.

Original authors: Zailin Ma, Jiansheng Yang, Yaodong Yang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zailin Ma, Jiansheng Yang, Yaodong Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize patterns, like distinguishing a cat from a dog in a photo. To do this, the robot uses a mathematical "translator" called a Random Feature Model. Think of this translator as a factory assembly line.

In a standard factory (traditional models), the workers (the activation functions) are stuck doing the exact same repetitive motion every single time. They are rigid. If the task changes slightly, the factory struggles to adapt because the workers can't change their routine.

This paper introduces a new, smarter factory called RFLAF (Random Feature Model with Learnable Activation Functions). Here's how it works, broken down into simple concepts:

1. The Problem: The Rigid Workers

In the old way, the factory uses a fixed "tool" (a fixed activation function) to process data. It's like trying to fix a flat tire with a hammer. It might work for some things, but it's terrible for others. The model can't learn which tool is best for the specific job; it just uses the one it was born with.

2. The Solution: The Shape-Shifting Toolkit

The authors propose giving the factory workers a shape-shifting toolkit. Instead of one fixed tool, the workers have a set of basic building blocks (called "basis functions"). These blocks are like:

Radial Basis Functions (RBFs): Think of these as smooth, bell-shaped curves (like a gentle hill).
Splines: Think of these as flexible, bendable rulers.
Polynomials: Think of these as curvy, wiggly lines.

The magic of RFLAF is that the model learns how to mix and match these blocks. It figures out, "For this specific task, I need 30% of a hill, 50% of a bendable ruler, and 20% of a wiggle." It creates a custom tool on the fly that fits the data perfectly.

3. The "Single Hill" Discovery

Before building the complex toolkit, the authors studied what happens if you just use one of these "hill" shapes (a single Radial Basis Function).

The Finding: They discovered a hidden mathematical formula (a "kernel") that describes exactly how this single hill interacts with data.
The Limit: They found that using just one hill isn't enough to make the model super smart. It's still a bit limited, like having only one type of screwdriver.

4. The Power of the Mix

The real breakthrough happens when you combine many of these hills with learnable weights.

The Result: By mixing many hills together, the model can represent a much wider variety of shapes and patterns. It becomes incredibly expressive, meaning it can understand complex data much better than the old rigid models.
The Cost: The authors prove mathematically that you don't need a massive factory to do this. You only need a tiny bit more "space" (parameters) to get a huge boost in performance. It's like upgrading a small car engine with a slightly better fuel injection system to get a race-car speed.

5. The Race: Hills vs. Rulers

The team tested their new factory against the old ones using real-world data (like recognizing handwritten numbers or predicting house prices).

The Winners: The models using Hills (RBFs) and Bendable Rulers (Splines) consistently beat the old rigid models.
The Speedster: While both Hills and Rulers were good, the Hills (RBFs) were the clear champions. They were three times faster to compute than the Rulers, while still delivering the best results.
The Losers: The "wiggly lines" (polynomials) were unstable and often broke down when the tasks got too complex.

6. The "Frozen" vs. "Unfrozen" Test

Finally, the authors did a final experiment. They took their new "smart factory" and let it change everything, not just the tools, but also the initial setup of the assembly line (unfreezing the first-layer parameters).

The Outcome: Even when compared to standard, modern neural networks (the current state-of-the-art), their "smart factory" approach held its own and often performed better. This proves that the idea of "learning the tool" is a powerful upgrade for any neural network.

Summary

In short, this paper says: "Stop using a hammer for everything. Give your AI a toolbox of shapes, let it learn how to mix them, and you'll get a smarter, faster, and more adaptable model." Specifically, using "hill-shaped" mixers (RBFs) is the sweet spot for getting the best performance with the least amount of extra computing power.

Technical Summary: Learning Expressive Random Feature Models via Parametrized Activations

1. Problem Statement

Random Feature (RF) models, introduced as a scalable alternative to kernel methods, approximate kernel functions using a finite number of random features. While computationally efficient, standard RF models typically employ fixed activation functions (e.g., ReLU, Sigmoid, Cosine). This rigidity limits their adaptability across diverse tasks, preventing the model from automatically searching for optimal activation functions to fit specific data distributions. Conversely, recent developments in learnable activation modules, such as Kolmogorov-Arnold Networks (KANs), have demonstrated effectiveness but often face scalability or optimization challenges. The paper addresses the gap between the efficiency of fixed RF models and the expressivity of learnable activation networks by proposing a framework that parameterizes activation functions within the RF setting.

2. Methodology

The authors introduce the Random Feature Model with Learnable Activation Functions (RFLAF).

Model Formulation

In RFLAF, the activation function $\sigma$ is not fixed but is parameterized as a weighted sum of basis functions $\{B_i(x)\}_{i=1}^N$ within the random feature framework. The model is defined as:
$\hat{f}(x; a, v) := \frac{1}{M} \sum_{m=1}^M \sum_{i=1}^N a_i B_i(w_m^\top x) v_m$
where:

$w_m \sim \mathcal{N}(0, I_d)$ are fixed random features sampled from a Gaussian distribution.
$B_i(x)$ are basis functions (e.g., Radial Basis Functions, B-splines, Polynomials) with pre-set centers and widths.
$a = (a_1, \dots, a_N)$ and $v = (v_1, \dots, v_M)$ are learnable parameters.

The authors focus on Radial Basis Functions (RBFs) as the representative basis for theoretical analysis, defined as $B_i(x) = \exp\left(-\frac{(x-c_i)^2}{2h_i^2}\right)$ .

Theoretical Approach

Single RBF Analysis: The authors first derive the explicit analytic form of the kernel induced by a single RBF activation. They establish the Taylor expansion of this rotation-invariant kernel and characterize the associated Reproducing Kernel Hilbert Space (RKHS).
Multiple RBFs & Approximation: Extending to multiple RBFs, they utilize the Gaussian Universal Approximation Theorem. They prove that a weighted sum of RBFs can approximate any continuous target activation function with compact support to an arbitrary error $\epsilon$ .
Generalization Bounds: Using Rademacher complexity, the authors derive generalization bounds for the learning process. They analyze the sample complexity required to achieve low excess risk, showing that the number of random features $M$ needs to scale with the square root of the sample size $n$ ( $M = \Theta(\sqrt{n})$ ) to achieve optimal performance, matching the sharpest results for standard RF models.

3. Key Contributions

The paper makes the following specific contributions:

Novel Kernel Derivation: The authors identify and provide the first analytic form of the kernel induced by a single RBF as an activation function, extending previous results on centered Gaussian activations to general non-centered cases.
Theoretical Guarantees: They provide rigorous approximation and generalization bounds for RFLAF. The theory guarantees that the model improves representational ability with a minimal cost of less than twice the number of parameters compared to standard RF models.
Empirical Validation: Extensive experiments demonstrate that RFLAFs with RBFs or B-splines consistently outperform standard RF models with fixed activations across various classification and regression tasks.
Efficiency Analysis: The study highlights that RBF-based RFLAFs are approximately three times more computationally efficient than B-spline-based RFLAFs while achieving superior or comparable performance.
Extension to Neural Networks: By unfreezing the first-layer parameters of RFLAFs, the authors validate the expressivity advantage of learnable activation components on regular two-layer neural networks (termed LANs), showing they can outperform standard MLPs and KANs in terms of convergence speed and stability.

4. Experimental Results

The authors evaluated RFLAF on seven real-world datasets (MNIST, CIFAR-10, Adult, Protein, CT, Workloads, MillionSongs) and synthetic datasets.

Performance: RFLAFs with RBFs and B-splines consistently achieved lower test losses and higher accuracies compared to standard RF models (RFMLP) with fixed activations (ReLU, Cos, Tanh, Sigmoid).
Efficiency: While B-splines performed well, RBFs were found to be 2–4 times faster in training and testing times. For instance, on the "Workloads" dataset, RBFs improved test loss by ~70% compared to the best fixed activation (Sigmoid) with less than a 30% increase in time.
Grid Size Sensitivity: Experiments showed that a small number of basis functions ( $N=16$ or $32$) is sufficient to significantly outperform standard RF models, offering a good trade-off between performance and computational cost.
Activation Recovery: On synthetic tasks, RFLAF successfully recovered ground-truth activation functions from data, demonstrating its ability to learn optimal non-linearities without grid searching.
Two-Layer Networks: When applied to regular two-layer networks (LAN), RBF-based models achieved the best performance among LAN, MLP, and KAN, converging faster than KAN and outperforming MLP.

5. Significance and Claims

The paper claims to provide a deeper understanding of learnable activation components within modern neural network architectures. By integrating learnable activations into the random feature framework, the work bridges the gap between the theoretical efficiency of kernel methods and the adaptability of deep learning.

The authors position RFLAF as a method that:

Expands Function Space: Substantially expands the represented function space compared to fixed RF models without a proportional increase in computational complexity.
Offers Minimal Overhead: Achieves significant performance gains with a minimal increase in parameter count and training time.
Validates Learnable Activations: Provides evidence that learnable activation modules are effective not just in complex architectures like KANs, but also in simpler, more scalable two-layer structures.

The work concludes that RFLAF opens future directions for random feature models, suggesting that parameterizing activations is a viable and efficient strategy for enhancing model expressivity in both kernel methods and neural networks.