Technical Summary: Rational Neural Networks have Expressivity Advantages

Problem Statement

Deep neural networks rely heavily on the choice of nonlinear activation functions to control expressivity and trainability. While modern architectures predominantly utilize fixed, smooth activations (e.g., GELU, Swish/SiLU, Mish, ELU/SELU/CELU) or piecewise-linear functions (e.g., ReLU and variants), these choices impose specific limitations on how efficiently networks can approximate functions with sharp transitions, kinks, or nearby complex singularities. The paper investigates whether replacing these fixed activations with trainable low-degree rational activation functions offers a theoretical and practical advantage in terms of parameter efficiency and expressive power.

Methodology

Theoretical Framework

The authors establish approximation-theoretic separations between rational neural networks and networks built from standard smooth or piecewise-linear activations.

• Rational Networks: Defined as feedforward networks where the scalar nonlinearity at each layer is a trainable rational function $r(x) = P(x)/Q(x)$ with low-degree polynomials $P$ and $Q$. The coefficients are learned during training.
• Approximation Metrics: Expressivity is measured by the number of trainable parameters (network size) required to achieve a uniform error $\epsilon$ on a fixed compact domain (e.g., $[-1, 1]^d$).
• Key Theoretical Tools:
  • Rational Approximation Theory: Leveraging classical results (e.g., Trefethen, Gonchar) showing that rational functions can approximate non-smooth features (like $|x|$ or ReLU) and functions with nearby singularities exponentially more efficiently than polynomials or smooth surrogates.
  • Constructive Proofs: The paper provides explicit constructions using Zolotarev functions, arithmetic-geometric-mean (AGM) iterations, and Halley's method to approximate smooth activations (like GELU) using rational networks.
  • Lower Bounds: Proving that approximating certain rational targets (specifically those with poles near the real line) using bounded-parameter smooth networks requires significantly more parameters due to curvature constraints.

Experimental Setup

The authors evaluate rational activations in three distinct domains, keeping architectures, optimizers, and training budgets fixed while varying only the activation function:

1. Supervised Vision: CIFAR-10 image classification using VGG4 and VGG8 architectures, and Tiny ImageNet classification using Transformer-based models (ViT, CaiT, Swin).
2. Offline Reinforcement Learning: Continuous-control locomotion benchmarks (HalfCheetah, Hopper, Walker2d) using the Minari medium datasets and algorithms IQL and TD3+BC.
3. Normalization Analysis: Investigating the interaction between adaptive rational activations and normalization layers (BatchNorm, LayerNorm), specifically testing "no-LayerNorm" variants in Transformers.

Key Contributions

1. Exponential Separation in Parameter Complexity:

   • Smooth $\to$ Rational: Any network built from standard smooth activations (GELU, Swish, SiLU, etc.) can be uniformly approximated by a rational network with a size overhead of only $O(\text{poly}(\log \log(1/\epsilon)))$. Specifically, the paper proves a size bound of $O(\log^3 \log(1/\epsilon))$ for GELU.
   • Rational $\to$ Smooth: Conversely, approximating a worst-case family of rational functions (with poles near the domain) using bounded-parameter smooth networks requires a size of at least $\Omega(\log(1/\epsilon))$.
   • Result: This establishes an exponential gap in expressivity, where rational networks can achieve the same accuracy with significantly fewer parameters than smooth-activation networks in regimes involving non-smooth structure.

2. Extension to Complex Architectures:

   • The separation results are lifted from scalar functions to full feedforward networks, gated activations, and Transformer-style architectures (specifically the MLP sublayers).
   • The paper demonstrates that rational networks can approximate gated smooth models with low overhead, while the reverse requires high overhead.

3. Normalization and Identifiability:

   • The authors formalize that standard normalization layers can be detrimental to adaptive rational activations. Normalization introduces learned affine transforms that can be absorbed by the rational coefficients, leading to non-identifiability and poor conditioning.
   • Furthermore, data-dependent stochasticity in normalization (e.g., BatchNorm) amplifies gradient noise due to the polynomial sensitivities of rational coefficients.
   • Finding: Removing normalization (e.g., "no-LayerNorm" Transformers) allows adaptive rationals to function as learnable rescaling mechanisms, often stabilizing training where fixed activations fail.

4. Constructive Proofs:

   • The paper provides constructive proofs based on classical approximation theory, including the use of Zolotarev functions and AGM constructions, to demonstrate how rational networks can efficiently realize complex non-linearities.

Experimental Results

• CIFAR-10: Rational activations consistently achieved the highest Top-1 test accuracy in both VGG4 and VGG8 models, outperforming ReLU, LeakyReLU, Swish, and GELU. In the augmented training pipeline, rational variants (initialized to mimic LeakyReLU or GELU) remained top-performing even when GroupNorm was applied, though GroupNorm induced a larger performance drop for rationals compared to fixed activations.
• Offline RL: On MuJoCo medium datasets, rational activations matched or surpassed the best fixed activations in 5 out of 6 settings (IQL and TD3+BC on HalfCheetah, Hopper, Walker2d). Notably, rational models with smaller widths (128) were competitive with or superior to much larger GELU models (384 width), validating the parameter-efficiency hypothesis.
• Tiny ImageNet (Transformers): Rational activations matched or improved upon the best EMA validation accuracy for ViT, CaiT, and Swin models. Crucially, in a "no-LayerNorm" ViT variant, fixed activations (ReLU, GELU) diverged to NaNs, while the rational variant remained stable and achieved high accuracy (61.24%), supporting the claim that normalization can be restrictive for adaptive rationals.
• Learned Shapes: Visualizations of learned rational functions showed they quickly reallocate curvature to high-density input regions, often developing localized non-monotone "gating" shapes similar to successful fixed activations like Swish and Mish.

Significance and Claims

The paper claims that rational neural networks offer a mathematically natural and computationally efficient foundation for deep learning. The core significance lies in the expressivity advantage: rational activations can represent functions with sharp transitions and complex singularities using exponentially fewer parameters than modern smooth activations.

The authors argue that this theoretical advantage translates directly to practice, resulting in:

• Higher Accuracy: Achieving better performance on vision and RL tasks with identical architectures.
• Faster Convergence: Reaching strong performance earlier in training.
• Parameter Efficiency: Achieving comparable or superior results with smaller network widths.
• Robustness in Normalization-Free Regimes: Enabling the training of deep Transformers without normalization layers, a regime where fixed activations often fail.

The work suggests that the field should reconsider the dominance of fixed smooth activations, proposing that trainable rational functions provide a principled alternative that better aligns with the structural requirements of complex data distributions.