Technical Summary: Universal Smoothness via Bernstein Polynomials

Problem Statement

The design of non-linear activation functions in Deep Neural Networks (DNNs) faces a persistent trade-off between computational efficiency and optimization stability.

• Piecewise Linear Functions (e.g., ReLU, Leaky ReLU): While computationally efficient due to simple thresholding operations, they suffer from non-differentiability at the origin. This singularity creates "kinks" in the loss landscape, leading to optimization instability and the "Dying ReLU" problem where neurons in negative states cease to update.
• Smooth Transcendental Functions (e.g., GELU, Swish, Mish): These functions address non-differentiability by introducing smooth transitions via Gaussian or exponential operations. However, they incur significant computational overhead during training and inference due to reliance on transcendental functions. Furthermore, many of these smooth functions act as expansive operators (Lipschitz constant $L > 1$), which can theoretically lead to gradient explosion in deep architectures.

The core challenge is to develop an activation function that retains the computational efficiency of linear units while ensuring the smoothness and gradient stability of transcendental functions, without incurring their computational costs.

Methodology

The paper proposes a constructive approximation framework based on Bernstein Polynomials to mollify the non-differentiable transition of the Leaky ReLU function.

1. The Bernstein Linear Unit (BerLU)

The authors introduce BerLU, a novel activation function that replaces the singularity at the origin with a quadratic transition region constructed via Bernstein polynomials.

• Construction: The function is defined piecewise. For $x < -\epsilon$ and $x > \epsilon$, it behaves as a linear function (with slope $\alpha$ for negative inputs and $1$ for positive inputs). In the transition interval $[-\epsilon, \epsilon]$, it employs a degree-2 Bernstein polynomial (a quadratic Bézier curve).
• Continuity Constraints: The control points of the polynomial are uniquely determined by imposing $C^0$ (continuity) and $C^1$ (differentiability) constraints at the boundaries $x = -\epsilon$ and $x = \epsilon$.
• Analytical Form: The resulting function is a simple quadratic polynomial in the transition region, requiring only basic arithmetic operations (addition, multiplication) rather than transcendental functions.
  $$\text{BerLU}(x) = \begin{cases} \alpha x & x < -\epsilon \\ \frac{1-\alpha}{4\epsilon}x^2 + \frac{1+\alpha}{2}x + \frac{(1-\alpha)\epsilon}{4} & -\epsilon \leq x \leq \epsilon \\ x & x > \epsilon \end{cases}$$
  Here, $\epsilon$ is a smoothing hyperparameter, and $\alpha$ is a learnable parameter initialized to 0.01.

2. Theoretical Guarantees

• Smoothness: The construction guarantees strictly continuous differentiability ($C^1$), eliminating the singularity at the origin.
• Lipschitz Continuity: The paper proves that BerLU is Lipschitz continuous with a constant $L_{\text{BerLU}} = \max(1, |\alpha|)$. Since $\alpha$ is typically small ($|\alpha| \ll 1$), the Lipschitz constant is effectively 1.
  • This classifies BerLU as a non-expansive operator. Unlike GELU, SiLU, or Mish, which have Lipschitz constants $> 1$ (e.g., GELU $\approx 1.084$), BerLU ensures that the gradient norm does not intrinsically amplify across layers, theoretically preventing gradient explosion in deep networks.

Key Contributions

1. General Smoothing Framework: A constructive approximation approach using Bernstein polynomials to transform non-smooth piecewise linear activations into strictly differentiable counterparts while preserving geometric properties.
2. High Computational Efficiency: BerLU replaces expensive transcendental operations with basic arithmetic, significantly reducing computational overhead during training and inference compared to GELU and Swish.
3. Theoretical Stability: Rigorous analysis demonstrates that BerLU guarantees a non-expansive Lipschitz constant of 1, ensuring stable gradient propagation and flatter loss landscapes.
4. Empirical Superiority: Extensive validation across Vision Transformers (ViT, DeiT, TNT) and Convolutional Neural Networks (ConvNeXt) on CIFAR and ImageNet benchmarks.

Experimental Results

The authors evaluated BerLU against state-of-the-art baselines (GELU, ELU, PReLU, CELU, SiLU, Mish) across multiple architectures and datasets.

• Classification Accuracy:
  • CIFAR-100: BerLU achieved an average accuracy of 50.6%, outperforming PReLU by 3.2% and GELU by 8.4%.
  • CIFAR-10: BerLU achieved 78.5% (ViT-Tiny), surpassing PReLU (78.0%) and GELU (70.4%).
  • ImageNet-1K: BerLU achieved 60.6% average accuracy on ViT/DeiT and 73.5% on ConvNeXt, consistently leading over GELU and SiLU.
• Computational Efficiency:
  • Inference Speed: BerLU demonstrated lower forward latency (55.8 ms) compared to GELU (59.2 ms) on ViT.
  • Memory Footprint: BerLU consumed the least peak memory (10.58 GB) among tested functions, significantly lower than GELU (11.53 GB) and SiLU (12.25 GB).
• Sensitivity Analysis: The smoothing parameter $\epsilon$ was found to be robust within the range $[10^{-4}, 10^{-1}]$, with optimal performance at $\epsilon = 10^{-2}$. Excessive smoothing ($\epsilon > 0.2$) degraded performance by eroding necessary non-linearity.

Significance and Claims

The paper claims that BerLU offers a robust, architecture-agnostic solution that resolves the efficiency-stability trade-off in deep learning. By leveraging constructive approximation theory, it achieves the optimization benefits of smooth functions (better gradient flow, flatter loss landscapes) without the computational penalty of transcendental operations.

Crucially, the authors highlight that BerLU's non-expansive nature ($L=1$) provides a theoretical safeguard against gradient explosion, a structural vulnerability present in popular smooth activations like GELU and Swish. The empirical results suggest that BerLU is not only a viable alternative but a superior choice for large-scale vision tasks, delivering state-of-the-art performance with reduced memory usage and faster inference.