Swish-T : Enhancing Swish Activation with Tanh Bias for Improved Neural Network Performance

1. Problem Statement

Activation functions are fundamental to deep learning for introducing non-linearity. While ReLU is efficient, it suffers from the "dying ReLU" problem (neurons becoming inactive). Variants like Leaky ReLU and PReLU address this but introduce non-differentiability at zero. Smooth approximations like GELU, Swish, and Mish have improved performance but still face limitations in adaptability and gradient flow control.

The authors identify that existing activation functions lack a mechanism to dynamically adjust the activation threshold and bias based on input values in a smooth, bounded manner. Specifically, they aim to:

• Overcome the "dying ReLU" issue by allowing broader acceptance of negative values during initial training.
• Provide a smoother, non-monotonic curve than the original Swish.
• Enhance gradient stability and adaptability across diverse network architectures without sacrificing computational efficiency.

2. Methodology: The Swish-T Family

The core proposal is the Swish-T family, which enhances the standard Swish function ($x \cdot \sigma(\beta x)$) by adding a Tanh bias term.

2.1 Core Design Principles

The bias function was selected based on three criteria:

1. Zero-centered: To maintain the zero-centered property of the original Swish.
2. Non-linearity: To adjust the bias dynamically based on the input $x$.
3. Bounded: To prevent the bias from disrupting the original function's properties, scaled by a hyperparameter $\alpha$.

The authors found that the combination Swish + $\alpha \tanh(x)$ yielded the best performance (79.02% Top-1 on CIFAR-100 with ResNet-18).

2.2 Variants

The paper proposes four specific variants to balance performance, computational speed, and stability:

1. Swish-T (Base):

   • Formula: $f(x) = x\sigma(\beta x) + \alpha \tanh(x)$
   • Features: Adds a trainable parameter $\beta$ and hyperparameter $\alpha$. It converges to $\pm\alpha$ in negative regions, allowing smoother handling of negative inputs.

2. Swish-TA (Efficiency):

   • Formula: $f_A(x) = \sigma(x)(x + 2\alpha) - \alpha$
   • Features: Simplifies the base equation by unifying coefficients to 1. It has no trainable parameters, making it computationally faster and suitable for scenarios where efficiency is critical.

3. Swish-TB (Adaptability):

   • Formula: $f_B(x) = \sigma(\beta x)(x + 2\alpha) - \alpha$
   • Features: Reintroduces the trainable parameter $\beta$ to Swish-TA. The bias depends on both $x$ and $\beta$, allowing dynamic adaptation to diverse input characteristics.

4. Swish-TC (Stability & Symmetry):

   • Formula: $f_C(x) = \sigma(\beta x)(x + \frac{2\alpha}{\beta}) - \frac{\alpha}{\beta}$
   • Features: Designed to ensure symmetry when $\beta$ changes sign. It prevents the maximum gradient from rapidly increasing (a potential issue in Swish-TB) and offers the most stable gradient flow.

2.3 Gradient Analysis

The authors derived analytical gradients for all variants. Notably, Swish-TC avoids direct multiplication of $\alpha$ and $\beta$ in the gradient calculation, which stabilizes the gradient flow and prevents exploding gradients during backpropagation.

3. Key Contributions

• Novel Activation Family: Introduction of the Swish-T family, which integrates a Tanh bias to create a smoother, non-monotonic activation curve that accepts negative values more effectively than Swish.
• Variant Specialization: Development of three distinct variants (TA, TB, TC) tailored for specific needs: computational speed (TA), dynamic adaptability (TB), and gradient stability (TC).
• Non-Parametric Efficiency: Demonstration that these functions can be used as non-parametric functions (by fixing $\beta$) without significant performance loss, offering faster training speeds and lower memory usage.
• Comprehensive Empirical Validation: Extensive testing across multiple datasets and architectures, proving superiority over state-of-the-art functions like ReLU, GELU, Swish, Mish, and SMU.

4. Experimental Results

The authors evaluated the Swish-T family on MNIST, Fashion MNIST, SVHN, CIFAR-10, and CIFAR-100 using architectures including ResNet-18, ShuffleNetV2, SENet-18, EfficientNetB0, MobileNetV2, and DenseNet-121.

• CIFAR-10 Performance:
  • Swish-TC achieved 94.27% Top-1 accuracy with ShuffleNetV2 (2.x), outperforming ReLU by 2.34%.
  • On DenseNet-121, Swish-TC matched SMU's top accuracy (94.95%) but demonstrated faster training times.
• CIFAR-100 Performance:
  • Swish-TC achieved 79.06% Top-1 accuracy with ResNet-18 (when $\beta$ was fixed at 6.0), surpassing ReLU by 4.12%.
  • Swish-T variants consistently outperformed ReLU, GELU, SiLU, Swish, and Mish across almost all architectures.
• Ablation Study (Fixed $\beta$):
  • The study revealed that in some architectures (like ResNet-18), the trainable $\beta$ struggled to converge to optimal values during training.
  • By fixing $\beta$ to 6.0 (a "magic number" derived from convergence analysis), the non-parametric versions (Swish-TC-6 and Swish-TB-6) achieved higher accuracy than their parametric counterparts while reducing training overhead.

5. Significance and Conclusion

The Swish-T family represents a significant advancement in activation function design by successfully integrating a bounded, non-linear bias into the Swish function.

• Performance: It consistently outperforms traditional and modern activation functions across diverse benchmarks.
• Flexibility: The family offers a trade-off between speed (Swish-TA) and performance (Swish-TC), allowing practitioners to choose based on their specific constraints.
• Practicality: The discovery that these functions perform exceptionally well as non-parametric (fixed $\beta$) functions is highly significant. It implies that Swish-T can be deployed in production models without the computational cost of updating activation parameters, offering a "plug-and-play" improvement over standard ReLU or Swish.

The paper concludes that Swish-T, particularly the Swish-TC variant, provides a robust, versatile, and high-performance alternative for modern deep learning architectures, effectively balancing gradient stability, learning speed, and model accuracy.