🤖 machine learning

SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations

The paper introduces SG-Blend, a per-layer adaptive activation function that learns an optimal interpolation between a novel bias-corrected Swish variant (SSwish) and GELU, demonstrating reduced training variance and superior performance across natural language processing and computer vision tasks compared to standard fixed activations.

Original authors: Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi

Published 2026-09-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Deep learning, the technology behind modern artificial intelligence, relies on vast networks of mathematical pathways that process information layer by layer. To function, these networks need special switches called activation functions. These switches decide how much information passes from one layer to the next, shaping the network's ability to learn complex patterns. For years, researchers have relied on a few standard switches, such as Swish and GELU, which act as the default settings for almost every modern AI model. However, these standard switches are rigid; they apply the exact same shape and behavior to every single layer of a network, regardless of whether that layer is at the beginning, middle, or end of the process. This one-size-fits-all approach creates a problem: while the network might work well on average, its performance can swing wildly depending on how it was randomly initialized, making it difficult to trust or reproduce results consistently.

A team of researchers has proposed a new solution called SG-Blend, a flexible switch that allows each layer of a neural network to find its own perfect balance between two different behaviors. Instead of forcing every layer to use the same rigid setting, this new method lets each layer learn how much it prefers one type of switch over another. The researchers found that by giving each layer this small amount of freedom, the entire network becomes significantly more stable. In tests on language models, this approach reduced the unpredictability of results by forty-two percent compared to the standard method, while also achieving the highest accuracy scores. The key discovery is that the best performance comes not from picking a single perfect switch, but from allowing the network to smoothly interpolate, or blend, between two well-known options, letting the early layers behave differently from the deep layers.

The core idea behind this work is that the rigid nature of current activation functions creates a mismatch with how modern neural networks are built. In older networks, a different type of normalization helped smooth out the flow of information, masking the flaws of these fixed switches. But in the newer, deeper architectures used for language and vision, that smoothing effect is gone. Without it, the fixed switches cause the flow of information to become unstable as it travels through the many layers of the network. The researchers observed that this instability leads to high variance, meaning that if you run the same training experiment twice with slightly different random starting points, you might get two very different results. One run might produce a highly accurate model, while the next might fail to learn effectively, simply because the fixed switches could not adapt to the specific needs of each layer.

To solve this, the team introduced a new mechanism that combines a corrected version of the Swish switch with the GELU switch. They did not simply mix them randomly; instead, they created a system where every single layer in the network has its own small set of adjustable knobs. One knob controls how much the layer leans toward the Swish style, another controls the sharpness of the transition, and a third adjusts the baseline level of the signal. During training, the network learns to set these knobs automatically. The researchers discovered that the network naturally settles into a state where most layers choose a middle ground, blending the two styles roughly equally. This suggests that neither pure Swish nor pure GELU is the perfect answer for every part of the network; rather, the optimal state is a custom blend that varies slightly from layer to layer.

The results of this approach were measured across several different tasks. In a study involving sentiment analysis on movie reviews, the new method matched the peak accuracy of the best existing models but did so with far greater consistency. While the standard method showed a wide gap between its best and worst results across different random starts, the new method kept the results tightly clustered. This consistency is crucial for practical applications, as it means a developer can train a model once and be confident it will perform well, rather than having to run dozens of experiments to find a lucky starting point. Furthermore, when tested on a large-scale language model trained to predict the next word in a sentence, the new method achieved the lowest error rate among all the models tested, proving that the benefits extend beyond simple classification tasks to complex generative models.

The researchers also investigated why this blending works so well by looking at how the network's internal signals flowed. They found that the new method maintained a steady and uniform flow of information from the first layer to the last, avoiding the spikes and drops that often occur with fixed switches. This stability was confirmed by observing that the new method's behavior sat neatly between the behaviors of its two components, effectively borrowing the strengths of both without introducing new weaknesses. Interestingly, the network also learned to adjust the baseline level of the signal differently for early layers compared to later ones, a nuance that fixed switches cannot achieve. This ability to adapt the internal state of each layer individually appears to be the secret to the improved stability and performance.

However, this flexibility comes with a cost. Because the new method requires the network to calculate two different switch behaviors for every layer and then blend them, it takes more time to train. The researchers measured this overhead and found that training a model with this new method took about thirty-three percent longer than using the standard GELU switch on the same hardware. While the network learns more reliably and produces better results, the extra time required is a significant factor for those who need to train models quickly or on limited computing resources. The team acknowledges this trade-off, noting that the benefit of reduced variance and higher accuracy must be weighed against the increased time and computational power needed to reach those results.

Despite the extra time required, the findings suggest a shift in how we think about neural network design. The success of SG-Blend indicates that the rigid, one-size-fits-all approach to activation functions may be a limitation of the past. By allowing the network to learn its own internal settings, researchers can build models that are not only more accurate but also more robust and predictable. The study demonstrates that the path to better artificial intelligence may not lie in finding a single, perfect mathematical formula, but in creating systems flexible enough to adapt their own internal mechanics to the specific demands of the task at hand. This approach offers a promising direction for future research, particularly for the large, complex models that drive modern language and vision technologies.

Technical Summary: SG-Blend

Problem Statement

Deep neural architectures across domains (NLP, computer vision) rely heavily on fixed-shape activation functions, primarily Swish and GELU. While effective, these functions exhibit structural rigidity: they apply the exact same nonlinear gating profile across all layers, regardless of depth or specific role within the network.

This rigidity is particularly problematic in Transformer Feed-Forward Network (FFN) blocks utilizing LayerNorm. Unlike BatchNorm, which normalizes across the batch dimension and historically co-evolved with activation functions to suppress gradient pathologies, LayerNorm does not normalize pre-activations across the batch. Consequently, the gradient-stabilizing effects seen in CNN pipelines are absent in modern transformers. This leads to:

Inconsistent gradient flow across depth.
High sensitivity to random initialization, resulting in severe seed-to-seed performance variance (e.g., in BERT-IMDB, GELU's worst-case performance trails its best by 1.42 percentage points, enough to reverse head-to-head comparisons).
Inability to adapt the gating shape to the specific needs of individual layers.

Methodology

The authors propose SG-Blend, a per-layer adaptive activation function that learns an interpolation between two well-understood activation families: a bias-corrected Swish variant (SSwish) and GELU.

1. SSwish (Bias-Corrected Swish)

Standard Swish is defined as $x \cdot \sigma(\beta x)$ . The authors introduce SSwish to address the positive DC bias inherent in Swish, which LayerNorm fails to suppress:
$\text{SSwish}(x; \beta, \gamma) = x \cdot \sigma(\beta x) - \gamma$

$\beta$ (Sharpness): A learnable parameter initialized to 1.0, constrained to $[0.1, 10]$ . It controls the transition sharpness.
$\gamma$ (Bias): A learnable bias parameter initialized to 0.0. It shifts the output mean to maintain near-zero activation statistics per layer, a property LayerNorm alone does not guarantee.

2. SG-Blend (Convex Interpolation)

SG-Blend performs a convex interpolation between SSwish and GELU:
$\text{SGBlend}(x; \alpha, \beta, \gamma) = \alpha \cdot \text{SSwish}(x; \beta, \gamma) + (1 - \alpha) \cdot \text{GELU}(x)$

$\alpha$ (Blend Coefficient): A per-layer learnable parameter initialized to 0.5, constrained to $(0, 1)$ via sigmoid reparameterization. It determines the preference between the SSwish and GELU families.
Parameter Count: Each Transformer FFN block adds only three scalar parameters ( $\alpha, \beta, \gamma$ ) trained end-to-end.

3. Implementation Details

GELU Approximation: The GELU component within SG-Blend uses the sigmoid approximation $x \sigma(1.702 x)$ for differentiability and closed-form gradients, while baseline comparisons use the exact error-function implementation.
Constraints: $\beta$ is projected to $[0.1, 10]$ after each gradient step; $\alpha$ is constrained via sigmoid; $\gamma$ is unconstrained.

Key Contributions

SSwish: A bias-corrected, parametric Swish variant with learnable sharpness ( $\beta$ ) and bias ( $\gamma$ ) to promote zero-centered activation statistics per layer.
SG-Blend: A per-layer convex interpolation mechanism between SSwish and GELU. It allows each layer to locate its own optimum along the continuum at a minimal cost (3 scalars per block).
Interpretability: The blend coefficient $\alpha$ serves as a direct, interpretable "preference score" for activation families per layer, enabling analysis of learned architectural dynamics.

Experimental Results

1. Gradient Stability (Deep Transformer Stacks)

In 24-block deep stacks (mimicking BERT-large), SG-Blend demonstrated superior gradient propagation stability compared to fixed activations.

Gradient Ratio: SG-Blend achieved a ratio of 2.007 (first-block norm ÷ last-block norm), sitting between GELU (1.953) and Swish (2.099).
Variance: Unlike ReLU or CurvatureTuning, which approached a ratio of 1.0 but suffered from catastrophic cross-seed variance (std dev ~50% of mean), SG-Blend maintained mathematical stability with low variance across 50 seeds.

2. BERT-Style IMDB Classification

Trained from scratch on IMDB (5 seeds, BERT-base configuration):

Accuracy: SG-Blend achieved 81.31% accuracy, statistically tied with Swish (81.32%) and outperforming GELU (81.23%).
Variance Reduction: Crucially, SG-Blend reduced seed-to-seed variance by 42% compared to GELU (Std: 0.27 pp vs. 0.47 pp). This implies that a practitioner running a single seed is significantly more likely to achieve near-optimal performance with SG-Blend.

3. Autoregressive Pretraining (WikiText-103)

Using a GPT-2-style decoder-only transformer (~25M parameters):

SG-Blend achieved the lowest validation perplexity (49.10), outperforming GELU (49.47) and ParametricSwish (49.82). This confirms the method's efficacy extends beyond fine-tuning to long-horizon autoregressive pretraining.

4. Computer Vision (CIFAR-10 & CIFAR-100)

CIFAR-10: SG-Blend achieved the highest accuracy on both ResNet18 (93.23%) and ResNet50 (92.71%), outperforming ReLU, Swish, and GELU.
CIFAR-100 (WideResNet-28-10): In a BatchNorm-heavy setting, ReLU retained its lead (80.36%), while SG-Blend (77.33%) outperformed GELU (77.01%) and Mish (75.73%). The authors note this is expected, as BatchNorm mitigates the specific gradient pathologies SG-Blend targets, and the result is based on a single seed.

5. Ablation Studies

Interpolation Structure: Even with $\alpha$ fixed at 0.5 (no learned blending), the blend structure outperformed pure GELU and pure SSwish, confirming the interpolation itself drives gains.
Depth Gradient: Analysis of learned parameters revealed a clear depth gradient for $\gamma$ (bias), where early layers learned a positive shift and deeper layers converged to near-zero. This suggests early layers benefit more from output centering, an adaptation impossible with fixed activations.

6. Computational Overhead

SG-Blend incurs a 33% increase in end-to-end training time compared to GELU (approx. 60s/seed vs. 45s/seed on T4). This is due to the need to evaluate both SSwish and GELU forward passes separately (11.7× forward latency per activation). The overhead is comparable to other multi-parameter learnable activations like Mish and SMU.

Significance and Claims

The paper claims that SG-Blend provides a robust, top-tier performance mechanism for Transformer FFN blocks by explicitly addressing the lack of gradient stabilization in LayerNorm-based architectures.

Reproducibility: The primary practical significance is the 42% reduction in seed-to-seed variance, offering a more reliable training experience for practitioners who cannot afford extensive hyperparameter or seed searching.
Adaptability: It demonstrates that different layers within the same network benefit from different activation shapes (evidenced by the learned $\gamma$ depth gradient), a capability fixed activations lack.
Generalization: The method generalizes effectively from classification to autoregressive pretraining and shows promise in vision tasks, though the authors remain modest about its superiority in BatchNorm-heavy CNNs where ReLU remains dominant.

The authors conclude that while the computational cost is a barrier, the gains in stability and performance make SG-Blend a compelling alternative to fixed activations, particularly in the Transformer domain.