💬 NLP

Bounded Hyperbolic Tangent: A Stable and Efficient Alternative to Pre-Layer Normalization in Large Language Models

The paper proposes Bounded Hyperbolic Tanh (BHyT), a stable and efficient drop-in replacement for Pre-Layer Normalization that eliminates depth-related instability through data-driven input bounding while achieving faster training and higher throughput compared to RMSNorm.

Original authors: Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, Kyungwoo Song

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hoyoon Byun, Youngjun Choi, Taero Kim, Sungrae Park, Kyungwoo Song

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a very tall tower out of blocks. In the world of Artificial Intelligence, these "blocks" are layers of a computer program (a Large Language Model) that learns to speak and reason.

For a long time, the standard way to build these towers was to use a "Pre-Layer Normalization" (Pre-LN) tool. Think of this tool as a quality control inspector who stops at every single floor of the tower to measure the blocks, check if they are the right size, and smooth them out before adding the next floor.

The Problem: The Inspector is Too Slow and the Tower Gets Wobbly

The paper points out two main issues with this old method:

It's Slow: Because the inspector has to stop and do math at every floor, the construction process gets bogged down. The more floors you add, the slower the whole project becomes.
The "Curse of Depth": As the tower gets taller, the blocks on the upper floors start to get huge and wobbly. The "signal" (the information) traveling up the tower gets distorted, making the tower unstable. The paper calls this the "curse of depth."

Some people tried to fix this by removing the inspector entirely and just using a simple "tanh" function (a mathematical curve that squishes numbers). This was fast, like building the tower without stopping. But, without the inspector, the blocks on the upper floors still grew too big and the tower became unstable again.

The Solution: BHyT (The Smart, Bounded Builder)

The authors propose a new method called BHyT (Bounded Hyperbolic Tanh). You can think of BHyT as a smart, self-regulating builder that combines the best of both worlds.

Here is how BHyT works, using simple analogies:

1. The "Speed Bump" (Bounded Input)
Instead of letting the blocks grow infinitely large as they go up the tower, BHyT puts up a "speed bump" or a fence. It uses a mathematical rule (based on the Chebyshev inequality, which is like a safety net) to say, "No matter how big the data gets, we will gently squeeze it back into a safe, comfortable range before it moves to the next layer."

Why this helps: It prevents the blocks from getting too huge (which causes instability) without needing a full, slow inspection at every single step.

2. The "One-Time Check" (Variance Approximation)
The old method (Pre-LN) calculated the exact size of the blocks at every single floor. BHyT is smarter:

It does a precise measurement once at the bottom of the floor.
For the second part of the floor, instead of measuring again, it uses a quick, educated guess (an approximation) based on the first measurement and the known design of the tower.
Why this helps: It saves a massive amount of time and computer power because it doesn't have to stop and count every single brick twice.

The Results: A Faster, Steadier Tower

The paper tested this new builder on models of different sizes (from small 374-million-block towers to larger 3-billion-block towers). Here is what they found:

Stability: The towers built with BHyT didn't wobble. The "blocks" (activations) stayed the right size all the way to the top, preventing the "curse of depth."
Speed: Because BHyT didn't stop to do heavy math at every step, it built the tower 1.6% faster during training and generated text 1.77% faster than the standard method.
Quality: Even though it was faster, the final tower was just as smart. It performed better on language tests and reasoning puzzles than the old methods, and it didn't "forget" what it learned after being fine-tuned.

Summary

In short, the paper argues that BHyT is a better way to build AI models. It replaces the slow, repetitive "quality inspector" with a smart, bounded builder that keeps the data in check with a safety fence and uses quick estimates to save time. This allows us to build taller, more stable, and faster AI models without sacrificing their intelligence.

Technical Summary: BHyT – A Stable and Efficient Alternative to Pre-Layer Normalization

1. Problem Statement

Large Language Models (LLMs) rely heavily on Pre-Layer Normalization (Pre-LN), typically implemented via RMSNorm, to stabilize training and enable effective transfer learning. However, as model depth increases, the standard Pre-LN design suffers from two critical limitations:

The Curse of Depth: Recent analyses indicate that the interaction between residual connections and Pre-LN drives a rapid growth in the magnitude and variance of hidden states across layers. This escalation pushes the block Jacobian toward the identity matrix, causing deeper layers to behave like costly identity mappings and destabilizing training.
Efficiency vs. Stability Trade-off: Existing solutions often address only one aspect. Stability-oriented methods like Peri-LN (normalizing before and after sublayers) or LayerNorm Scaling (LNS) suppress variance growth but incur significant computational overhead due to repeated statistical computations. Conversely, efficiency-oriented, normalization-free methods like Dynamic Tanh (DyT) remove statistical overhead but fail to explicitly control depth-wise residual growth, leading to potential saturation and instability in deep networks.

The central challenge is to design a mechanism that combines the low overhead of bounded transformations with explicit control over residual-stream scale and variance to prevent depth-wise drift without sacrificing training throughput.

2. Methodology: Bounded Hyperbolic Tanh (BHyT)

The authors propose BHyT, a drop-in replacement for Pre-LN that couples a hyperbolic tangent nonlinearity with explicit, data-driven input bounding.

Core Mechanism

Unlike DyT, which relies on a learnable global scalar to rescale inputs, BHyT explicitly bounds the input to the $\tanh(\cdot)$ function using data-dependent statistics.

Probabilistic Input Bounding: To prevent saturation, BHyT constrains the pre- $\tanh$ argument to lie within a predefined interval $[-\lambda, \lambda]$ with high probability. Using Chebyshev's inequality, the method derives a scaling rule based on the input's mean ( $\mu_x$ ) and standard deviation ( $s_x$ ).
Ideal Form ( $BHyT^*$ ): Theoretically, the transformation rescales the input $x$ as:
$BHyT^*(x) = \gamma \odot \tanh\left(\frac{\lambda}{\kappa s_x + |\mu_x|} x\right)$
where $\gamma$ is a learnable scale parameter, $\lambda$ defines the target range, and $\kappa$ controls the scaling conservativeness based on a target probability $p$ . This ensures the input remains in the non-saturating linear regime of $\tanh$ with high probability.

Practical Implementation: Variance Approximation

Computing exact mean and variance at every Pre-LN site within a Transformer block would negate efficiency gains. To address this, BHyT introduces a block-level variance approximation:

First Site (Before Attention): Computes the exact per-instance variance of the input.
Second Site (Before MLP): Instead of recomputing the exact variance of the residual output ( $x + h_{Attn}$ $x + h_{A tt n}$ ), it approximates the variance using the previously computed input variance and a model-dependent estimate of the attention output variance.
- The attention output variance is approximated as $\tilde{s}^2_{hAttn} \approx \frac{1}{Td} \|W_V W_O\|_F^2 \cdot \frac{\lambda_{Attn}^2}{\kappa^2}$ , assuming uniform attention weights and zero-mean Gaussian entries.
- This approximation allows the second normalization to reuse statistics or use precomputed model parameters, avoiding a second reduction operation in the critical forward path.

Theoretical Stability

The paper provides a finite-depth variance propagation bound. Theorem 3.5 states that for a network of depth $L$ , if the hyperparameters satisfy $\lambda/\kappa < 1/\sqrt{L}$ , the output variance under BHyT is strictly smaller than that under LayerNorm Scaling (LNS) for every layer up to depth $L$ . This offers a theoretical guarantee that BHyT suppresses depth-wise variance growth more effectively than LNS.

3. Key Contributions

Proposal of BHyT: A drop-in replacement for Pre-LN that combines a bounded $\tanh(\cdot)$ transformation with data-driven input bounding to control depth-wise activation growth.
Theoretical and Practical Efficiency:
- Derivation of a finite-depth variance bound proving BHyT yields a smaller variance-propagation bound than LNS under specific conditions.
- Introduction of a block-level variance approximation that reduces repeated exact variance computations, significantly lowering latency.
Empirical Validation: Comprehensive evaluation on Llama-374M, Llama-1B, and Llama-3B models. The study demonstrates that BHyT improves the stability-efficiency trade-off while maintaining strong downstream performance.

4. Experimental Results

Experiments were conducted on Llama-style models (374M, 1B, and 3B) pretrained on the C4 corpus with token budgets proportional to model size (approx. 20 tokens per parameter).

Stability: BHyT successfully suppressed depth-wise growth in activation magnitude and variance, outperforming RMSNorm and DyT. The activation statistics remained controlled across layers, comparable to stability-oriented baselines like Peri-LN.
Performance:
- Pretraining: BHyT achieved the lowest pretraining loss and perplexity across all model scales. It also attained the highest average downstream accuracy on 3-shot benchmarks (e.g., ARC, MMLU, Winogrande), outperforming the second-best method by up to 0.6 points on the 1B model.
- Post-SFT: After supervised fine-tuning on Lima1K, BHyT models showed lower perplexity on C4 and WikiText-2 (indicating less forgetting and better generalization) and higher downstream accuracy compared to RMSNorm, Peri-LN, LNS, and DyT.
- Scale: In a 20B-token comparison on Llama-3B, BHyT outperformed the strong stability-oriented baseline Peri-LN by 3.4 percentage points in average downstream accuracy (pretraining) and 2.5 points (post-SFT).
Efficiency:
- Training Throughput: BHyT achieved an average of 1.6% faster training compared to RMSNorm.
- Generation Throughput: BHyT achieved 1,199.9 tokens/s on Llama-1B, surpassing RMSNorm by 1.77% and Peri-LN by 18.0%. While DyT remained the fastest (due to zero statistics), BHyT offered a superior trade-off between speed and stability.
- Ablation: The variance approximation mechanism provided a 14.9% speedup over the exact-form BHyT* without materially degrading convergence or downstream performance.

5. Significance and Claims

The paper claims that BHyT offers a practical path to deeper and more efficient foundation models. By unifying the low overhead of bounded transformations with explicit control over residual-stream variance, BHyT addresses the "curse of depth" without the computational penalties of double-normalization schemes.

The authors assert that scalable training can improve efficiency without sacrificing stability. BHyT is presented not merely as a theoretical alternative but as a robust, drop-in replacement for RMSNorm that enables stable pretraining and strong transfer performance across varying model scales (from 374M to 3B parameters) while reducing latency in both training and inference.