🤖 machine learning

Unlocking Feature Learning in Gated Delta Networks at Scale

This paper derives and validates Maximal Update Parametrization ( $\mu$ P) scaling rules for Gated Delta Networks, demonstrating that these rules enable zero-shot hyperparameter transfer and stable training across model widths, unlike standard parametrization.

Original authors: Yifeng Liu, Quanquan Gu

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yifeng Liu, Quanquan Gu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a giant robot to write stories. The bigger the robot gets (more "brain cells" or parameters), the harder it is to teach it the right lessons. Usually, if you find the perfect teaching speed (learning rate) for a small robot, you have to completely re-tune that speed when you switch to a giant robot. It's like finding the perfect water pressure for a garden hose; if you switch to a fire hose, that same pressure might either do nothing or blow the hose apart.

This paper is about a new type of robot brain called a Gated Delta Network. These are special because they are very efficient at remembering long stories without getting overwhelmed, unlike the standard "Transformer" brains used today. However, because they work differently, the old rules for teaching them didn't work.

Here is the breakdown of what the authors did, using simple analogies:

1. The Problem: The "One-Size-Fits-All" Rule Doesn't Work

For a long time, scientists used a standard rulebook (called Standard Parametrization) to teach these robots. This rulebook said, "If you double the size of the robot, keep the teaching speed the same."

The Result: This worked fine for the old, standard robots. But for these new, efficient Gated Delta Networks, it failed. When they tried to use the same teaching speed on a big robot as a small one, the big robot either learned nothing or went crazy.

2. The Solution: A New "Rulebook" (µP)

The authors created a new, smarter rulebook called Maximal Update Parametrization (µP). Think of this as a "Universal Translator" for teaching speeds.

The Goal: To find a set of rules where the teaching speed you find for a tiny robot works perfectly for a massive robot without any changes. This is called "zero-shot transfer."
The Challenge: These new robots have a special "memory loop" (they remember things by updating a state over and over). The old math didn't know how to handle this loop, so the authors had to do new math to figure out exactly how to adjust the teaching speed for every part of the robot.

3. The Discovery: Different Parts Need Different "Teaching Speeds"

The authors discovered that not all parts of this new robot brain are the same. They found two specific parts that needed special treatment, which was a surprise:

The "Gatekeepers" (Gating Weights): Imagine the robot has little doors that decide what information to let in. The authors found that the "knobs" controlling these doors need to be turned much more gently (a slower learning rate) than the rest of the brain. If you turn them too fast, the robot forgets how to open the doors.
The "Volume Knobs" (Scalar Parameters): There are also little volume knobs that adjust the strength of the signal. These needed to be turned much more aggressively (a faster learning rate) than the rest of the brain.

It's like tuning a complex musical instrument: most strings need a standard tuning, but the bass strings need a very loose turn, while the tiny high-pitched strings need a very tight turn, or the music sounds terrible.

4. The Proof: It Works in the Real World

The authors didn't just do the math on paper; they built these robots and tested them.

The Experiment: They trained these robots on a massive library of text (FineWeb-Edu) to see how well they learned.
The Result:
- When they used the old rules, the robots of different sizes needed completely different teaching speeds.
- When they used their new rules, the teaching speed found for the smallest robot worked perfectly for the largest robot. The robots learned steadily and stably, no matter how big they got.

Summary

This paper is a "user manual" for training a specific, highly efficient type of AI. The authors figured out that because this AI works differently than the standard ones, you can't just copy-paste the training settings. They derived a new set of settings that allow you to train a tiny version of the AI, find the perfect settings, and then instantly apply those exact same settings to a giant version of the AI, saving a huge amount of time and computer power.

Technical Summary: Unlocking Feature Learning in Gated Delta Networks at Scale

Problem Statement
The scaling of Large Language Models (LLMs) faces two primary bottlenecks: the steep computational cost of training larger models and the quadratic complexity of standard Transformer self-attention. Linear recurrent architectures, such as Gated Delta Networks (GDN), have emerged as efficient alternatives that maintain linear-time inference while capturing long-range dependencies. However, a critical gap exists in scaling these models effectively. While Maximal Update Parametrization (µP) has successfully enabled zero-shot hyperparameter transfer (specifically learning rates) for standard Transformers and some linear models, its application to Gated Delta Networks remains unexplored. The core challenge lies in the architecture's recurrent state dynamics, which are updated via data-dependent gating mechanisms and full-matrix delta rules rather than the scalar recurrences or standard feedforward structures assumed by existing µP derivations. Prior work on Structured State Space Models (SSMs) like Mamba has shown that vanilla µP fails for diagonal SSMs, but GDN's use of full-matrix state updates renders those specific corrections inapplicable.

Methodology
The authors address this gap by rigorously deriving the µP formulation for Gated Delta Networks. The methodology involves propagating coordinate-size estimates through the entire forward pass, including the unique gating mechanisms and recurrent state transitions, to determine the necessary scaling rules for initialization, forward multipliers, and learning rates.

Key analytical steps include:

Forward Pass Analysis: The authors derive the coordinate sizes of projected features, the latent state $S_t$ , and the readout. They identify that the L2-normalization of queries and keys, combined with the rank-one write updates, results in specific variance scalings. Crucially, they propose inserting a $\sqrt{d}$ multiplier before the RMSNorm layer to ensure the input to the normalization layer maintains $\Theta(1)$ coordinate size, preventing the disruption of gradient scaling required for feature learning.
Gating Mechanism Analysis: The paper analyzes the non-linear gating parameters $\alpha_t$ (decay) and $\beta_t$ (write strength). Unlike standard weights, these are derived from scalar pre-activations involving trainable weight vectors ( $W_\alpha, W_\beta$ ) and scalar parameters ( $a_{log}, b$ ). The analysis reveals that these gating components do not follow standard µP scaling laws due to their specific roles in the state update equation.
Backward Pass and Learning Rate Derivation:
- AdamW: Due to AdamW's normalization of gradients by their second moment, the effective update magnitude is consistent across weight classes, simplifying the learning rate requirements.
- SGD: Under plain SGD, the raw gradient magnitudes vary significantly across different weight classes. The authors derive that the gating weight matrices ( $W_\alpha, W_\beta$ ) require a learning rate scaling of $\Theta(1/\sqrt{d})$ , and the scalar gating parameters ( $a_{log}, b$ ) require a scaling of $\Theta(\sqrt{d})$ . These scalings deviate from the standard µP setting where learning rates are typically uniform or follow a simple $1/n$ rule.

Key Contributions

Theoretical Derivation: The paper provides the first complete µP formulation for Gated Delta Networks. It derives coordinate-size estimates through the full forward pass and establishes principled initialization variances and forward multipliers for all weight classes.
Novel Scaling Rules: The authors discover that the gating components require non-standard learning rate scalings under SGD:
- Gating weight matrices ( $W_\alpha, W_\beta$ ) require $\Theta(1/\sqrt{d})$ scaling.
- Scalar gating parameters ( $a_{log}, b$ ) require $\Theta(\sqrt{d})$ scaling.
Empirical Validation: The authors pretrain GDN language models across multiple widths (ranging from 256 to 1536 hidden dimensions) using both AdamW and SGD optimizers.
- Results: Under the proposed µP configuration, the optimal learning rate transfers zero-shot across all model widths for both optimizers. In contrast, Standard Parametrization (SP) and the original µP configuration fail to transfer, with optimal learning rates shifting significantly as model width increases.
- Stability: The proposed configurations enable stable learning-rate transfer, confirming the theoretical derivations.

Significance and Claims
The paper claims that its analysis resolves the open problem of parametrizing Gated Delta Networks for feature learning at scale. By demonstrating that the derived µP formulation allows for zero-shot hyperparameter transfer, the work validates the correctness of the theoretical derivation and its practical utility. The authors emphasize that this is the first work to derive and validate a µP-consistent parametrization specifically for Gated Delta Networks, distinguishing it from prior work on diagonal SSMs. The findings suggest that proper parametrization is essential for scaling linear recurrent models and that the specific gating mechanisms in GDN necessitate unique scaling rules that differ from both standard Transformers and other linear architectures. The authors hope these derivations will guide future research into the scaling laws of other linear or hybrid architectures.