🤖 machine learning

Gate-Zero Growth: A Geometric Framework for Function-Preserving Continual Learning

This paper introduces "gate-zero growth," a function-preserving continual learning framework that utilizes zero-initialized gates to induce rank separation in the functional Jacobian, thereby enabling safe capacity expansion with near-zero forgetting and controlled functional drift.

Original authors: Dante Lok

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dante Lok

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master chef who has spent years perfecting a single, delicious recipe. You know exactly how to make it taste just right. Now, imagine someone asks you to cook a massive banquet for a thousand people, but you only have the ingredients for a small dinner. The obvious solution is to buy a bigger kitchen and hire more cooks, but there's a catch: if you just throw new people into the kitchen and start cooking, the original recipe might get ruined. The new cooks might accidentally change the spices in the old dish, or the new pots might clash with the old ones, and suddenly, your famous dish tastes like mud. This is the daily struggle of "Continual Learning" in artificial intelligence. AI models are like those master chefs; they learn from data, but when we try to teach them new things or make them bigger, they often suffer from "catastrophic forgetting," where they forget everything they knew before. Scientists have been trying to find a way to grow these AI brains without breaking the parts that already work, hoping to build smarter systems without having to throw away the old ones and start from scratch.

This paper introduces a clever new trick called "Gate-Zero Growth" to solve that kitchen disaster. Think of the AI model as a multi-layered cake. Usually, if you want to make the cake taller, you just stack more layers on top. But if you just plop a new layer on, it might squish the cake below it or change how the whole thing tastes. The authors propose a different way: they add new layers, but they attach them with a special "gate" that starts completely closed (set to zero). Because the gate is closed, the new layers are invisible to the cake; they don't touch the flavor or the texture at all. The cake tastes exactly the same as before, even though it's now much bigger. This is called "function-preserving" growth because the original function (the taste) is preserved perfectly.

The researchers tested this idea by taking a medium-sized AI model (about 300 million parameters) and growing it into a giant one (about 857 million parameters). They added new layers using these "zero gates" and then tried to teach the giant model a new language task. The results were impressive: when they used their special "Gate-Zero" method, the model remembered its old knowledge almost perfectly, with almost zero forgetting. In fact, the old knowledge stayed so intact that the model's performance on the old task barely changed at all.

However, the paper also warns us about what happens if you don't use this trick. They tested a "naive" method where they just stacked new layers without the zero gates (which they call "Gstack"). This was like adding a new layer of cake that immediately started squishing the old one. The result was a disaster: the model forgot its old knowledge almost entirely, and the new cake tasted terrible. This proves that simply adding more parts isn't enough; you need the right "gate" mechanism to keep the old parts safe.

The paper also discovered something interesting about how to train these new, giant models. They found two main ways to run the training. The first is "Isolation," where you freeze the old parts of the model completely and only let the new parts learn. This is the safest bet, guaranteeing the old recipe stays exactly the same. The second way is "Freeze-Nothing," where you let the whole model learn together. Surprisingly, this second method actually made the model better at the new task, and it didn't ruin the old task either—it just changed the flavor slightly in a way that was actually an improvement, not a mistake.

The authors explain that this works because of a hidden geometric structure in the math behind the AI. When the gates are zero, the new parts of the model are mathematically "flat" and don't interfere with the old parts. It's like adding a new room to a house that is currently sealed off; you can build the new room however you want, and it won't change the layout of the old rooms until you decide to open the door. The paper shows that this "Gate-Zero" idea isn't just a one-off trick; it's the same underlying principle behind several other popular AI techniques, making it a fundamental rule for safely growing AI brains.

In short, this paper shows us that if we want to build bigger, smarter AI models without losing the knowledge they already have, we shouldn't just throw new parts onto the old ones. Instead, we should add them with a "zero gate" that keeps them out of the way until they are ready. This allows the AI to grow like a tree adding new branches without shaking the roots, ensuring that the wisdom of the past remains intact while the future gets built.

Technical Summary: Gate-Zero Growth

Problem Statement

The dominant paradigm for increasing language model capability is training larger models from scratch. Model growth—expanding an existing trained model by adding parameters—offers a compelling alternative, provided the expanded model can be fine-tuned without catastrophic forgetting of the knowledge acquired by the smaller model.

While Function-Preserving (FP) growth methods exist (e.g., Net2Net, bert2BERT), they typically satisfy preservation only at the moment of growth by construction. The geometric structure created in parameter space by these initializations, and why certain Continual Learning (CL) strategies (like freezing old parameters) succeed or fail on the grown model, has remained unclear. Existing literature often explains CL behavior operationally rather than through the underlying geometry of the growth operator.

Methodology: Gate-Zero Growth

The authors propose Gate-Zero Growth, a geometric framework for FP depth expansion in CL. The core mechanism involves inserting new residual blocks where the contribution is gated by a scalar $\alpha_\ell$ , initialized to zero ( $\alpha_0 = 0$ ).

Theoretical Framework

The paper establishes that under a transversality condition, Gate-Zero Growth induces a specific geometric structure in the functional Jacobian and Fisher Information Matrix:

Rank Separation: At the growth point ( $\alpha=0$ ), the functional Jacobian $J(\theta')$ decomposes cleanly. The $K$ new-weight directions are exactly flat (zero gradient contribution to the output), while the new gate directions provide the only first-order source of functional variation.
Jacobian Decomposition: The post-growth Jacobian rank satisfies $rank(J(\theta')) = r + rank((I - P_{old})J_\alpha)$ , where $r$ is the rank of the old Jacobian. Under transversality, the new gate directions are linearly independent of the old parameter space.
Sparse Fisher Structure: The empirical Fisher Information Matrix exhibits a sparse block structure. Specifically, the new-weight Fisher block and all cross-terms involving new weights are identically zero. This implies that second-order methods (like EWC) place no penalty on new-weight movement at the growth point.
Controlled Departure: As gates open during CL (i.e., $\alpha$ moves away from 0), the departure from the FP locus is bounded. Function drift is $O(\|\alpha\|^2)$ and Jacobian leakage is $O(\|\alpha\|_\infty)$ .

Unification

The authors argue that this geometric analysis unifies seemingly distinct methods:

LoRA (with $B=0$ initialization)
ReZero (residual scaling)
Zero-init Adapters
Gate-Zero Growth

All share the structural property of adding parameters through an additive branch factoring through a zero-initialized gate. Conversely, non-FP methods like Gstack (which duplicates blocks without zero-initialization) fail to preserve the Jacobian structure, causing immediate functional drift.

Key Contributions

Gate-Zero Growth Operator: A specific FP operator for continual learning that adds residual blocks via zero-initialized gates. It achieves exact function preservation at growth and near-zero forgetting ( $\Delta A < 0.1$ ) during CL at scale (300M $\to$ 857M).
Geometric Framework: A unified theoretical system proving that zero-initialization induces rank separation, sparse Fisher blocks, and exact coordinate-projection isolation. This framework applies to LoRA, ReZero, and zero-init adapters, establishing Gate-Zero Growth as the canonical instance of this shared local geometry.
Predictive Structural Validation: The paper provides three falsifiable predictions verified empirically:
- Coordinate freezing is exactly preserving only under FP growth (Gstack + Isolation is structurally undefined).
- LoRA-CL admits no separate "Isolation" row because its frozen-base construction is structurally already Isolation.
- $\Delta A < 0.1$ for every zero-init FP operator tested under Isolation.

Experimental Results

The framework was evaluated on a 300M $\to$ 857M Transformer adapted from WikiText-103 (Task A) to BookCorpus (Task B).

Function Preservation at Growth

Gate-FP: Exact function preservation (bit-exact for dense, within numerical tolerance for MoE). $\Delta PPL_A = 0.00$ .
Gstack (Negative Control): Immediate degradation of Task A performance ( $\Delta PPL_A = +46.11$ ) before any CL training begins.

Continual Learning Performance

Under the Isolation protocol (freezing all old parameters):

Gate-FP + Isolation: Achieved $\Delta A = +0.04$ (near-zero forgetting) while reducing Task B perplexity from 560.75 to 28.41.
Gstack + Isolation: Failed structurally. Since Gstack does not preserve the function at growth, freezing old parameters does not preserve the function. The method suffered catastrophic forgetting ( $\Delta A > +1000$ ).
Comparison to Scratch: A model trained from scratch (857M) on both datasets reached a final state of $PPL_A=43.23$ , but its best validation checkpoint was $17.40$. The authors note that Gate-FP + Isolation does not necessarily outperform a properly stopped scratch model but offers a structural guarantee of preservation.

MoE Validation

In a Mixture-of-Experts (MoE) setting (706M $\to$ 2.5B), Gate-FP + Isolation achieved $\Delta A = +0.20$ , confirming the preservation mechanism transfers across architectures. However, plasticity was lower than in dense models (Task B PPL improved only 12x vs. 20x in dense), attributed to clone-block redundancy where new experts failed to differentiate sufficiently from frozen sources.

Ablation Studies

Freezing Strategy: The "binding constraint" for preservation is freezing old weights, not just old gates. "Freeze Nothing" (allowing old weights to update) actually yielded better joint performance ( $\Delta A = -1.98$ , indicating improvement on Task A, though technically a drift from the original function) but at the cost of exact preservation.
Growth Timing: The preservation property is robust to the maturity of the base model (growing at 25%, 50%, 75%, or 100% of training).
Gate Initialization: Starting with a small non-zero gate ( $\alpha_0 = 0.1$ ) allows faster gradient flow but results in a non-FP starting point; CL training can recover some performance, but exact preservation is lost.

Significance and Claims

The paper claims that Gate-Zero Growth provides the first unified geometric explanation for why "freezing old parameters" works in function-preserving growth. It posits that the success of Isolation is not merely a heuristic but a consequence of rank separation and exact coordinate orthogonality induced by zero-initialization.

Canonical Instance: The authors establish Gate-Zero Growth as the canonical instance of a shared local geometry governing safe capacity activation in CL, unifying LoRA, ReZero, and adapters under one theoretical umbrella.
Two Operating Points: The framework recommends two distinct operating points:
1. Isolation: For exact preservation ( $\Delta A \approx 0$ ), suitable for regulatory or A/B testing scenarios where bit-equivalent behavior on old data is required.
2. Freeze-Nothing: For joint frontier optimization (best performance on both tasks), accepting a controlled departure from the original function.

The authors explicitly state limitations: the geometric estimators (Hessian, gradient covariance) are coarse local diagnostics, and the MoE plasticity gap remains an open problem requiring further investigation into MoE-specific operators. The paper does not claim to solve all CL problems but specifically addresses the structural geometry of growth operators.