💬 NLP

Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate

This paper demonstrates that decoder-only Transformers can continue learning effectively by stacking new blocks onto a frozen substrate with a bounded active parameter budget, proving the viability of modular, layer-wise expansion even under extreme constraints like a low-rank binary token interface, albeit with a tradeoff in final perplexity compared to fully trainable monolithic models.

Original authors: A. Bochkov

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: A. Bochkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a skyscraper, but you have a very strict rule: you can only work on the top floor at any given time. Once a floor is finished, you seal it with concrete, lock the doors, and you are never allowed to go back inside to fix or change anything. You also have a limited budget for construction workers, so you can't hire a whole new crew every time you add a floor; you just move your small team up to the new level.

This paper asks a simple question: Can you still build a useful, tall skyscraper under these extreme rules?

Here is the breakdown of the research using everyday analogies:

1. The "Frozen Foundation" Problem

Usually, when training a giant AI (like a large language model), the computer needs to remember how to adjust every single part of the building as it learns. This requires a massive amount of memory, like needing a huge warehouse to store blueprints for every brick.

The researchers wanted to see if they could train a model while keeping the "active" part (the part they can change) small and constant.

The Strategy: They start with a small building. They train it, then freeze it (lock it down). Then, they add a new floor on top and only train that new floor.
The Catch: They never go back to the old floors to tweak them. They also keep the number of "workers" (trainable parameters) roughly the same, no matter how tall the building gets.

2. The "Bad Elevator" Test

To really stress-test this idea, the researchers created an extreme scenario. Imagine the ground floor (where the building connects to the street) is broken.

The Setup: Instead of a rich, detailed entrance, the building only has a tiny, 16-bit binary code (like a simple "on/off" switch) to identify which room you are in. It's a very poor, low-quality interface.
The Question: If the entrance is so bad and the lower floors are frozen solid, can the upper floors still learn to do complex tasks?
The Result: Surprisingly, yes. Even with this terrible, frozen entrance, the AI was able to grow taller and learn useful things. It proved that you don't necessarily need a perfect, flexible entrance for the whole building to function; the upper floors can figure out how to make sense of the limited information they receive.

3. The "Renovation" Compromise (LoRA)

In the longer experiments, the researchers realized that if the lower floors are too frozen, the building gets a bit rigid. To fix this without breaking their "small budget" rule, they used a technique called LoRA.

The Analogy: Think of LoRA as adding thin, flexible scaffolding or sticky notes to the walls of every floor. You aren't rebuilding the walls (the main structure is still frozen), but you can make small, temporary adjustments to how the rooms connect.
The Benefit: This allowed the model to "readjust" globally without needing to unlock and retrain the entire massive building.

4. The Trade-off: Quality vs. Efficiency

The paper is very honest about the results. It does not claim this method is better than the standard way of building (where you can tweak every floor at once).

The Standard Way: If you have enough memory and money, building the whole thing at once (monolithic training) produces a slightly "smarter" building (lower perplexity, better scores).
The New Way: The "grow-as-you-go" method produces a building that is slightly less perfect but is much more efficient. It uses significantly fewer "active workers" (about 105 million vs. 247 million in their tests) and requires less memory.

The Bottom Line

The paper concludes that useful learning can happen even under very strict constraints.

You can keep the bottom of the model frozen.
You can keep the number of trainable parts small.
You can even start with a very poor, fixed input interface.

While this isn't the "best" way to build the smartest possible AI, it proves that you don't need to unlock the whole building to keep adding useful height. It's a viable strategy for situations where memory is tight or you want to grow a model in stages without massive computational costs.

What the paper does NOT say:

It does not claim this is the best way to build AI for general use.
It does not claim this method produces the highest possible intelligence.
It does not suggest this is ready for clinical or commercial deployment without further testing.

It simply proves that growth is possible even when you are handcuffed by strict memory and training rules.

Technical Summary: Growing Transformers: Modular Composition and Layer-wise Expansion on a Frozen Substrate

1. Problem Statement

Large-scale language model training is frequently constrained by optimizer-state memory rather than parameter count alone. In standard dense monolithic pretraining, maintaining optimizer states for all trainable parameters creates a scaling bottleneck under fixed accelerator memory budgets.

This paper investigates a constrained training regime to answer a specific feasibility question: Can model depth continue to grow if the token interface is fixed, previously trained dense blocks remain frozen, and the active trainable parameter set is kept approximately constant? The authors explicitly position this as a study of tradeoffs and viability, not a claim that this regime surpasses tuned dense monolithic pretraining.

2. Methodology

The proposed regime, termed "Constructive Growth," operates under four strict constraints:

Fixed Token Interface: The input embedding layer is fixed and deterministic throughout training. The paper studies two variants:
- A frozen visual Unicode interface (from prior work).
- An extreme stress test: a frozen 16-dimensional binary token-ID code. Here, each token is represented by a 16-bit binary code (its token ID) deterministically lifted to the model dimension ( $d_{model}$ ) via coordinate repetition. This results in an input embedding matrix with a rank of at most 16, ensuring no learned high-dimensional semantics exist at the input.
Frozen Substrate: Once a block (or group of blocks) is trained, it is frozen. It is never reopened for full-parameter updates.
Bounded Active Budget: As depth increases, only the newest added blocks and the Language Model (LM) head are trained. The number of active trainable parameters remains approximately constant.
Optional LoRA Readjustment: In longer runs, LoRA (Low-Rank Adaptation) stages are interleaved between growth phases. These stages reallocate the same active trainable budget to adapters across all layers to allow limited global readjustment without reopening the dense weights.

Training Procedure:

Initialize a shallow model above the fixed interface.
Train the current top stack.
Freeze the trained dense blocks.
Add a new block group on top.
Train only the new group and the LM head.
(Optional) Interleave LoRA stages to adjust global representations while maintaining the active parameter budget.

3. Key Contributions and Claims

The paper makes narrow, empirically grounded claims rather than broad algorithmic superiority claims:

Definition of a Constrained Regime: The authors define a specific training protocol combining a fixed deterministic token interface, frozen lower blocks, and a bounded active parameter budget.
Feasibility of Growth on Minimal Interfaces: The study demonstrates that useful continued learning is possible even above a 16-dim binary token-ID interface, where the input subspace dimension is at most 16. This suggests that semantically rich trainable input embeddings are not a strict prerequisite for forming useful internal representations in deeper layers.
Quality-Memory Tradeoff: The paper quantifies the tradeoff between active parameter count and final perplexity.
- In a controlled 9-layer study, the constructive frozen model uses 105.0M active trainable parameters compared to 180.5M for an interface-matched monolithic frozen baseline and 247.6M for a fully trainable monolithic baseline.
- This reduction in active parameters comes at a cost: final training perplexity increases from 3.71 (monolithic frozen) to 4.12 (constructive frozen).
Viability of Long-Horizon Growth: A 16-layer experiment on a 68.9B-token dataset (FineWeb-Edu + Cosmopedia) using the 16-dim binary interface and interleaved LoRA stages reached 28.92% MMLU. This serves as a viability demonstration that growth can proceed under extreme constraints, though the changing data mixture prevents a clean causal comparison.

4. Results

Controlled 9-Layer Study:
- Constructive growth was stable, showing transient loss spikes upon stacking new layers followed by rapid recovery.
- The constructive model (105M active params) achieved downstream performance (MMLU, ARC, SQuAD) in the same broad range as monolithic baselines, though with higher perplexity.
- The constructive model trained above the 16-dim binary interface converged stably, proving that the extreme rank constraint (rank $\le$ 16) does not prevent the model from learning useful representations in the upper layers.
Long-Horizon 16-Layer Run:
- A 269.7M parameter model (base size) trained with interleaved LoRA stages achieved 28.92% MMLU.
- The authors note that because the data mixture changed across stages (increasing instruction/SFT mixing), this result is interpreted as a demonstration of viability rather than a causal estimate of growth efficiency.

5. Significance and Limitations

Significance:
The paper's primary contribution is a "narrow but useful claim": useful continued learning can proceed above a fixed minimal interface under a bounded active trainable-parameter budget. This is significant for practitioners with limited memory budgets who wish to expand model depth without the memory overhead of maintaining optimizer states for the entire model. It challenges the assumption that learned, high-capacity input embeddings are strictly necessary for depth growth.

Modesty and Limitations:
The authors are explicit about what the paper does not claim:

It does not claim that constructive growth matches the quality of tuned dense monolithic pretraining (perplexity is worse).
It does not claim that random-initialized stacking is the best generic growth strategy compared to function-preserving methods (e.g., Net2Net).
The controlled comparisons are single-run experiments; small metric differences are not statistically established.
The 9-layer study uses a common protocol without baseline-specific hyperparameter tuning, making it a regime comparison rather than an optimized best-vs-best comparison.
The long-horizon run confounds causal attribution due to changing data mixtures.
The 16-dim binary interface uses a specific deterministic code assignment; systematic comparisons of alternative code assignments are left for future work.

In conclusion, the paper establishes that while dense monolithic training remains the stronger baseline for quality when resources allow, a constrained regime of layer-wise expansion on a frozen substrate is a viable alternative for scenarios where active parameter budgets must be strictly bounded.