💬 NLP

From Growing to Looping: A Unified View of Iterative Computation in LLMs

This paper unifies depth growth and layer looping as complementary methods for inducing iterative computation in LLMs, demonstrating that they share convergent mechanistic signatures and can be effectively combined to significantly enhance reasoning capabilities.

Original authors: Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

Published 2026-02-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ferdinand Kapl, Emmanouil Angelis, Kaitlin Maile, Johannes von Oswald, Stefan Bauer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to solve a complex math problem or a logic puzzle. You have two main ways to make it smarter: you can either give it a bigger brain (more unique parts) or teach it to think longer (reusing the same parts over and over).

This paper explores two specific ways to make Large Language Models (LLMs) better at reasoning, and discovers that they are actually two sides of the same coin.

Here is the breakdown using simple analogies:

1. The Two Strategies: "Looping" vs. "Growing"

Think of a neural network as a factory assembly line where a product (the answer) moves from one station to the next.

Strategy A: Looping (The "Reusing" Method)
Imagine a factory where you only have 4 stations, but you tell the product to go through those same 4 stations 6 times in a row.
- How it works: The model uses the exact same "brain cells" (weights) repeatedly. It's like a student reading the same textbook chapter over and over to really understand it.
- The Benefit: You get a very deep thinker without needing to build a massive factory. It saves money on building the factory (parameters) but uses more electricity (compute) to run the conveyor belt longer.
Strategy B: Depth Growing (The "Expanding" Method)
Imagine you start with a small factory with 4 stations. As the product moves down the line, you suddenly copy and paste the middle stations to make the line longer.
- How it works: You start training a small model, and halfway through, you duplicate the middle layers to make the model deeper.
- The Benefit: Because you started small and grew, you didn't have to build the whole massive factory from scratch. It saves a lot of electricity (training compute) while still ending up with a deep factory.

2. The Big Discovery: They Are Cousins

For a long time, scientists thought these two methods were just different tricks. But this paper says: "Wait a minute, they are doing the exact same thing internally."

When the researchers looked inside the "brain" of both models, they found a shared signature:

The "Late Bloomers": In both models, the most important thinking happens at the very end of the line. The early stations just do basic prep work; the final stations do the heavy lifting.
The "Rhythm": Both models develop a repeating pattern. Just like a song with a chorus that repeats, these models have a specific rhythm where they refine their answer in cycles.

The Analogy:
Think of solving a riddle.

The Looped model is like a detective who reads the clues, thinks, reads them again, thinks harder, and reads them a third time.
The Grown model is like a detective who starts with a small notebook, then magically adds more pages to the middle of the notebook as they work, allowing them to write down more thoughts in the middle of the process.
The Paper's Conclusion: Even though they look different on the outside, both detectives end up using the same "thinking rhythm" to solve the riddle. They are both just iterative (step-by-step) thinkers.

3. The Superpower: Mixing and Matching

The most exciting part of the paper is that because these two methods are so similar, you can mix them to get the best of both worlds.

"Grow First, Loop Later":
The researchers took a model that was "grown" (expanded during training) and, after it was finished training, they told it to loop (repeat) its middle section during the actual test.
- The Result: The model got twice as smart at solving logic puzzles, even though it was never trained to do this! It's like teaching a student to study a chapter, and then telling them, "Hey, on the test, read that same chapter one more time before you write the answer."

4. Why Does This Matter?

This is a game-changer for efficiency:

Cheaper Training: "Growing" models are cheaper to train because you don't need to build the whole big model from day one.
Smarter Reasoning: "Looping" makes models better at logic without needing more memory.
The Best of Both: By growing a model first and then letting it loop a specific section during the test, you get a model that is highly efficient to train but incredibly powerful at reasoning.

Summary in One Sentence

This paper proves that making a model think by repeating its steps (looping) or by growing its steps (depth-growing) creates the same powerful "thinking rhythm," and combining these tricks allows us to build smarter AI that is cheaper to train and better at solving hard problems.

1. Problem Statement

Large Language Models (LLMs) typically improve reasoning capabilities by scaling parameters and data. However, reasoning often does not scale linearly with parameter count alone. While techniques like Chain-of-Thought (CoT) generate longer textual traces, they increase inference compute via token generation rather than enhancing internal latent computation.

Two distinct architectural approaches have emerged to induce iterative computation (refining representations over multiple steps within the model's forward pass):

Looping (Universal Transformers): Reusing a block of layers across depth by tying weights, allowing deep computation without increasing unique parameters.
Depth Growing: Training shallow-to-deep models by duplicating middle layers during training, which reduces pre-training compute costs.

The Core Question: Do the reasoning gains observed in both looped and depth-grown models stem from the same underlying mechanism, or are they superficially similar but mechanistically distinct? Furthermore, can these techniques be combined to create more robust and scalable reasoning models?

2. Methodology

The authors conduct a comprehensive empirical and mechanistic analysis comparing Standard, Looped, and Depth-Grown transformers at two scales (360M and 1.7B parameters).

A. Model Architectures

Standard Baseline: Standard Transformer with untied weights across all layers.
Looped Models:
- Fully Looped: A block of $L$ unique layers is repeated $k$ times (e.g., Loop(4×6) = 4 unique layers, 24 effective depth).
- Hybrid Looped: Unique encoder/decoder blocks with a middle recurrent block (e.g., Loop(4-4×4-4)).
Depth-Grown Models:
- MIDAS: Duplicates the middle block during training.
- LIDAS: Duplicates the exact layer-wise middle (symmetric growth), shown to be superior.
- Training Protocol: Models start shallow and grow depth via duplication, reducing total training FLOPs compared to training a full-depth model from scratch.

B. Evaluation Benchmarks

The study evaluates performance across 22 tasks, with a specific focus on Reasoning Primitives (e.g., variable assignment, copying sequences), Math Word Problems, Open/Closed-book Q&A, and Language Modeling (Lambada, HellaSwag).

C. Mechanistic Diagnostics

To understand how these models compute, the authors employ:

Depth-Utilization Diagnostics: Depth scores, top-5 vocabulary overlap, and Tuned Lens early-exit accuracy to measure reliance on later layers.
Residual Stream Analysis: Measuring L2 norms and sublayer contributions (Attention vs. MLP) to detect periodic patterns.
Intervention Experiments:
- Layer Swapping: Testing robustness to reordering layers.
- Local Effect Removal: Removing a layer's contribution to subsequent layers to measure dependency.
Inference-Time Scaling: Repeating middle blocks at inference time without retraining.

3. Key Contributions

1. Empirical Trade-offs

Looping: Improves reasoning under fixed parameter budgets (iso-param) and is competitive under fixed inference budgets (iso-inference). It trades unique parameters for sequential compute.
Depth Growing: Achieves similar or better reasoning performance with significantly less training compute (approx. 80% of baseline pre-training FLOPs) while maintaining broad capabilities.

2. Mechanistic Unification

The paper provides evidence that looping and growing share a convergent depth-wise signature:

Late-Layer Reliance: Both approaches shift indispensable computation to later layers, unlike standard models where early layers often plateau.
Periodic Patterns: Both exhibit recurring patterns in residual stream updates and sublayer contributions aligned with the block size (e.g., a 4-layer cycle). This suggests both induce a form of iterative refinement within the latent space.

3. Robustness and Design Principles

Order Sensitivity: Fully looped models are highly sensitive to layer order (swapping layers degrades performance significantly).
Hybrid Robustness: Depth-grown models and hybrid looped models (with unique encoder/decoder layers) are more robust to layer swapping. This suggests that unique outer layers stabilize the network, while the middle acts as a flexible iterative core.

4. Adaptability and Composability

Better Adaptation: Both looped and grown models adapt more efficiently to In-Context Learning (ICL) and Supervised Fine-Tuning (SFT) than standard baselines.
"Grow First, Loop Later": A novel finding is that depth-grown models (trained without weight tying) can be retrofitted to loop at inference time. Repeating a middle block at inference improves reasoning accuracy by up to 2× on primitives, despite never being trained to loop.
Cooldown Synergy: Depth-grown models achieve the largest gains when exposed to high-quality, math-heavy data mixtures during the "cooldown" (final training stage). Combining this with retrofitted looping yields the strongest reasoning performance.

4. Key Results

Reasoning Performance: On Reasoning Primitives, looped models (e.g., Loop(4-4×4-4)) and depth-grown models (LIDAS) significantly outperform standard baselines of equivalent size.
Inference Scaling: Repeating a 4-layer middle block in a grown model at inference time boosts accuracy on tasks like "Copying Real Words" and "Variable Assignment" by up to 2×. Standard models show little to no benefit from this.
Fine-Tuning: With only 64 fine-tuning examples, LIDAS and hybrid looped models surpass chance on variable assignment tasks, whereas baselines remain near random even with 128 examples.
Math Cooldown: Using high-quality math data (Nemotron-CC-Math-4+) during cooldown significantly boosts reasoning. LIDAS + Math Cooldown + Retrofitted Looping achieves the highest scores on GSM8K and Math Word Problems.

5. Significance and Conclusion

This paper establishes a unified theoretical framework for iterative computation in LLMs, demonstrating that depth growing and looping are complementary methods that induce similar internal mechanisms (iterative refinement via depth-periodic patterns).

Practical Implications:

Efficiency: Depth growing offers a path to high-reasoning models with reduced pre-training costs.
Scalability: Looping allows for test-time scaling (increasing depth at inference) without retraining, particularly effective when applied to the middle blocks of grown models.
Recipe for Reasoning: The authors propose a "Grow First, Loop Later" strategy:
- Train a model using depth growth (e.g., LIDAS) to reduce compute.
- Apply high-quality math data during cooldown.
- Retrofit recurrence (looping) to the middle block during the final stages or at inference.

This approach yields the strongest reasoning performance under matched data and inference constraints, suggesting that inducing iterative computation is a critical factor for scaling reasoning capabilities in LLMs, independent of the specific architectural implementation (growing vs. looping).