Thinking Deeper, Not Longer: Depth-Recurrent Transformers for Compositional Generalization

1. Problem Statement

Current Large Language Models (LLMs) rely on Horizontal Chain-of-Thought (HCoT), where reasoning is externalized by generating a sequence of tokens. This paradigm faces three fundamental limitations:

• Context Window Exhaustion: Each reasoning step consumes tokens, rapidly depleting the finite context window.
• Fixed Computational Depth: The depth of processing is fixed by the number of Transformer layers (e.g., 32 layers), regardless of task complexity. A simple task receives the same processing depth as a complex recursive one.
• Error Propagation: Intermediate reasoning steps are generated in natural language, making them susceptible to hallucinations and compounding logical errors.

The paper argues that to solve tasks requiring variable-depth reasoning (e.g., multi-hop graph traversal, nested logic), models need Vertical Chain-of-Thought (VCoT): increasing computational depth in the latent space without generating additional tokens.

2. Methodology: Depth-Recurrent Transformers

The authors propose a Depth-Recurrent Transformer architecture that decouples computational depth from parameter count and context length. The model iteratively applies a single shared-weight Transformer block in latent space for $T$ steps.

Core Architecture Components

The architecture consists of a task-specific Perception Interface (encoding input) and a task-invariant Reasoning Core (iterative refinement).

A. The Invariant Reasoning Core (Stability Mechanisms)
To enable stable unrolling for 20+ steps, three specific mechanisms are introduced:

1. Silent Thinking Objective:
   • Mechanism: Supervision (Cross-Entropy loss) is applied only at the final recurrence step ($T$). No intermediate auxiliary losses are used.
   • Purpose: Prevents the model from learning "heuristic shortcuts" (statistical guesses) to satisfy per-step supervision. It forces the network to develop genuine multi-step reasoning paths.
2. LayerScale Initialization:
   • Mechanism: Learnable per-channel scaling vectors ($\Gamma$) are initialized to $10^{-4}$ and applied after attention and feed-forward sub-layers.
   • Purpose: During early training, this forces the network to act almost as an identity mapping, protecting fragile hidden states from the noise of randomly initialized weights. As training progresses, the scale increases to activate reasoning capacity.
3. Identity-Biased Gated Recurrence:
   • Mechanism: A GRU-like gate combines the new candidate state $\tilde{H}^{(t)}$ with the previous state $H^{(t-1)}$. The gate bias ($b_z$) is initialized to $-2.0$.
   • Purpose: Since $\sigma(-2.0) \approx 0.12$, the model defaults to retaining $\approx 88\%$ of the previous state. This creates a "gradient highway," ensuring signal propagation remains stable over deep temporal unrolling and preventing vanishing gradients.

B. Task-Specific Perception Interfaces
The paper tests three interfaces with decreasing inductive biases:

1. Topological Masking (Graph): Uses an adjacency mask in self-attention to strictly enforce message passing along graph edges.
2. Relative Positional Encoding (Logic): Uses Rotary Position Embeddings (RoPE) to handle hierarchical structures based on relative distance.
3. Standard Sequence Attention (Unstructured Text): Removes structural priors; the input facts are shuffled, forcing the core to autonomously discover latent routing paths without structural hints.

3. Key Contributions

• Architectural Innovation: A depth-recurrent Transformer that achieves stable recurrence over 20+ steps with fewer than 1M parameters, decoupling depth from parameter count.
• Silent Thinking Paradigm: Demonstrates that intermediate supervision is detrimental to algorithmic reasoning, causing models to collapse into heuristics. Final-step-only supervision is essential for OOD generalization.
• Computational Frontier: Identifies a distinct boundary in the accuracy heatmap (Thinking Steps $\times$ Task Complexity) where performance transitions sharply from chance to near-perfect as compute depth scales with task difficulty.
• Generalization Profiles: Reveals that the same reasoning core exhibits qualitatively different behaviors based on the perception interface:
  • Graph: Precise but brittle (strict 1-step = 1-hop constraint).
  • Logic: Approximate but robust (graceful degradation).
  • Text: Autonomous latent routing (discovering pointers in unstructured data).

4. Experimental Results

The model was evaluated on three compositional reasoning tasks with Out-of-Distribution (OOD) testing:

Ablation Study (Intermediate Supervision):
When intermediate supervision was applied to the graph task, the model achieved ~73% accuracy on 12-hop paths in just 1 step (impossible under strict topology). This confirmed the model learned statistical shortcuts (e.g., guessing based on node degree) rather than true reasoning, leading to a complete collapse in OOD generalization.

5. Significance and Future Work

• Mechanistic Insight: The paper provides a mechanistic view of "vertical" reasoning, contrasting it with the prevailing "horizontal" token generation. It suggests that for algorithmic tasks, intermediate states are latent computations, not semantic features, and thus should not be supervised.
• Efficiency: Offers a path to scaling test-time compute without increasing context window usage or token generation costs.
• Limitations: Current models are small (<1M params), perception interfaces are manually designed, and theoretical generalization bounds are not formally proven.
• Future Direction: Integrating this depth-recurrent core with pre-trained LLMs to combine the reasoning capabilities of VCoT with the linguistic fluency of standard Transformers.

In summary, this work proposes that "thinking deeper" (recurrent latent iteration) is a more robust and efficient paradigm for compositional generalization than "thinking longer" (generating more tokens), provided the architecture is stabilized against gradient issues and trained without intermediate supervision.