Technical Summary: Hidden Decoding at Scale

Problem Statement

Scaling Large Language Models (LLMs) has traditionally relied on enlarging the Transformer backbone, which requires costly rounds of pretraining and increases serving costs. For already-strong foundation models, further backbone expansion is often impractical. While recent work suggests that allocating more computation per token (even with fixed parameters) can improve reasoning and accuracy, existing approaches face significant scalability hurdles:

• Depth-Recurrent (Looped) Transformers: These reuse the same Transformer blocks multiple times per token to increase computation. However, they are difficult to scale to frontier models (e.g., 100B+ parameters) because the iterative nature of looped computation breaks the pipeline parallelism assumptions used to train the largest models, leading to GPU idle time.
• Sequence-Length Expansion: Simply feeding longer inputs is compatible with standard training pipelines but requires a method to expand computation without merely increasing the context window for the same token.

The paper addresses the challenge of improving an existing Transformer backbone by increasing per-token computation without altering the backbone's depth or width, while maintaining compatibility with pipeline parallelism and large-scale training infrastructure.

Methodology: Hidden Decoding

The authors propose Hidden Decoding, a sequence-length scaling method applied during Continued Pretraining (CPT). The core mechanism expands each input token into $n$ parallel streams processed within a single forward pass.

1. Multi-Stream Token Expansion

• Embedding Expansion: Instead of a single vocabulary embedding table, the model utilizes $n$ independent embedding tables ($E_1, \dots, E_n$). An input token $x_i$ is expanded into a sequence of $n$ representations: $(E_1(x_i), E_2(x_i), \dots, E_n(x_i))$.
• Sequence Construction: These representations are interleaved to form an expanded sequence of length $nL$ (where $L$ is the original sequence length).
• Training Objective: Only the final stream ($E_n$) is supervised. The model predicts the next token $x_{i+1}$ based on the hidden state of the final stream position. The first $n-1$ streams act as "latent computation states" with no direct loss, allowing them to refine the representation internally before the final prediction.

2. Stream-Factorized Attention

To prevent the attention cost from scaling quadratically ($O(n^2 L^2)$) with the expanded sequence length, the authors introduce Stream-Factorized Attention:

• Intra-Stream Layers: Most layers restrict attention to positions within the same stream. This reduces the complexity to roughly $O(nL^2)$.
• Cross-Stream Layers: A small subset of layers (e.g., full attention or sliding-window layers from the base model) allows mixing between streams.
• Efficiency: This design keeps the added attention cost near-linear in $n$ rather than quadratic, making the method trainable for models with over 100B parameters.

3. Progressive Expansion and Initialization

To manage the training of large expansion factors, the authors employ Progressive Expansion (e.g., $n=1 \to 2 \to 4 \to 8$). They use Cyclic Replication Initialization, where new embedding tables are initialized by copying existing ones (e.g., $E_{n+k} \leftarrow E_k$). This allows the model to reproduce the behavior of the smaller configuration initially and then adapt smoothly.

4. Architectural Optimizations (WeLM Specific)

For the WeLM backbone, the authors leverage a KV-Mirror design. Since intermediate streams carry no loss and only contribute context via Keys and Values (KV), the mirrored layers (which compute KV from earlier layers) can skip processing intermediate streams entirely. This further reduces training time by approximately 20% in their 80B model setup.

Key Contributions

1. Frontier-Scale Fixed-Backbone Scaling: The paper demonstrates the first successful application of sequence-length scaling at the 100B+ MoE scale. They trained WeLM-HD4-80B and WeLM-HD4-617B (with $n=4$), achieving performance improvements over their non-HD counterparts without enlarging the Transformer backbone.
2. Efficient Sequence-Length Expansion: The introduction of Stream-Factorized Attention enables training with expanded sequences at a cost that is near-linear in $n$ (e.g., a 4x expansion costs ~4.4x–5.1x per batch, far below the 16x cost of dense attention).
3. Expansion-Factor Scaling: Experiments show that increasing the expansion factor $n$ (from 2 to 8) consistently improves language modeling loss and downstream accuracy, validating $n$ as a practical scaling knob for fixed backbones.

Experimental Results

The authors evaluated Hidden Decoding on frontier-scale MoE models and dense baselines:

• Frontier-Scale Performance (WeLM-HD4-80B & 617B):

  • WeLM-HD4-80B outperformed WeLM-80B on all nine shared hard benchmarks, with significant gains in SciCode (+4.2), PHYBench (+4.0), and FrontierMath (+3.2).
  • WeLM-HD4-617B improved upon WeLM-617B across all nine benchmarks, including GPQA Diamond (+2.1), HLE (+1.8), and FrontierMath (+2.0).
  • These models were trained using an early SFT-only post-training recipe to ensure a controlled comparison, isolating the effect of Hidden Decoding.

• Scaling with Expansion Factor ($n$):

  • On the dense Qwen3-8B-Base, increasing $n$ from 1 to 8 resulted in monotonic improvements in average accuracy (e.g., MMLU improved from 78.2 to 81.6).
  • On the 80B MoE, increasing $n$ from 1 to 8 raised MMLU from 85.0 to 87.5 and lowered Pile-test BPB from 0.386 to 0.378.

• Ablation Studies:

  • Supervision Design: Supervising only the final stream yielded better results than applying loss to all streams or summing stream outputs.
  • KV Retention: Retaining separate KV caches for each stream (as opposed to sharing them) improved accuracy, confirming that intermediate streams provide distinct, useful context.
  • Attention Composition: Models with only a few full cross-stream layers (e.g., 4 out of 27) performed nearly as well as models with full cross-stream mixing in all layers, validating the efficiency of Stream-Factorized Attention.

• Inference Throughput:

  • Hidden Decoding maintains high throughput in latency-sensitive, small-batch scenarios (retaining 83–88% of baseline throughput at batch size 1).
  • Throughput decreases in long-context, high-batch scenarios (dropping to ~27% at batch size 32 with 32k input), reflecting the increased per-token computation.

Significance and Claims

The paper claims that Hidden Decoding provides a practical, engineering-realistic path for scaling frontier LLMs when further backbone expansion is too costly.

• Compatibility: Unlike looped Transformers, Hidden Decoding fits naturally with pipeline parallelism and standard large-model training stacks because the extra computation is manifested as a longer sequence rather than iterative depth.
• Effectiveness: It is the first demonstrated method to successfully scale sequence-length expansion to the 100B+ MoE regime, yielding measurable improvements in reasoning and knowledge benchmarks.
• Scalability: The method proves that the expansion factor $n$ is a viable scaling axis, offering performance gains that grow with $n$ without requiring new model architectures or massive retraining of the backbone.

The authors position this work as a solution to the "scaling wall" faced by strong foundation models, offering a way to extract more capability from existing parameters through efficient latent computation.