Technical Summary: Delta Attention Residuals

Problem Statement

Standard residual connections in deep transformers accumulate all preceding sublayer outputs via fixed additive coefficients ($h_l = h_{l-1} + f(h_{l-1})$). While this provides gradient highways, it lacks a mechanism to selectively aggregate information across depth. Attention Residuals (Kimi, 2025) proposed replacing fixed aggregation with learned softmax attention over previous layer outputs to enable selective cross-layer routing.

However, this paper identifies a critical failure mode in standard Attention Residuals: routing collapse due to source redundancy. In Attention Residuals, the source representations ($s_i$) are cumulative hidden states ($h_i = \sum v_j$). As depth increases, adjacent cumulative states share a large common prefix, causing their pairwise similarity to grow. This redundancy reduces the contrast in routing candidates, forcing the softmax attention weights toward a near-uniform distribution. Empirically, the maximum routing weight in deep layers degrades to approximately 0.2, rendering the mechanism ineffective at selecting informative states and effectively averaging over all sources.

Methodology: Delta Attention Residuals

To address source redundancy, the authors propose Delta Attention Residuals, which route over deltas (the change introduced by each sublayer) rather than cumulative states.

Core Mechanism

Instead of routing over $s_i = h_i$, the method routes over $v_i = h_{i+1} - h_i$, representing the specific contribution of a sublayer. This is implemented via an additive routing formulation:
$$\hat{h}_l = \tilde{h}_l + \sum_i \alpha_{i \to l} \cdot v_i$$
Where:

• $\tilde{h}_l$ is the current residual stream (preserved by default).
• $v_i$ are the delta sources (per-sublayer outputs).
• $\alpha_{i \to l}$ are learned attention weights computed via softmax over normalized deltas.

Granularity Variants

The principle is applied at two granularities:

1. Delta AttnRes: Routes over every individual sublayer output (Attention and MLP), providing $2L$ sources for $L$ layers.
2. Delta Block: Aggregates consecutive sublayer outputs within a block into a single block delta ($\Delta_b$), reducing the number of sources to $\approx L/B$ for efficiency.

Key Design Advantages

1. Structural Diversity: Unlike cumulative states which converge to near-linear relationships, deltas are structurally diverse because attention and MLP outputs occupy different subspaces and capture different abstraction levels. This maintains high routing contrast (max weight $\approx 0.6$).
2. Residual Preservation: The additive formulation preserves the residual stream, ensuring no information is lost at block boundaries (unlike replacement routing which discards the stream).
3. Safe Initialization: With zero-initialized query weights, the routing weights become uniform, and the routing output becomes a bounded perturbation. This reduces the mechanism to an identity map at initialization, enabling the fine-tuning of pretrained models without disrupting the learned residual stream.

Key Contributions

1. Identification of Routing Collapse: The paper demonstrates that source redundancy in cumulative-state Attention Residuals causes routing sharpness to degrade to $\sim 0.2$ in deep layers, leading to near-uniform attention distributions.
2. Delta Attention Residuals: A new architecture that routes over deltas instead of cumulative states. This maintains sharp routing ($\sim 0.6$) and consistently outperforms both standard residuals and Attention Residuals across scales from 220M to 7.6B parameters.
3. Fine-Tuning Compatibility: The additive formulation allows existing pretrained transformers to be converted into Delta Attention Residuals via standard fine-tuning, overcoming the "loss landscape valley" issue where models fail to utilize cross-layer weights during fine-tuning.

Experimental Results

The authors evaluated the method on Qwen3-architecture models trained from scratch on FineWeb-Edu and fine-tuned on downstream tasks.

• From-Scratch Training:

  • Performance: Delta methods consistently outperformed baselines. At the 1044M scale, Delta AttnRes achieved a validation perplexity of 29.13 (vs. 29.70 for baseline, -1.9% improvement). At the 7.6B scale, Delta Block achieved 16.00 PPL (vs. 17.43 for baseline, -8.2% improvement).
  • Failure of Alternatives: Standard Attention Residuals (AttnRes) degraded significantly at larger scales (e.g., +6.9% worse than baseline at 1044M) due to the compounding information loss from replacement routing and periodic resets.
  • Efficiency: While per-sublayer Delta AttnRes offers the best perplexity, it incurs high memory overhead. Delta Block (block-level grouping) offers a practical default, matching the quality of Delta AttnRes with significantly lower memory and throughput costs (e.g., 26% memory overhead vs. 3.5x for full Delta AttnRes at 1044M).

• Fine-Tuning:

  • When fine-tuning a pretrained Qwen3-0.6B model, Delta Block achieved 55.6% average accuracy on 8 downstream benchmarks, outperforming the standard baseline (55.0%) and AttnRes (54.1%).
  • Crucially, Delta Block avoided the large loss spike observed in AttnRes at initialization, converging smoothly due to its safe initialization properties.

• Routing Analysis:

  • Visualizations confirm that Delta Block maintains sharp routing (max weight $\sim 0.6$–$1.0$) throughout the depth, whereas AttnRes collapses to $\sim 0.2$ in deep layers.
  • Deep layers in Delta models selectively concentrate weights on specific early outputs (including token embeddings), whereas AttnRes distributes attention diffusely.

Significance and Claims

The paper claims that Delta Attention Residuals solve the fundamental issue of source redundancy in cross-layer routing. By routing over "what changed" (deltas) rather than "what accumulated" (cumulative states), the method enables selective, high-contrast routing that scales effectively to deep networks.

The authors emphasize that this approach is orthogonal to other architectural improvements. Its primary significance lies in:

1. Scalability: It enables deeper, more effective cross-layer connections without the degradation seen in previous attention-based residual methods.
2. Practicality: The additive formulation and safe initialization allow for the seamless integration of cross-layer routing into existing pretrained checkpoints via fine-tuning, a capability previously hindered by initialization instability.
3. Efficiency: The Delta Block variant provides a high-performance, low-overhead solution suitable for large-scale training, balancing routing expressivity with computational constraints.