Technical Summary: Dual Attention Residuals (DAR)

Problem Statement

Recent advancements in Transformer architectures have extended residual pathways along two complementary axes: historical retrieval and multi-stream residuals.

• Historical Retrieval: Methods like Transparent Attention, DenseFormer, and Attention Residuals (AttnRes) allow later layers to access representations from earlier depths. In standard single-stream retrieval, the representation determining the depth weights (the "address") and the content being retrieved (the "value") originate from the same residual trajectory. This creates an "address-content coupling" where the selection mechanism is constrained by the single stream's state.
• Multi-Stream Residuals: Methods like Hyper-Connections and mHC expand the pathway into multiple learned trajectories (streams) to transport and mix information.

While these approaches have been studied in isolation, a straightforward combination—equipping each stream with an independent historical retriever—fails to leverage the interaction between streams. In such a setup, each trajectory determines its own depth selection without influence from the other, preventing the complementary signals of one stream from guiding the retrieval of the other. The core problem addressed is whether these two degrees of freedom (depth selection and trajectory maintenance) can interact productively to improve representation diversity and retrieval efficiency.

Methodology: Dual Attention Residuals (DAR)

The authors propose Dual Attention Residuals (DAR), a mechanism that integrates multi-stream interaction into depth-wise historical retrieval through reciprocal cross-stream addressing.

Core Mechanism

DAR modifies the residual pathway around each Transformer branch without altering the branch's internal computation (attention or feed-forward). For a system with two streams ($j \in \{0, 1\}$):

1. Reciprocal Key-Value Routing: For a target stream $j$, the depth weights are computed using keys derived from the opposite stream ($1-j$), while the values are retrieved from the target stream's own history ($j$).
   • $k^j_r = \text{Norm}(H^{1-j}_r)$ (Key from opposite stream)
   • $v^j_r = H^j_r$ (Value from self-stream)
   • This allows the opposite stream to provide a complementary signal for selecting historical depths, even if the target stream's own representation would not strongly match the query.
2. State Mixing and Gated Writes: The retrieved states from both streams are mixed to form the input for the unchanged Transformer branch. The branch's output is then used to update both residual trajectories via gated writes and constrained mixing.
   • A gating mechanism ($\alpha$) mixes the retrieved states.
   • A transport mechanism ($\rho$) and a write gate ($\beta$) update the partial block state, ensuring stable transport of existing partial states within a block.

Variants

• Full DAR: Operates at the layer granularity ($K=1$), where every Transformer layer contributes a completed entry to the retrievable history.
• Block DAR: Partitions layers into blocks of size $K$. It retrieves from completed block histories and optionally includes the current partial block state as a candidate. This reduces overhead by operating on coarser depth granularities.

Design Comparisons

The paper evaluates several routing variants to isolate the contribution of cross-stream interaction:

• DAR-SelfKV: Keys and values both come from the same stream (standard single-stream retrieval per stream).
• DAR-CrossV: Keys from self-stream, values from opposite stream.
• DAR-FixedKV: Fixed assignment (Stream 0 keys, Stream 1 values for both).
• DAR (Proposed): Reciprocal cross-stream keys with self-stream values.

Key Contributions

1. Interaction of Trajectories: The paper demonstrates that independently evolving residual trajectories can serve not only as parallel information pathways but also as complementary signals for selecting historical depths.
2. DAR Architecture: The proposal of a mechanism that realizes this interaction through reciprocal cross-stream keys and self-stream values while leaving the Transformer branches unchanged.
3. Empirical Validation: Comprehensive experiments across dense models (0.1B to 1B parameters) and a 7B sparse-MoE model, showing consistent improvements over standard residuals and Attention Residuals.
4. Analysis of Redundancy: Through representation geometry (CKA) and intervention analysis, the authors show that DAR preserves depth-wise diversity and avoids the functional imbalance or redundancy observed in alternative two-stream designs.

Experimental Results

The authors evaluated DAR under matched training budgets against standard residual connections, Attention Residuals (AttnRes), and mHC.

• Validation Loss: DAR consistently achieved the lowest validation loss.
  • In the 1B dense setting, DAR-Block outperformed both the residual baseline and AttnRes-Block.
  • In the 7B sparse-MoE setting, DAR achieved the lowest loss, with training curves remaining below baselines throughout training.
  • Across 0.1B, 0.3B, and 0.5B dense models, DAR variants reduced validation loss by 2.03% to 5.12% relative to the residual baseline.
• Efficiency: While DAR incurs a higher computational cost than standard residuals (approx. 17.5% throughput reduction in 7B MoE due to cross-stream retrieval and gating), it remains competitive with AttnRes. The memory overhead is primarily driven by the historical retrieval mechanism itself, with the cross-stream routing adding a smaller, measurable cost.
• Ablation Studies:
  • Routing: The gain is not explained by additional value projections (SepV) or simple cross-stream value aggregation. The specific reciprocal cross-stream key / self-stream value routing (DAR) yielded the best performance.
  • Redundancy: Analysis of Cross-Stream Linear CKA showed that DAR maintains low similarity between unmatched depths, indicating distinct trajectories. In contrast, variants like DAR-SelfKV and mHC-2 showed broad high-similarity regions, suggesting redundancy.
  • Intervention: "Rescue gain" analysis revealed that in DAR, retaining either stream recovers the clean output significantly better than retaining neither. In contrast, other variants (like DAR-SelfKV) showed strong asymmetry, where one stream became functionally dominant while the other was underutilized.

Significance and Claims

The paper claims that DAR successfully brings multi-stream flexibility into depth-wise retrieval. By allowing one trajectory to influence the depth selection of another, DAR enables the model to access historical contexts that might be missed by a single-stream retriever.

The authors emphasize that the performance gains are not merely due to the presence of an additional stream or value projection, but specifically due to the reciprocal cross-stream selection mechanism. This design preserves depth-wise diversity and avoids the functional imbalance where one stream dominates the other, a common issue in alternative two-stream designs. The work suggests that historical retrieval and multi-stream topologies are not mutually exclusive but can be synergistically combined to improve Transformer representation learning.