Technical Summary: A Structural Theory of Position Bias in Transformers

Problem Statement

Transformer models exhibit systematic position biases, including primacy bias (preference for early tokens), recency bias (preference for recent tokens), and the "Lost-in-the-Middle" (LiM) phenomenon, where information in the center of long contexts is underutilized. While empirical evidence suggests these biases are architectural rather than semantic, the theoretical origins remain poorly understood.

A critical gap exists between existing theoretical analyses and practical Transformer behavior. Recent work by Wu et al. [2025] demonstrated that in attention-only causal Transformers, cumulative attention must collapse onto the first token as depth increases. However, this prediction contradicts empirical observations in modern, deep Transformers, which do not exhibit such collapse. The paper attributes this discrepancy to the omission of residual connections in prior theoretical frameworks, despite their central role in actual architectures.

Methodology

The authors develop a structural theory of position bias based on residual-aware cumulative attention rollout. The framework analyzes information propagation across layers independent of training objectives, data, or optimization dynamics.

Core Framework

1. Residual-Aware Transition: Unlike previous models that treat residuals as fixed convex combinations (e.g., $0.5I + 0.5A$), the authors model residual connections as discretizations of continuous-time dynamics. They define a layer- and input-dependent mixing coefficient $\lambda_t(x)$ representing the relative strength of the attention update versus the identity path:
   $$\lambda_t(x) := \frac{\| \text{Attn}(X^{(t-1)}) \|}{\| X^{(t-1)} \| + \| \text{Attn}(X^{(t-1)}) \|}$$
   The transition matrix is defined as $R^{(t)} = (1 - \lambda_t)I + \lambda_t A^{(t)}$, where $A^{(t)}$ is the dataset-averaged attention kernel.

2. Cumulative Rollout: Information propagation to depth $T$ is captured by the rollout matrix $P^{(T)} = R^{(T)} \cdots R^{(1)}$. The distribution of the last row, $p^{(T)}(j)$, quantifies the cumulative influence of input token $j$ on the final output.

3. Four Architectural Forces: The finite-depth analysis isolates four forces shaping the rollout distribution under a "stochastically monotone attention kernel" assumption:

   • Causal Masking: Induces a primacy drift (mass shifts toward earlier positions).
   • Residual Connections: Induce a recency drift (mass shifts toward recent positions) by preserving identity paths.
   • Positional Encodings: Biases favoring recent keys (e.g., ALiBi) compound to induce recency drift.
   • Content Contributions: Modeled as a constant background plus a diagonal self-attention term, these modulate the drift (positive diagonal content increases recency; negative increases primacy).

4. Infinite-Depth Analysis: The authors extend the analysis to the infinite-depth limit, establishing a dichotomy based on the summability of mixing coefficients $\sum \lambda_t$.

Key Contributions

1. Residual-Aware Attention Rollout

The paper introduces a formulation that explicitly models residual connections as persistent identity paths. This aligns the propagation model with modern Transformer architectures, moving beyond the restrictive "attention-only" assumption.

2. Structural Origin of Position Bias

The authors prove that Transformer architecture alone induces broad, often U-shaped influence profiles. The interaction between causal masking (primacy) and residual connections (recency) creates a structural prior where both early and recent tokens receive elevated influence relative to intermediate positions. This provides a theoretical explanation for the Lost-in-the-Middle phenomenon even in the absence of semantic content.

3. Resolution of the Attention Collapse Discrepancy

The paper resolves the mismatch between Wu et al. [2025] and empirical reality. It demonstrates that attention collapse is not inevitable under causal masking alone. Instead, collapse depends on the cumulative attention-mixing strength ($\sum \lambda_t$):

• Finite Total Mixing ($\sum \lambda_t < \infty$): Attention does not collapse to the first token; the diagonal entries of the rollout matrix remain strictly positive.
• Infinite Total Mixing ($\sum \lambda_t = \infty$): Collapse occurs, and attention mass concentrates on the first token.

4. Empirical Validation

The theory is validated against measured input-token influence in pretrained language models (BLOOM, MPT, Falcon).

• Controlled Rollout: Simulations show that "attention-only" rollout collapses to the first token, while "residual-aware" rollout produces the observed U-shaped profile.
• Quantitative Match: The predicted influence profiles from the residual-aware model closely match empirically estimated distributions (measured via gradient-based attribution), with significantly lower Wasserstein distances compared to attention-only models.

Results

• Finite Depth: The combination of causal masking and residual connections yields a U-shaped influence profile. This profile is robust across different models and datasets.
• Infinite Depth: The theoretical analysis proves that residual connections fundamentally alter deep attention dynamics. Collapse is conditional on the accumulation of mixing strength, not just the presence of causal masking.
• Empirical Alignment:
  • In models like bloom-176b and mpt-7b, the theoretical rollout (without content) reproduces the dominant U-shaped structure of the measured influence.
  • Adding a coarse content abstraction (constant + diagonal) further refines the fit, shifting mass toward recent positions and aligning closely with empirical data.
  • The "attention-only" variant fails to reproduce empirical profiles, confirming the necessity of modeling residuals.

Significance and Claims

The paper claims to provide an architecture-faithful framework for understanding position bias. Its primary significance lies in:

1. Bridging Theory and Practice: It resolves the open problem identified by Wu et al. [2025] by showing that residual connections prevent the inevitable collapse predicted by attention-only theories.
2. Explaining Lost-in-the-Middle: It establishes that the U-shaped influence profile (and thus the LiM effect) arises from a structural prior inherent to the Transformer architecture, rather than solely from semantic or training dynamics.
3. Predictive Power: The theory accurately predicts the shape of influence distributions in pretrained models using only architectural parameters (masking, residuals, positional encoding biases) and measured mixing coefficients, without requiring access to specific training data or token semantics.

The authors maintain a modest scope, characterizing their work as describing architectural priors in information propagation rather than end-to-end representation learning. They note that while the U-shaped prior plausibly underlies the Lost-in-the-Middle task-performance phenomenon, closing the loop on specific retrieval-style task degradation is left for future work.