Attention Sinks and Outliers in Attention Residuals

Technical Summary: OASIS – Outlier- and Sink-Aware Routing for AttnResidual Architectures

Problem Statement
The paper addresses critical stability and robustness issues arising in AttnResidual architectures, a recent design that replaces standard single-level Softmax normalization with a dual-normalization scheme governing both token-level and depth-level attention routing. While AttnResidual enhances representational expressiveness, the authors identify a structural failure mode: the dual-normalization design exacerbates attention sinks and activation outliers.

In standard Transformers, attention sinks (e.g., the <|begin_of_text|> token) accumulate probability mass to stabilize representations. However, in AttnResidual, the additional simplex constraint on depth routing forces "no-op" (identity) updates to be expressed through real tokens and real depth branches rather than an explicit null channel. This constraint necessitates extreme pre-Softmax logits to approximate identity updates, leading to:

1. Intensified Attention Sinks: Deepening concentration on early tokens.
2. Amplified Activation Outliers: Hidden states exhibit heavy-tailed distributions (high kurtosis) and large infinity norms.
3. Quantization Brittleness: These outliers degrade inference stability and make low-bit quantization (e.g., W4A4, W8A8) significantly less robust.
4. Depth Collapse: Routing weights collapse toward a single dominant branch, effectively reverting the architecture to a vanilla-like residual behavior.

Methodology: OASIS
The authors propose OASIS (Outlier- and Sink-Aware routing via Inter-layer Null Signaling), a lightweight intervention that modifies the normalization layer without altering the training objective or the core architectural skeleton. OASIS operates on two fronts:

1. Null-Aware Routing (Softmax1):
   The method replaces standard Softmax operators at both the token and depth levels with Softmax1. This function introduces an explicit null state (denoted as $\emptyset$) into the normalization denominator.

   • Token Level: Allows attention mass to be routed to a "null" state rather than forcing it onto real tokens.
   • Depth Level: Allows depth routing weights to absorb "no-op" pressure via a null layer, preventing the concentration of mass on specific depth branches.
   • Mechanism: By decoupling non-informative mass from the competition among real tokens/branches, Softmax1 reduces the need for extreme logit amplification.

2. Token-to-Depth Null Coupling:
   Recognizing that token-level and depth-level routing remain decoupled in standard Softmax1, OASIS introduces a coupling mechanism. It aggregates token-level null evidence (the probability mass routed to the null state across attention heads) into a scalar statistic for each source branch. This statistic is then injected into the depth-routing logits.

   • Effect: Branches exhibiting high "no-op" behavior (high null mass) are downweighted in the depth routing, while informative branches are upweighted. This prevents the depth router from assigning significant weight to uninformative layers.

Key Contributions

• Identification of Structural Failure: The paper formally identifies that the dual-normalization design of AttnResidual structurally amplifies outliers and sinks because it lacks an explicit null pathway for near-identity updates.
• Theoretical Analysis: The authors provide theoretical proofs (Lemmas 5.1, 5.2; Theorems 5.1, 5.2) demonstrating that:
  • Without a null channel, near-no-op constraints force attention mass onto low-contribution tokens, creating outliers.
  • Dual normalization leads to depth collapse where routing concentrates on minimal-update branches.
  • Softmax1 introduces a feasible null route that decouples no-op behavior from real-token allocation, theoretically reducing structural pressure.
• OASIS Framework: A practical, minimal modification (replacing Softmax with Softmax1 and adding a coupling term) that mitigates these pathologies.

Experimental Results
The authors evaluated OASIS on Llama-3.2-1B and Qwen3-0.6B backbones across three real-world datasets (BookCorpus, Wiki40B, WikiText-2, GSM8K).

• Outlier Suppression: Relative to the Vanilla AttnResidual baseline, OASIS achieved an average reduction of 84.14% in maximum infinity norm and 96.77% in average kurtosis.
• Quantization Robustness:
  • Under W8A8 quantization, OASIS lowered perplexity by 73.55% compared to Vanilla.
  • Under W4A4 quantization, OASIS improved GSM8K Pass@1 accuracy by 23.47%.
• Attention Sink Mitigation: Qualitative analysis of attention maps showed that OASIS effectively redistributed attention away from the dominant first-token sink, yielding smoother, more local causal attention patterns compared to both Vanilla and intermediate baselines (like AoS or OutEffHop).
• Comparison: OASIS consistently outperformed baselines including Clip Attention, Gated Attention, OutEffHop, and sparse normalizers (Sparsemax, Entmax15) in both outlier metrics and downstream task performance under aggressive quantization.

Significance and Claims
The paper claims that OASIS provides a principled solution to the robustness limitations of dual-normalized architectures. By introducing explicit null capacity, the method stabilizes residual routing, allowing models to handle aggressive compression (low-bit quantization) and long-context inference without the degradation typically caused by attention sinks and activation outliers. The authors emphasize that these improvements are achieved without changing the training objective, suggesting that explicit null routing is a fundamental mechanism for stabilizing complex residual architectures.

Limitations
The authors note that their evaluation is currently limited to specific backbone models (1B and 0.6B parameters) and that broader validation on larger models is required. They also caution that excessive null allocation could potentially suppress useful weak signals or alter the interpretability of attention maps, suggesting future work on adaptive control of the null strength.