🤖 machine learning

A Unifying View of Attention Sinks: Two Algorithms, Two Solutions

This paper reveals that visually similar attention sinks in transformers actually represent two distinct mechanisms—adaptive "nop" suppression and global "broadcast" aggregation—necessitating a dual intervention strategy combining gating and registers to effectively address both.

Original authors: Lukas Fesser, Mozes Jacobs, Thomas Fel, Andy Keller, Sham Kakade

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lukas Fesser, Mozes Jacobs, Thomas Fel, Andy Keller, Sham Kakade

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a crowded room of people (the "tokens" in an AI model) trying to talk to each other. In a specific type of AI called a Vision Transformer, researchers noticed a strange pattern: no matter who is speaking, everyone seems to keep turning their heads to look at just one or two specific people in the room. In the AI's "attention map," this looks like a dark, vertical stripe.

For a long time, scientists thought this was a bug—a glitch where the AI got stuck staring at one spot. But this paper argues that this "staring" isn't a mistake. Instead, it's actually the AI using two completely different strategies that just happen to look the same from the outside.

Here is the breakdown of the two strategies, using simple analogies:

1. The "Do Nothing" Button (Adaptive NOP)

The Analogy: Imagine a teacher in a classroom. Sometimes, the teacher has nothing useful to add to the conversation. Instead of saying "um" or "maybe," the teacher points to a specific student (the "sink") and says, "Just look at Alex; he's not saying anything, so we'll just stay exactly as we are."

What's happening:

The Goal: The AI wants to stop changing its current thought. It needs to say, "I have no new information to add right now."
The Trick: Because the AI is forced to point at someone, it points at a "null" token (a token with almost zero information). By pointing at this empty token, the AI effectively hits a "Do Nothing" button.
The Clue: If you check the "energy" (value) of this special token, it is almost zero. It's a ghost token used only to cancel out updates.

2. The "Coffee Shop" (Broadcast)

The Analogy: Now imagine a different scenario. The teacher has a very important piece of news (like "It's raining outside") that everyone needs to know. Instead of whispering it to each student one by one, the teacher stands in the center of the room (the "sink") and shouts the news. Everyone turns to look at the teacher to hear the update.

What's happening:

The Goal: The AI wants to share a piece of global information with everyone in the room.
The Trick: The "sink" token acts as a hub. It gathers information and then "broadcasts" it to every other token. Everyone updates their state by adding this shared piece of information.
The Clue: If you check the "energy" of this special token, it is high and full of content. It's actively writing a message to the group.

Why This Matters

The paper's main discovery is that you can't tell the difference just by looking at the attention map. Both strategies look like a vertical stripe where everyone stares at one spot.

If you see a stripe, you don't know if the AI is hitting the "Do Nothing" button or shouting "Important News!"
The Problem: Previous attempts to "fix" these stripes (like adding special "register" tokens or "gating" mechanisms) were guessing.
- Some fixes assumed the AI was doing "Do Nothing" and tried to stop it.
- Others assumed the AI was "Broadcasting" and tried to help it.
- Because the researchers didn't know which one was happening, their fixes were often incomplete.

The Solution: A Two-Pronged Approach

The authors created a way to diagnose which strategy the AI is using:

Check the "Energy": If the token is empty (low energy), it's a "Do Nothing" button.
Check the "Structure": If the token is full of data and creates a simple, shared pattern, it's a "Broadcast."

The Result: When they applied this to real AI models (like those that recognize images), they found that both strategies are happening at the same time, but in different parts of the brain:

Early layers often use the "Do Nothing" strategy.
Deeper layers often use the "Broadcast" strategy.
Even the "Register" tokens (special tokens added to help the AI) were being used for both purposes.

The Fix: The paper shows that to truly fix the issues caused by these patterns, you need to use both tools together:

Use Gating to handle the "Do Nothing" tokens (letting them suppress updates cleanly).
Use Registers to handle the "Broadcast" tokens (giving them a dedicated space to share info).

When they combined these two methods, the AI became more stable and better at understanding the detailed spatial layout of images (like recognizing where objects are in a picture), even though its ability to just "name" the picture didn't change much.

Summary

The paper teaches us that just because an AI looks like it's staring at one spot, it doesn't mean it's broken. It might be taking a break (Do Nothing) or sharing a secret (Broadcast). To fix AI behavior, we first need to understand why it's staring, and then use the right tool for the job.

Technical Summary: A Unifying View of Attention Sinks

Problem Statement

Attention sinks—tokens that attract disproportionate attention mass across many queries, appearing as vertical stripes in attention maps—are ubiquitous in softmax-based Transformers, particularly Vision Transformers (ViTs) and Large Language Models (LLMs). While these patterns are well-documented, their underlying computational mechanisms remain ambiguous. The central problem addressed by this work is that visually identical sink patterns may reflect fundamentally different algorithms. Current mitigation strategies, such as gating mechanisms and register tokens, are often applied heuristically without a unified framework to distinguish why a sink exists. Consequently, interventions designed to "fix" sinks may inadvertently remove useful computations or fail to address the true source of instability.

Methodology

The authors propose a unifying framework that decomposes attention sinks into two distinct computational hypotheses, deriving specific, measurable diagnostics for each.

1. Two Hypotheses for Attention Sinks

The paper posits that sinks arise from one of two mechanisms:

Adaptive NOP (No-Operation): The sink acts as a suppression mechanism. When an attention head has no useful information to contribute for a specific token, it routes attention to a "null" token to effectively perform an identity operation on the residual stream.
- Signature: The sink token exhibits negligible value norms ( $\|v_s\| \approx 0$ ).
- Mechanism: To trigger this, the model learns a gating mechanism, evidenced by a dominant singular value in the query-key interaction matrix ( $W_Q W_K^\top$ ) or massive activation norms at the sink position.
Broadcast: The sink acts as a communication hub. It aggregates global information and redistributes it to other tokens.
- Signature: The sink token has non-negligible value norms and induces low-rank (approximate rank-1) attention outputs.
- Mechanism: The sink token writes a shared payload to the residual stream, increasing the similarity of token representations across the sequence.

2. Synthetic Validation

To isolate these mechanisms, the authors trained minimal synthetic models on two specific tasks:

Conditional Damping (NOP Task): A single-head transformer trained to suppress updates for tokens aligned with a specific vector. Results showed convergence to a sink with near-zero value norms and a spectral spike in the interaction matrix.
Global Broadcast Task: A two-layer transformer trained to distribute a specific token's information to all other positions. Results showed a modular solution where one layer acts as a broadcast hub (rank-1 update, high value norms) and the subsequent layer acts as an identity.

3. Empirical Diagnostics on Pretrained Models

The authors applied these diagnostics to pretrained ViTs (DINOv2, OpenCLIP) and models with register tokens. They analyzed:

Value Norm Ratios: Comparing the norm of the sink token's value vector to the average.
Stable Rank: Measuring the rank of the attention-induced update matrix.
Layer-wise Transitions: Tracking which tokens serve as sinks across different network depths.

Key Results

1. Coexistence of Mechanisms

Analysis of pretrained ViTs reveals that both mechanisms coexist within the same model but are specialized:

Layer Specialization: In early layers, the [CLS] token often serves as a sink. In deeper layers, this role transitions to specific patch tokens.
Head Specialization: Sink behavior is not uniform across all heads; specific heads act as sinks for nearly 80% of inputs, while adjacent heads do not.
Dual Regimes: When projected onto diagnostic axes, sinks separate into two distinct clusters:
- NOP Sinks: Found primarily in early-to-mid layers, characterized by suppressed value norms (< 0.2x baseline).
- Broadcast Sinks: Found primarily in deeper layers, characterized by standard value norms and strict rank-1 updates.

2. Reinterpretation of Register Tokens

The study investigates register tokens (learned tokens added to the input sequence), which were originally proposed to absorb attention mass for broadcasting.

Repurposing: Diagnostics reveal that register tokens are not exclusively used for broadcasting. A majority of register sinks exhibit the NOP signature (suppressed value norms), indicating the model repurposes them as "trash cans" to suppress updates, not just as "coffee cups" for global information.
Implication: Neither gating (targeting NOP) nor registers (targeting broadcast) alone suffices to address all sink behaviors.

3. Synergistic Mitigation

The authors tested the combination of Gated Attention (which allows heads to suppress outputs, addressing NOP) and Register Tokens (which provide a substrate for broadcasting).

Dense Prediction Gains: In LeJEPA-trained ViT-L models, combining gating and registers yielded the best performance on dense spatial prediction tasks (ADE20K and Pascal VOC 2012), outperforming baselines and single-mitigation variants.
Classification Stability: Global classification accuracy (ImageNet-1k) remained largely unchanged across variants, suggesting that sink mitigation primarily improves localized patch-level representations rather than global image understanding.
Mechanism Confirmation: The combined approach successfully suppressed NOP sinks (reducing token norms) while redirecting broadcast sinks to the register tokens.

Significance and Claims

The paper claims that the visual signature of an attention sink is computationally ambiguous and that effective intervention requires first identifying the underlying algorithm.

Unification: The work provides a unified framing that separates the "NOP" and "Broadcast" hypotheses, offering operational diagnostics (value norms and stable rank) to distinguish them.
Complementarity: It demonstrates that gating and register tokens are complementary because they target different computational regimes. Combining them yields stronger stability and performance in tasks requiring high-fidelity spatial representations.
Mechanistic Insight: The findings suggest that "artifacts" like attention sinks are often genuine computational primitives (either suppression or global coordination) rather than purely pathological failures.

The authors conclude that while their diagnostics and combined mitigation strategy improve dense prediction, the broader question of how to architect attention to support broadcasting without inducing pathological sinks remains an open area for future research.