📊 statistics

Which Question Is Your Attention Metric Answering? Attention Rows as Compositional Data

This paper argues that standard attention metrics often yield misleading conclusions because they fail to distinguish between a model's focus on a dominant "sink" token and its distribution over content tokens, proposing instead to treat attention rows as compositional data to accurately separate these factors and avoid artifacts in model analysis and pruning.

Original authors: Marios Papamichalis, Regina Ruane

Published 2026-08-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Marios Papamichalis, Regina Ruane

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Inside the vast neural networks that power modern artificial intelligence, there is a mechanism called attention that allows the system to decide which parts of a sentence or image are most important at any given moment. When this mechanism operates, it produces a list of probabilities for every word or token, showing how much focus is placed on each one. For years, researchers have studied these lists to understand how the machines think, using standard mathematical tools to compare them. They assumed that if two lists looked similar, the underlying thought processes were similar too. However, a new analysis reveals that this assumption rests on a hidden choice that scientists have been making without realizing it, a choice that can completely flip their conclusions about what the machine is actually doing.

The core of the issue lies in a peculiar behavior found in almost every large language model: a tendency to dump a massive amount of attention onto a single, uninformative token, usually the very first word of a sentence. Researchers call this an "attention sink." In many cases, this single token absorbs eighty percent or more of the total attention, leaving the remaining tokens to share the tiny remainder. When scientists want to compare two different attention patterns, they face a dilemma. Do they include this dominant first token in their comparison, or do they remove it to see how the rest of the information is distributed? The paper demonstrates that this is not a harmless technicality. Depending on which option a researcher chooses, the same two attention patterns can appear either nearly identical or completely different. This means that decades of previous research, which often relied on comparing these patterns without explicitly stating how they handled the first token, may have drawn conclusions based on an arbitrary convention rather than the actual structure of the model's thought process.

To solve this, the authors treated the attention lists as a specific type of data known as compositional data, where the only thing that matters is the relative proportion of the parts, not their absolute size. By applying a specialized mathematical framework designed for this kind of data, they were able to separate the "sink" question from the "content" question. They proved that the standard tools used by the community, such as cosine similarity and entropy, mix these two distinct questions together. When the attention sink grows larger, these standard tools report that the attention is becoming more focused and the system is collapsing into a simpler state. But the new analysis shows that this is often an illusion. The attention is not necessarily becoming more focused on the content; it is simply that the first token is hogging more of the spotlight. The actual distribution of attention among the meaningful words can remain completely unchanged while the standard metrics scream that a collapse has occurred.

The researchers tested this theory across ten different pre-trained models, ranging from small experimental networks to massive models with billions of parameters. They found that the choice of convention changes the verdict in a startling number of cases. In some models, nearly half of the comparisons between different parts of the network flipped from one conclusion to the opposite when the researchers switched from keeping the sink to dropping it. For instance, a pair of attention heads that appeared to be the closest neighbors in a network when the sink was included could become distant strangers once the sink was removed. Conversely, heads that looked unrelated with the sink included could appear to be identical twins without it. This instability meant that the famous "head clustering" maps, which scientists use to categorize what different parts of the brain are doing, were often capturing the size of the sink rather than the actual function of the parts. In one major re-analysis of a standard clustering pipeline, the most prominent structure found in the data vanished entirely once the sink was properly accounted for, revealing that the structure was an artifact of the measurement method, not a feature of the model.

The implications for training and improving these models are significant. The paper shows that much of what looks like "entropy collapse"—a sign that a model is becoming unstable or over-simplified during training—is actually just the attention sink growing larger. In models with one billion parameters, ninety-five percent of the apparent drop in complexity was due to the sink taking up more space, not the content becoming less diverse. This distinction matters because it changes how engineers should prune, or remove, unnecessary parts of the network. When the researchers used the new, sink-separated method to decide which parts of the network to cut, they found that using the old methods could make the model's performance crash, increasing its error rate by more than a hundred times in some cases. The new method allowed them to identify and remove redundant parts without breaking the model, but only if they first separated the noise of the sink from the signal of the content.

Ultimately, this work does not claim that the old math was wrong, but rather that it was answering a different question than the one researchers thought they were asking. The old tools measured the total allocation of attention, including the massive bias toward the first token. The new tools measure how the remaining attention is shared among the actual content. The authors provide a clear map of when each approach is safe to use and when it leads to disaster. They show that for models with strong attention sinks, the standard methods are unreliable for understanding the internal logic of the machine. By separating the two channels, researchers can now see the true geometry of the model's attention, distinguishing between a system that is genuinely simplifying its thoughts and one that is simply letting its first token dominate the conversation. This separation offers a more honest view of how these complex systems organize information, ensuring that future discoveries about their inner workings are based on the content of their thoughts, not the quirks of how they are measured.

Technical Summary: Attention Rows as Compositional Data

Problem Statement

Transformer attention mechanisms produce, for every query position, a probability distribution over keys. These rows lie on a simplex, making them compositional data in the sense of Aitchison (1982): vectors of positive parts carrying only relative information. A pervasive empirical fact in modern large language models (LLMs) is the existence of attention sinks, where a significant portion of the probability mass concentrates on a single coordinate (typically the first token or <bos>).

Standard interpretability tools—such as cosine similarity, Jensen–Shannon (JS) divergence, Euclidean distance, and Shannon entropy—treat these rows as standard Euclidean vectors. However, these metrics are subcompositionally incoherent. This means that conclusions drawn from comparing full attention rows (including the sink) can be reversed if the sink is masked and the remaining content is renormalized. The paper identifies that the field's standard summaries mix two distinct questions under an unreported convention:

Sink Question: How much attention does the sink token receive?
Content Question: How is the remaining probability mass distributed among content tokens?

When the sink dominates the mass (e.g., >80% in Llama-3.1-405B), standard metrics conflate these channels. A head with a large sink and uniform content may appear "close" to another head with a large sink but different content, purely due to the shared sink mass, even if their content distributions are orthogonal. Conversely, dropping the sink can artificially separate heads that share identical content distributions. This convention-dependence leads to unstable taxonomies, misleading pruning decisions, and misinterpretations of training dynamics (e.g., "entropy collapse").

Methodology

The authors apply Compositional Data Analysis (CoDA) to the measurement methodology of attention analysis.

Theoretical Framework

Aitchison Geometry: The paper treats attention rows as points in the Aitchison simplex. It utilizes the Centered Log-Ratio (CLR) transform to map the simplex to a Euclidean subspace where the Aitchison distance ( $d_A$ ) is the natural metric.
Orthogonal Decomposition: The authors prove that the Aitchison distance admits an exact orthogonal decomposition into a sink term and a content term:
$d_A(p, q)^2 = d_A^\perp(p, q)^2 + (b(p) - b(q))^2$
where $b(p)$ is the sink balance and $d_A^\perp$ is the distance between the sink-dropped, renormalized content compositions.
Axiomatic Characterization: The paper characterizes the Aitchison distance as the unique metric satisfying five axioms derived from the transformer's own symmetries: perturbation invariance (shared logit biases), powering homogeneity (temperature scaling), permutation invariance, the parallelogram law, and metric properties.
Entropy Decomposition: Shannon entropy is decomposed exactly as $H(p) = H_b(s) + (1-s)H(\pi)$ , separating the binary entropy of the sink mass ( $s$ ) from the conditional entropy of the content ( $\pi$ ).

Experimental Protocol

The authors validate these theoretical claims across ten pretrained models from five families (GPT-2, Pythia, Qwen2.5, Llama-3.2, ViT-B/16).

Reversal Rate Measurement: They sample 50,000 head triples and measure the frequency with which the "closer head" verdict flips between the "with-sink" and "sink-dropped" conventions for classical metrics versus the content distance $d_A^\perp$ .
Pruning Experiments: They prune the most redundant heads (10–30% sparsity) using different distance metrics (JS, $d_A^\perp$ , full $d_A$ ) and measure the impact on perplexity (for LMs) or top-1 agreement (for ViT).
Downstream Tests: They prospectively specified and tested predictions regarding taxonomy stability (re-clustering BERT/LLM heads), scale laws (sink mass vs. parameters), and training curves (entropy collapse decomposition).

Key Contributions

Diagnosis of Convention-Dependence: The paper proves (Theorem 1) that for any set of three heads where two share a content distribution but differ in sink mass, and a third shares the sink mass but has mirrored content, classical metrics (Cosine, JS, Euclidean) will reverse their "closest neighbor" verdicts depending on whether the sink is kept or dropped. This reversal occurs on a set of positive measure, which is not a rare edge case but a typical regime in sink-dominated models.
Coherent Toolkit: The authors introduce a toolkit based on Aitchison geometry:
- $d_A^\perp$ (Content Distance): The Aitchison distance computed on sink-dropped rows, which is invariant to sink mass changes.
- $C_A^\perp$ (Content Dispersion): A measure of content concentration independent of the sink.
- Exact Identities: An orthogonal Pythagorean decomposition of distance and an exact identity for entropy decomposition.
Failure Geometry: In the sink-dominated regime, classical pairwise dissimilarities collapse into vanishing bands (e.g., cosine similarities > 0.94 for sink mass $\ge$ 0.8), rendering nearest-neighbor rankings and clustering artifacts of the sink mass rather than content structure.
Empirical Validation:
- Reversal Rates: On ten models, 17–47% of "closer-head" verdicts flip between conventions.
- Taxonomy Instability: Re-clustering BERT and Llama heads with and without the sink yields Adjusted Rand Indices (ARI) near chance (0.01–0.05), indicating that the prominent structures in standard pipelines are artifacts of the convention.
- Pruning Catastrophe: Pruning based on classical metrics (JS) in strong-sink models (e.g., Llama-3.2-1B) can inflate perplexity by orders of magnitude (e.g., 15.3 $\to$ 2094), whereas pruning based on content distance ( $d_A^\perp$ ) remains stable.
- Entropy Collapse: Measured entropy collapse during training is largely driven by the growth of the sink, not a sharpening of content attention. For Pythia-1B, 95% of the entropy drop is attributable to the sink growing, not content collapse.

Results and Findings

Metric Stability: The content distance $d_A^\perp$ is identical whether computed on full rows (projected) or explicitly dropped rows, providing a stable, convention-invariant metric for content similarity.
Pruning Regimes: The paper maps distinct regimes for pruning. In models with heterogeneous sink masses (high cross-head dispersion), classical metrics fail. In models with uniform sinks, classical metrics may work, but $d_A^\perp$ remains robust. A frozen out-of-sample predictor based on sink dispersion correctly identified the regime for one model, abstained on another, and failed on a third, suggesting the rule is descriptive rather than universally validated.
Training Dynamics: The decomposition of entropy reveals that "entropy collapse" is often a "phantom" collapse caused by the sink absorbing mass, while the content distribution remains stable or even expands.

Significance and Claims

The paper claims to be the first work to bring log-ratio geometry to the measurement methodology of attention analysis and the first to explicitly separate the sink and content channels.

Measurement, Not Mechanism: The authors explicitly state they do not claim that transformers compute using log-ratios. Rather, they argue that the analysis of attention rows as compositions is necessary to answer well-posed questions about relative allocation among content tokens.
Convention Awareness: The primary contribution is demonstrating that standard conclusions in interpretability (head taxonomies, pruning scores, entropy collapse) are often convention-dependent artifacts. The paper provides the mathematical tools to separate these channels, allowing researchers to choose the appropriate estimand (sink-inclusive vs. content-only) and report it explicitly.
Practical Impact: The separation matters practically: using the wrong channel for pruning can lead to catastrophic performance degradation, and misinterpreting entropy collapse can lead to incorrect diagnoses of training stability.

The paper concludes that while the classical toolkit answers a convention-dependent question, the Aitchison toolkit answers a well-posed one, separating the "sink question" from the "content question" exactly.