ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting

1. Problem Statement

The paper addresses two critical limitations inherent in attention-based re-ranking methods using Large Language Models (LLMs). While these methods offer an efficient, zero-shot alternative to generation-based re-ranking by deriving relevance scores directly from internal attention weights, they suffer from:

• Signal Concentration: Attention mechanisms tend to concentrate the vast majority of "attention mass" on a very small subset of tokens within a few documents. Consequently, many candidate documents receive negligible attention scores, making it difficult for the re-ranker to distinguish between moderately relevant and irrelevant documents.
• Lexical Bias: Attention weights often disproportionately highlight tokens that are lexically identical to the query (e.g., common nouns or named entities). This leads to false positives, where documents with mere surface-level keyword overlap are ranked highly despite lacking semantic relevance.

These issues result in suboptimal ranking precision, particularly in scenarios involving long contexts or documents with high lexical overlap but low semantic relevance.

2. Methodology: ReAttn

The authors propose ReAttn, a post-hoc re-weighting strategy that modifies existing attention scores without requiring additional training, fine-tuning, or supervision. The method operates in two sequential stages:

A. Cross-Document IDF Weighting (Token-Level)

To mitigate lexical bias, the method down-weights query tokens that appear frequently across the entire candidate document set.

• Mechanism: For every query token $t$ appearing in a document, the method calculates a normalized Inverse Document Frequency (IDF) weight based on the token's frequency across all $N$ candidate documents.
• Formula: $w(t) = \frac{\log((N+1)/(df(t)+1))}{\log(N+1)}$, where $df(t)$ is the document frequency.
• Effect: Ubiquitous tokens (high $df$) receive lower weights, while distinctive tokens (low $df$) retain higher weights. This reduces the influence of common lexical overlaps and emphasizes terms that provide stronger discriminative evidence.

B. Entropy-Based Regularization (Document-Level)

To address signal concentration, the method encourages a more balanced distribution of attention across informative tokens within each document.

• Mechanism:
  1. Token-level scores within a document are normalized into a probability distribution.
  2. Shannon Entropy is computed for this distribution. High entropy indicates a broad, balanced focus on multiple tokens; low entropy indicates an overly narrow focus on a few tokens.
  3. A dispersion weight is assigned to each document based on its deviation from the mean entropy of the batch. Documents with higher entropy (better coverage) are up-weighted, while those with low entropy (over-concentrated) are penalized.
• Effect: This prevents the "attention collapse" where only top-ranked documents receive signals, ensuring that moderately relevant documents with distributed evidence are not overlooked.

3. Key Contributions

1. Problem Identification: The paper formally identifies and analyzes signal concentration and lexical bias as the primary bottlenecks in current attention-based re-ranking approaches.
2. ReAttn Framework: It introduces a novel, training-free post-hoc strategy combining cross-document IDF weighting and entropy-based regularization to refine attention signals.
3. Empirical Validation: Extensive experiments demonstrate that ReAttn consistently improves the performance of existing attention-based baselines (ICR and QRhead) across diverse datasets and model architectures, achieving competitive results against supervised cross-encoders without the computational cost of training.

4. Experimental Results

The authors evaluated ReAttn on 11 BEIR datasets (re-ranking task) and LongMemEval/CLIPPER (long-context reasoning task) using three base LLMs: Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B.

• Re-Ranking Performance (BEIR):

  • ReAttn consistently improved the average nDCG@10 across all models.
  • For Llama-3.1-8B, the best baseline (QRhead) achieved 50.0 nDCG@10, which was boosted to 50.9 with ReAttn.
  • The improvements were particularly significant on datasets with high lexical overlap (e.g., DBPedia, FiQA), confirming the efficacy of the IDF component.
  • ReAttn outperformed traditional supervised retrievers like MSMARCO-MiniLM (48.0) and GTR-T5-base (45.2) while remaining fully parameter-free.

• Long-Context Reasoning:

  • In long-context scenarios, ReAttn improved Recall@K and downstream end-to-end reasoning accuracy.
  • The entropy regularization was crucial here, helping the model identify sparse but critical evidence distributed across long sequences rather than collapsing attention onto the beginning of the context.

• Ablation Studies:

  • IDF only: Provided the majority of gains on lexically dominated datasets.
  • Entropy only: Provided larger gains on heterogeneous, long-document datasets.
  • Combined: The full ReAttn model outperformed either component in isolation, proving they address non-overlapping failure modes.

5. Significance

• Efficiency & Interpretability: ReAttn offers a highly efficient alternative to generation-based re-ranking (which suffers from high latency and prompt sensitivity) and supervised cross-encoders (which require labeled data and fine-tuning). It leverages the intrinsic interpretability of attention mechanisms.
• Zero-Shot Capability: It enables state-of-the-art zero-shot re-ranking performance by simply refining the internal signals of pre-trained LLMs, making it applicable to any instruction-tuned LLM without retraining.
• Robustness: The method significantly enhances the robustness of attention-based retrieval, particularly in "hard-negative" scenarios where irrelevant documents share keywords with the query, and in long-context tasks where attention saturation is a known issue.

In conclusion, ReAttn demonstrates that simple, mathematically grounded post-hoc adjustments to attention weights can significantly unlock the potential of LLMs for information retrieval, bridging the gap between efficient attention-based scoring and the performance of supervised models.