Information Representation Fairness in Long-Document Embeddings: The Peculiar Interaction of Positional and Language Bias

1. Problem Statement

The paper addresses a critical flaw in state-of-the-art text embedding models used for Retrieval-Augmented Generation (RAG) and search systems: Information Representation Fairness.

• The Core Issue: When long documents composed of multiple distinct segments (e.g., a newspaper page with multiple articles, or a legal document with appendices) are embedded, the resulting vector representation is not a uniform average of all parts. Instead, the model exhibits systematic biases:
  • Positional Bias: Early segments are over-represented, while later segments are marginalized ("lost in the middle" or "lost at the end").
  • Language Bias: Segments in high-resource languages (e.g., English) are over-represented compared to those in lower-resource languages (e.g., Hindi, Korean), regardless of their position.
• Consequence: In a semantic search scenario, if a user queries information located in a later segment or a lower-resource language section, the document embedding may not reflect that content, rendering the information "invisible" to the retrieval system.

2. Methodology

The authors propose a novel evaluation framework and a mitigation strategy based on attention calibration.

A. Evaluation Framework

To quantify these biases without confounding factors, the authors constructed a permutation-based experimental setup:

• Corpus: A multilingual comparable corpus of Wikipedia articles across six languages (English, Chinese, German, Italian, Korean, Hindi).
• Document Construction: Documents are formed by concatenating $n$ segments ($n \in \{3, 4, 5, 6\}$). Crucially, the authors generate all $n!$ permutations of these segments for a given set. This ensures that any observed bias is due to position or language, not content.
• Metrics:
  1. Positional Fairness: Measured by the cosine similarity ($\sigma$) between the global document embedding ($e_D$) and the standalone embedding of a specific segment ($e_{s}^{iso}$). A fair model should yield equal similarity regardless of the segment's position.
  2. Information Retention: Measured by the cosine similarity ($\tau$) between a segment's standalone embedding and its contextualized embedding within the long document. This quantifies how much semantic information is lost or distorted during contextualization.
• Statistical Analysis: Ordinary Least Squares (OLS) regression is used to estimate deviations from fairness, treating segment positions as categorical variables.

B. Models Analyzed

Two state-of-the-art multilingual encoder models were tested:

1. mGTE: Uses the start-of-sequence token (<s>) embedding for pooling.
2. Jina-v3: Uses mean pooling over contextualized token representations.

C. Proposed Solution: Inference-Time Attention Calibration

The authors hypothesize that positional bias stems from front-loaded attention distributions in the pooling token (specifically the <s> token), where early tokens receive disproportionate attention mass.

• Method: A training-free, inference-time intervention.
• Mechanism:
  1. The key sequence is partitioned into fixed-size contiguous "baskets" (e.g., size $B=128$).
  2. The attention scores of the <s>-query token are re-weighted to enforce uniform attention mass across all baskets.
  3. This preserves the fine-grained distribution within a basket but ensures later baskets receive as much total attention as earlier ones.
• Implementation: Requires only two forward passes (one to estimate bias, one to apply calibration) and operates on specific transformer layers ($L_C$).

3. Key Results

A. Empirical Findings on Bias

• Strong Positional Bias: Both models exhibit an "L-shaped" representation profile. The first segment is significantly over-represented. For example, in mGTE with German documents, the similarity drop for the 2nd and 3rd segments is substantial (coefficients $\approx -0.5$).
• Language Interaction:
  • English/Chinese Advantage: In mixed-language documents, English and Chinese segments maintain high representation even when placed in later positions, effectively counteracting positional bias.
  • Lower-Resource Disadvantage: Segments in German, Italian, Korean, and Hindi suffer severe marginalization when placed later in the document.
• Attention Analysis: Self-attention analysis of the <s> token confirms a front-loaded distribution. The first basket of tokens receives the highest attention mass, directly correlating with the over-representation of the first segment.

B. Effectiveness of Attention Calibration

• Bias Reduction: Applying the calibration method significantly flattens the "L-shaped" profiles.
  • Monolingual: The similarity scores for later segments increase, while the over-representation of the first segment decreases, leading to a more uniform distribution.
  • Mixed-Language: The method successfully mitigates the "language penalty." For instance, in mixed English-German documents, calibrated embeddings allow later German segments to be represented more fairly, reducing the dominance of the first English segment.
• Semantic Fidelity: Control experiments confirm that calibration does not degrade the semantic content of the embeddings. In fact, the similarity between the calibrated document embedding and the standalone segment embedding often increases for later segments, indicating that more meaningful information is being captured.

4. Key Contributions

1. Evaluation Framework: Introduced a permutation-based framework to isolate and quantify positional and language biases in long-context embeddings.
2. Discovery of Interaction: Revealed that positional bias is not uniform but interacts with language resource levels, where high-resource languages can "override" positional disadvantages.
3. Mechanism Identification: Linked the observed bias to front-loaded attention distributions in pooling-token embeddings.
4. Mitigation Strategy: Proposed a training-free, inference-time attention calibration method that redistributes attention mass evenly, substantially improving information representation fairness without retraining the model.

5. Significance and Limitations

• Significance:
  • RAG & Search Reliability: The findings highlight a major reliability issue in current RAG systems; critical information in later parts of documents or less common languages may be systematically ignored.
  • Fairness in AI: Provides a concrete definition and metric for "Information Representation Fairness," moving beyond simple accuracy metrics.
  • Practical Utility: The proposed calibration method is immediately deployable as a post-processing step for existing models, offering an easy fix for practitioners.
• Limitations:
  • Proxy Metric: The fairness metric relies on cosine similarity, which is a geometric proxy and may not perfectly capture intrinsic information contribution.
  • Scope: The study focuses on encoder-based models; decoder-based or hybrid architectures may exhibit different bias patterns.
  • Data Source: The evaluation uses Wikipedia articles, which may differ structurally from real-world long documents (e.g., legal texts, reports).
  • Downstream Impact: While the embedding space is improved, the direct impact on end-task retrieval performance (e.g., recall@k in a live search engine) requires further validation.

In conclusion, the paper demonstrates that current long-document embedding models are inherently biased toward the beginning of the text and high-resource languages. By identifying the attention mechanism as the root cause and offering a simple calibration fix, the authors provide a pathway toward more equitable and reliable semantic search systems.