Reliable Evaluation Protocol for Low-Precision Retrieval

1. Problem Statement

The paper addresses a critical reliability issue in evaluating retrieval systems (such as those used in RAG pipelines) when operating under low-precision numerical formats (e.g., BF16, FP16).

• The Core Issue: Lowering numerical precision reduces the granularity of representable floating-point numbers. This causes distinct relevance scores to collapse into identical values, creating spurious ties among candidate documents.
• The Consequence: Standard evaluation metrics (like nDCG, MRR, Recall) are "tie-oblivious." When ties occur, these metrics arbitrarily break them based on internal ordering (e.g., document IDs or memory layout).
  • This introduces high variability in evaluation results; the same model can yield significantly different scores depending on the random tie-breaking order.
  • It leads to misleading conclusions, such as ranking a lower-performing model higher than a superior one, or systematically overestimating performance (positive bias).
• Current Gap: Mainstream benchmarks (e.g., MTEB) lack principled mechanisms to handle these ties, relying on arbitrary truncation that renders low-precision evaluation unstable and untrustworthy.

2. Methodology

The authors propose a two-part protocol to resolve spurious ties and quantify evaluation uncertainty:

A. High-Precision Scoring (HPS)

• Concept: Instead of running the entire model in high precision (which negates efficiency gains), HPS upcasts only the final scoring step (the logits) to FP32.
• Mechanism:
  1. The model runs its forward pass in low precision (e.g., BF16).
  2. The resulting logits are converted to FP32 before being passed through the final scoring function (softmax, sigmoid, or pairwise product).
  3. Since every BF16/FP16 value is exactly representable in FP32, the magnitude is preserved, but the subsequent scoring function operates with higher granularity.
• Benefit: This collapses spurious ties by distinguishing scores that were previously quantized to the same value, while incurring negligible computational overhead (only a small tensor conversion).

B. Tie-aware Retrieval Metric (TRM)

• Concept: Even with HPS, some ties may remain (or for theoretical completeness). TRM provides a robust statistical framework to evaluate tied candidates without arbitrary bias.
• Components:
  1. Expected Score ($\mathbb{E}[M]$): Calculates the closed-form expectation of the metric over all possible permutations of tied candidates. This removes the randomness of tie-breaking.
  2. Score Range ($Range(M)$): Computes the difference between the maximum possible score ($M_{max}$, where relevant items in a tie group are ranked first) and the minimum possible score ($M_{min}$, ranked last). This quantifies the uncertainty introduced by the tie.
  3. Score Bias ($Bias(M)$): Measures the deviation between the standard tie-oblivious score ($M_{obl}$) and the expected score ($\mathbb{E}[M]$). A large positive bias indicates the standard metric is overestimating performance due to favorable (but arbitrary) tie-breaking.

3. Key Contributions

1. Formal Identification: The paper is the first to formally analyze and demonstrate that low-precision inference introduces spurious ties that destabilize retrieval evaluation, leading to non-deterministic and biased results.
2. HPS Protocol: A lightweight, drop-in solution that restores ranking granularity by upcasting only the final scoring operation, effectively eliminating spurious ties with minimal cost.
3. TRM Framework: A new evaluation methodology that reports expected values, ranges, and biases, providing a transparent view of evaluation stability and uncertainty.
4. Empirical Validation: Extensive experiments across multiple models (Qwen, BGE, E5), scoring functions (Softmax, Sigmoid, Dot Product), and datasets (MIRACL, AskUbuntu, MTEB-R) demonstrating the protocol's effectiveness.

4. Experimental Results

The authors evaluated five models on three datasets under FP32, BF16, and BF16+HPS regimes.

• Reduction in Instability:
  • In standard BF16, the Qwen3-Reranker model showed massive instability: a 25.59% range in nDCG@10 and 38.03% in MRR@10.
  • Applying HPS reduced these ranges dramatically to 1.13% and 1.21%, respectively, bringing them close to FP32 stability.
  • For Sigmoid and Dot Product models, HPS reduced tie-induced variance by an order of magnitude or made them perfectly deterministic (Range = 0).
• Correction of Ranking Errors:
  • In BF16, the tie-oblivious metric incorrectly ranked Qwen3-Reranker (75.04) higher than gte-multilingual-reranker (74.48).
  • The Tie-aware Expectation ($\mathbb{E}[M]$) correctly reversed this, showing Qwen (68.38) was actually worse than gte (74.22). HPS confirmed this reversal (73.35 vs 74.34).
• Bias Analysis:
  • Standard BF16 evaluations showed a systematic positive bias (up to +9.08%), meaning they consistently overestimated model performance because relevant documents were often placed earlier in arbitrary tie-breaking orders.
  • TRM exposed this bias, allowing researchers to correct for it.
• Efficiency: HPS adds negligible time and memory overhead (approx. 5ms for 1M elements on an H200 GPU) while recovering near-FP32 ranking correlation ($\rho \approx 0.999$).

5. Significance

• Reliability for RAG: Since retrieval quality directly impacts downstream generation in RAG systems, this protocol ensures that the selection of retrieval models is based on accurate, stable metrics rather than numerical artifacts.
• Efficiency vs. Accuracy Trade-off: It enables the use of efficient low-precision hardware (BF16/FP16) for retrieval without sacrificing the fidelity of the evaluation process.
• Standardization: The paper proposes a new standard for reporting retrieval metrics (Expected Value + Range + Bias), moving the field away from arbitrary tie-breaking toward statistically robust evaluation.
• Accessibility: The solution is easy to implement (often a single line of code change) and does not require retraining models, making it immediately adoptable by the research community.

In conclusion, the paper provides a necessary "patch" for the evaluation infrastructure of modern retrieval systems, ensuring that efficiency gains from low-precision computing do not come at the cost of evaluation integrity.