Technical Summary: How Important is Recall for Measuring Retrieval Quality?

Problem Statement

In Retrieval-Augmented Generation (RAG) systems, the quality of the retrieved documents (the "top-K" selection) directly impacts the quality of the response generated by a Large Language Model (LLM). Standard retrieval metrics, such as the F-measure, rely on Recall ($R = n_p / N_p$), which requires knowing the total number of relevant documents ($N_p$) in the knowledge base (KB) for a given query.

However, in realistic, large-scale, and evolving KBs, $N_p$ is typically unknown in advance. Consequently, Recall cannot be computed, rendering the F-measure and similar metrics inapplicable for real-time evaluation. Existing strategies to circumvent this include creating fixed benchmarks (which become obsolete as the KB changes), estimating $N_p$ via pooling or extrapolation, or using metrics that do not require $N_p$ (e.g., nDCG). The paper investigates which approach best correlates with the actual quality of the LLM's generated response when the total number of relevant documents is unknown.

Methodology

Dataset Construction

The authors constructed a benchmark dataset ("retrieval-response") comprising queries, positive (relevant) documents, and negative (irrelevant) documents from four sources:

1. ARXIV: Scientific abstracts.
2. HotpotQA: Paragraphs and sentences.
3. MSMARCO: Web search snippets.
4. Natural Questions: Web queries.

The dataset features a relatively low number of relevant documents ($N_p \in [2, 15]$). The authors varied the retrieval size $K$ across a range relative to $N_p$ (both $K < N_p$ and $K \ge N_p$).

Retrieval and Evaluation Setup

• Embeddings: Four popular embedding models (all-MiniLM-L12-v2, multilingual-e5-small, e5-small-v2, multi-qa-MiniLM-L6-cos-v1) were used to retrieve top-K documents.
• Response Generation: An LLM (GPT-4o-mini) generated responses based only on the top-K retrieved documents.
• Quality Scoring: The generated responses were compared against an "ideal" response (generated using all positive documents). The LLM assigned a quality score (1–5 Likert scale) based on content completeness and alignment.
• Validation: The LLM's scoring was validated against human judgments, showing strong correlation (Krippendorff's $\alpha \approx 0.77–0.82$).

Metrics Compared

The study evaluated the correlation between the following retrieval metrics and the LLM response quality score:

1. F-measure: Requires exact knowledge of $N_p$.
2. Fe (F-estimated): Uses a crude estimate of $N_p$ (count of positives in the top $2K$) instead of the true $N_p$.
3. nDCG: Does not require $N_p$; accounts for document ranking order.
4. T (Proposed): A new metric defined as $T = \frac{(1-\alpha)n_p - \alpha n_n}{K}$, where $n_p$ is the number of positives and $n_n$ is the number of negatives in the top-K. It is designed to be independent of $N_p$.

The authors analyzed Spearman, Pearson, and Kendall Tau correlations across different datasets, embedding models, and $K/N_p$ ratios.

Key Results

1. Correlation with Response Quality

• Metric Performance: The proposed T metric generally performed as well as or better than the standard F metric in scenarios where $K < N_p$ or $K \approx N_p$ (specifically on ARXIV and HotpotQA-sentences).
• nDCG Limitations: nDCG generally showed lower correlation with response quality than F or T, except in specific cases with high $K/N_p$ ratios on complex datasets (like ARXIV). In these high-ratio scenarios, the order of documents (relevant first, irrelevant later) became critical for the LLM, favoring nDCG.
• Estimation vs. Reality: Surprisingly, Fe (using an estimated $N_p$ from the top $2K$) sometimes outperformed the true F metric at high $K/N_p$ ratios. The authors hypothesize that relevant documents appearing in the top $2K$ are inherently more important to the LLM than those appearing later, making the "estimated" $N_p$ a better proxy for the useful relevant documents.

2. Impact of $K/N_p$ Ratio

• Low/Moderate $K/N_p$: When the number of retrieved documents is comparable to or less than the total relevant documents, the presence of irrelevant documents (negatives) is the primary driver of poor response quality. Metrics focusing on the balance of positives and negatives (like T and F) perform best.
• High $K/N_p$: When $K$ significantly exceeds $N_p$, the dataset contains many irrelevant documents. Here, the ordering of these documents matters. If the LLM receives relevant documents early in the context, performance is higher. This is where nDCG gains an advantage, though it remains inferior to T and F on simpler datasets (MSMARCO, Natural Questions).

3. Embedding Model Sensitivity

The choice of retrieval metric was found to be more dependent on the dataset type and the $K/N_p$ ratio than on the specific embedding model used.

Key Contributions

1. Introduction of Metric T: The authors propose a simple, normalized retrieval quality measure (T) that does not require knowledge of the total number of relevant documents ($N_p$). It balances the count of relevant documents ($n_p$) against irrelevant ones ($n_n$) within the top-K.
2. Comprehensive Benchmark: Creation of a diverse dataset linking retrieval configurations (various $K$, $N_p$, and embeddings) to LLM-generated responses and quality scores, facilitating the study of retrieval quality in the absence of ground-truth recall.
3. Empirical Insights:
   • Demonstrated that T is a robust alternative to F for real-world evaluation where $N_p$ is unknown.
   • Showed that nDCG is not universally superior; its advantage is limited to high $K/N_p$ ratios on complex datasets where document order significantly impacts LLM performance.
   • Observed that estimating $N_p$ via the top $2K$ (Fe) can sometimes yield better correlations with response quality than using the true $N_p$, suggesting that "early" relevant documents carry more weight for LLMs.

Significance and Claims

The paper claims that in realistic RAG settings where the total number of relevant documents is unknown, Recall is not strictly necessary to measure retrieval quality effectively.

• Practical Utility: The T metric offers a "well-founded and convenient" solution for real-life evaluations, avoiding the need for fixed benchmarks or complex $N_p$ estimation techniques.
• Contextual Nuance: The study highlights that the "best" metric depends on the operational context (specifically the $K/N_p$ ratio and document complexity). While F (or T) is generally sufficient, nDCG may be preferred only when dealing with large retrieval windows ($K \gg N_p$) on complex data where the ordering of irrelevant documents confuses the LLM.
• LLM-Centric Evaluation: The work reinforces the shift toward evaluating retrieval systems based on their downstream impact on LLM response quality rather than abstract information retrieval metrics alone.

The authors remain modest regarding the scope, noting that their experiments focused on moderate $N_p$ ranges and that latency considerations for real-time evaluation (e.g., using LLMs to label relevance on the fly) were not fully explored.