Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

Technical Summary: Evaluating RAG Metrics in Applied Contexts

Problem Statement
Evaluating Retrieval Augmented Generation (RAG) systems remains a significant challenge, particularly when automatic metrics must approximate human judgment. While "LLM-as-a-judge" metrics are increasingly popular and often appear superior to classical metrics like BLEU, their relevance on specific datasets is difficult to predict a priori. Evaluation criteria vary widely (e.g., factuality, relevance, completeness), and without empirical validation, it is unclear whether a specific metric provides a reliable approximation of the desired criterion or enables the reliable comparison of different system iterations.

Methodology
The authors conducted an empirical study to evaluate the relevance of RAG metrics from four libraries: Ragas, DeepEval, RAGChecker, and Opik. The study utilized a proprietary dataset of 96 question-answer pairs derived from French business documents in the telecommunications domain.

• Data Construction: Annotators (company employees) created questions, reference answers, and reference spans (extracts from documents containing the necessary information) from a corpus of 479 documents. The dataset includes various question types (boolean, conditional, "how," etc.) and spans of varying lengths (1 to 202 words).
• System Configuration: A RAG system was deployed using a hybrid retriever (dense + BM25) to fetch the top 5 spans, which were then fed into a GPT-3.5 generator to produce answers.
• Evaluation Protocol:
  • Reference Scores: Two expert evaluators scored the system's generated responses on a 1–5 scale based on a rubric measuring factuality and relevance. The Pearson correlation between the two evaluators was 0.85, establishing the reliability of the human baseline.
  • Metric Comparison: The study calculated Pearson correlations between:
    1. Overall metrics (from the four libraries) and the average human reference scores.
    2. Retrieval metrics and a word-level recall score (used as the reference for retrieval performance).
    3. Retrieval metrics and the human reference scores (based on the premise that better retrieval correlates with better responses).
  • Scope: The study focused on a single RAG system to test the methodology's advantages and limitations, rather than to definitively rank the libraries against each other.

Key Findings

• Overall Metrics: Traditional metrics like METEOR showed surprisingly strong correlations with human scores. Conversely, generation metrics (e.g., faithfulness/hallucination scores) and Opik's moderation metric showed weak correlations with human evaluations, likely because they lack access to reference answers. RAGChecker's metrics, particularly recall, demonstrated very strong correlations.
• Retrieval Metrics: While confidence intervals were wide due to the small sample size (N=96), DeepEval's contextual precision achieved a correlation above 0.5 with recall without using reference spans. In contrast, Ragas's non-LLM metrics showed very weak correlations, potentially due to the use of Levenshtein distance on spans of highly variable sizes.
• The "Surprising" Correlation: Several metrics, notably RAGChecker's "claim recall," exhibited correlations with human scores (approaching 0.7) that were stronger than their correlation with the retrieval recall metric itself.

Limitations and Interpretation
The authors explicitly highlight a critical limitation regarding the interpretation of correlations. Because the study evaluated only one RAG system, it is impossible to disentangle whether a metric's high correlation with human scores is due to its ability to measure retrieval/generation quality or its sensitivity to input characteristics (e.g., question difficulty).

• Illustrative Example: A metric that simply scores based on question difficulty would correlate highly with human scores (as easy questions yield better answers) but would be useless for comparing system performance.
• Specific Observation: The high correlation of RAGChecker's "claim recall" with human scores may partially reflect the ease of extracting information from retrieved spans or question difficulty, rather than pure retrieval quality.
• Utility: Despite this, the methodology successfully identifies metrics with poor correlations, effectively eliminating them as candidates for the specific evaluation criterion.

Significance and Future Directions
The paper concludes that its primary contribution is the application of a methodology to test metric reliability, rather than a definitive ranking of libraries. The authors align their findings with recent literature suggesting that correlation studies involving a single system are methodologically limited.

The paper advocates for future research and practice to integrate multiple RAG systems into correlation studies. By varying system outputs, researchers can better isolate metric behavior from input characteristics, thereby reducing confounding factors and increasing the statistical power to discriminate between metrics. The authors suggest that developing procedures to optimize the number of systems and questions required for such studies would help practitioners validate evaluation metrics more reliably while minimizing annotation costs.