URAG: A Benchmark for Uncertainty Quantification in Retrieval-Augmented Large Language Models

1. Problem Statement

While Retrieval-Augmented Generation (RAG) has become a standard approach for enhancing Large Language Models (LLMs) with external knowledge, current evaluation frameworks focus almost exclusively on accuracy (correctness). This creates a critical blind spot:

• Overconfidence and Hallucinations: RAG systems often generate overconfident answers even when retrieval results are noisy, contradictory, or irrelevant.
• Lack of Uncertainty Metrics: Existing benchmarks do not provide a unified framework to quantify how retrieval affects model confidence, calibration, and reliability.
• Open-Ended Evaluation Challenges: Measuring uncertainty in free-form text generation is difficult because semantic validity is hard to define, and standard metrics (like perplexity) do not correlate well with factual correctness.

The paper argues that understanding the interplay between retrieval noise, parametric knowledge, and model uncertainty is essential for deploying trustworthy RAG systems in high-stakes domains like healthcare and finance.

2. Methodology: The URAG Framework

The authors introduce URAG, a comprehensive benchmark designed to evaluate both accuracy and predictive uncertainty across diverse RAG architectures and domains.

A. Benchmark Design & Pipeline

• Task Reformulation: To enable principled uncertainty quantification, the authors convert open-ended generation tasks into Multiple-Choice Question Answering (MCQA). This constrains the output space, allowing for the application of statistical uncertainty measures.
• Distractor Generation: A key innovation is the pipeline for generating high-quality "distractors" (incorrect answers).
  • The system generates plausible but incorrect answers that share the same semantic type as the correct answer.
  • It creates synthetic supporting documents for these wrong answers to increase plausibility.
  • An NLI (Natural Language Inference) model is used to filter and iterate on these distractors, ensuring they are "confusing" (i.e., entailed by the correct answer or logically similar) to prevent models from exploiting superficial cues.
• Domains & Datasets: The benchmark covers 5 domains (Healthcare, Code, Science/Math, General Text, Research) using 8 datasets (e.g., CRAG, HealthVer, ODEX, OlympiadBench, SciFact).
• RAG Methods Evaluated: The study tests 8 distinct RAG methods, ranging from training-free (Naive RAG, HyDE, Fusion) to end-to-end (FiD, Self-RAG) and complex reasoning pipelines (RAPTOR, RAT, REPLUG).

B. Uncertainty Quantification via Conformal Prediction

Instead of relying on entropy or temperature scaling, URAG uses Conformal Prediction (CP) to generate statistically valid uncertainty sets.

• Mechanism: For a given query, CP constructs a prediction set $S_\alpha(x)$ containing candidate answers such that the true answer is included with probability at least $1-\alpha$.
• Metrics:
  • Accuracy (Acc): Standard top-1 accuracy.
  • Set Size (SS): The average number of options in the prediction set. A larger SS indicates higher uncertainty; a smaller SS indicates higher confidence.
  • Coverage Rate (CR): Verifies if the true answer falls within the prediction set (targeting a specific risk level $\alpha$).
• Scoring Rules: The paper utilizes two nonconformity scores: LAC (Least Ambiguous Class) and APS (Adaptive Prediction Sets).

C. Experimental Controls

To isolate the effects of retrieval, the authors introduce specific diagnostic settings:

1. Irrelevant Context: Injecting irrelevant documents to test robustness against noise.
2. Knowledge Isolation: Splitting test sets into "LLM-Correct" (where the model knows the answer without retrieval) and "LLM-Incorrect" (where retrieval is necessary) to see how retrieval reshapes confidence.
3. Prompt Manipulation:
   • Self-Aware: Exposing the model to its own confidence scores.
   • Wrong-Aware: Deliberately providing misleading confidence scores to test if the model prioritizes retrieved evidence or deceptive cues.

3. Key Contributions

1. URAG Benchmark: The first unified benchmark to jointly evaluate accuracy and uncertainty across multiple RAG methods and domains.
2. MCQA Construction Pipeline: A novel method for generating challenging, plausible distractors using iterative NLI-based filtering to ensure high-quality uncertainty measurement.
3. Systematic Analysis: A deep dive into how retrieval depth, noise, and model architecture influence the accuracy-uncertainty trade-off.

4. Key Results & Findings

The evaluation of 8 RAG methods on Llama-3.1-8B revealed several critical insights:

• Observation 1: Architecture Matters.

  • Simple, modular methods (e.g., Naive RAG, Fusion, HyDE) generally achieve the best trade-off between high accuracy and low uncertainty (small set sizes).
  • Complex reasoning pipelines (e.g., RAT) often suffer from lower accuracy and higher uncertainty, suggesting that multi-step reasoning can introduce noise or attention diversion.
  • FiD showed the poorest calibration, with high uncertainty and low accuracy, likely due to its encoder-decoder limitations in fusing multiple passages.

• Observation 2: The Accuracy-Uncertainty Correlation.

  • Generally, there is a negative correlation between accuracy and uncertainty (higher accuracy $\rightarrow$ lower uncertainty).
  • Exception: REPLUG breaks this trend. It achieves high accuracy but exhibits high uncertainty (large set sizes). This is because REPLUG aggregates probabilities from independent passages, reducing reliance on any single source and thus avoiding overconfidence, even when correct.

• Observation 3: Retrieval Noise and Irrelevance.

  • When retrieval introduces irrelevant or noisy documents, the relationship between accuracy and uncertainty breaks down.
  • Models with strong parametric knowledge (e.g., on code tasks like ODEX) are less affected by noise.
  • However, in domains where the LLM lacks knowledge, noisy retrieval significantly degrades both accuracy and calibration, leading to confident errors.

• Observation 4: Sensitivity to Confidence Cues.

  • RAG systems are surprisingly brittle to misleading confidence cues (Wrong-Aware prompting). When presented with deceptive confidence scores, models often ignore the retrieved evidence and follow the prompt, leading to increased hallucinations and accuracy drops.
  • REPLUG was found to be largely invariant to these prompt perturbations, maintaining stability.

• Observation 5: Domain Specificity.

  • No single RAG method is universally reliable. For example, Naive RAG performs best in Healthcare (prioritizing caution), while HyDE excels in Research tasks (hypothesis-driven retrieval), and RAT is strong in Math (iterative refinement).

5. Significance

• Trustworthiness: URAG provides a necessary tool for assessing the reliability of RAG systems, not just their correctness. It highlights that high accuracy does not guarantee low uncertainty or safety.
• Safety in High-Stakes Domains: The findings suggest that in sensitive fields (healthcare, law), systems must be designed to recognize uncertainty rather than just maximizing accuracy. Simple modular approaches may be safer than complex reasoning chains in noisy environments.
• Future Research: The benchmark establishes a baseline for developing RAG systems that can better balance confidence with evidence quality, detect retrieval noise, and resist prompt injection attacks that manipulate perceived confidence.

In conclusion, URAG demonstrates that retrieval is a double-edged sword: it can improve factual grounding but also amplify overconfident hallucinations if the retrieval process is noisy or if the system architecture is not robust to uncertainty. The benchmark serves as a critical step toward building more trustworthy AI agents.