Evaluating RAG Reliability under Clean, Misleading, and Mixed Retrieval

Technical Summary: Evaluating RAG Reliability under Clean, Misleading, and Mixed Retrieval

Problem Statement
Retrieval-Augmented Generation (RAG) systems are designed to enhance the factual reliability of Large Language Models (LLMs) by grounding responses in external evidence. However, a critical vulnerability exists in real-world information environments where retrieved content often contains heterogeneous data, including plausible but incorrect information (misinformation). Current evaluation frameworks often assume retrieved context is reliable or focus primarily on security-oriented "poisoning" attacks. There is a lack of systematic analysis regarding how RAG systems behave when the retrieved context contains varying levels of misinformation, particularly in scenarios where the model already possesses the correct answer in its parametric (internal) knowledge. The core problem is determining whether and how misleading evidence overrides correct internal knowledge and influences model confidence.

Methodology
The study proposes an evaluation protocol to test RAG robustness across four distinct contextual scenarios:

1. Clean: Context containing only correct passages.
2. Poisoned: Context containing only incorrect (misleading) passages.
3. Mixed-33: Context with one incorrect and two correct passages.
4. Mixed-67: Context with two incorrect and one correct passage.

Experimental Setup:

• Dataset: A subset of 100 factoid questions was sampled from the TruthfulQA dataset. Crucially, the study filtered for questions that the models answered correctly in a Parametric-Only setting (without retrieval), ensuring that any subsequent errors could be attributed to contextual influence rather than knowledge gaps.
• Models: Experiments were conducted using GPT-4o (a strong closed-weight model) and LLaMA-3.1-8B (a smaller open-weight model) to assess the impact of model capacity.
• Context Construction: Clean contexts were retrieved from Wikipedia using BM25. Misleading ("poisoned") contexts were synthetically generated by LLaMA-3.1-8B to be plausible, informative, and factually contradictory to the gold answer, mimicking realistic misinformation without relying on proprietary APIs for generation.
• Evaluation Metrics: The paper introduces three specific metrics to move beyond simple accuracy:
  1. Parametric Override Rate (POR): Measures the frequency with which retrieved context causes a model to abandon a correct parametric answer in favor of an incorrect one.
  2. Confidence Inflation (CI): Quantifies the increase in the model's assertiveness (confidence) when generating incorrect answers under misleading contexts compared to clean contexts.
  3. Poison Ratio Curve (PRC): Tracks the evolution of overall model accuracy as the proportion of misleading passages in the context increases from 0% to 100%.

Key Results

• Parametric Override: Both models exhibited significant susceptibility to overriding correct internal knowledge. Even in the Clean scenario, a substantial portion of correct parametric answers were overridden (POR of 0.378 for GPT-4o and 0.417 for LLaMA-3.1), suggesting that the mere presence of retrieved context can interfere with internal reasoning. As the proportion of incorrect context increased, POR rose systematically. In the Poisoned scenario, both models abandoned correct knowledge in over 56% of cases.
• Model Sensitivity: LLaMA-3.1 demonstrated higher sensitivity to contradictory evidence than GPT-4o, showing a sharper decline in performance in mixed scenarios (e.g., POR jumped to 0.583 in Mixed-67 for LLaMA-3.1).
• Confidence Inflation: Misleading contexts caused models to express incorrect answers with significantly higher confidence. In the fully poisoned scenario, both models displayed similar levels of confidence inflation (approx. 1.6), indicating that overconfidence in the face of misinformation is a general characteristic of RAG systems rather than an architecture-specific flaw.
• Accuracy Degradation: The Poison Ratio Curve revealed a clear inverse relationship between the amount of misinformation and model accuracy. Statistical tests (Cochran's Q) confirmed that the degradation in accuracy across context conditions was significant for both models.

Significance and Contributions
The paper claims to provide the first systematic evaluation framework that explicitly measures the extent to which models deviate from correct parametric knowledge under the influence of retrieved context. Its primary contributions include:

1. Behavioral Insight: Demonstrating that RAG reliability is not solely about retrieval accuracy but is heavily dependent on how models resolve conflicts between internal knowledge and external evidence.
2. New Metrics: Introducing POR, CI, and PRC to quantify the "override" effect and the dangerous combination of incorrect answers with high confidence.
3. Realistic Assessment: Highlighting that RAG systems are vulnerable even in "mixed" environments where misinformation is not dominant, challenging the assumption that correct information in the context will naturally prevail.

The authors conclude that in misinformation-rich environments, the reliability of retrieved evidence is as critical as its relevance. The study suggests that future RAG systems require credibility-aware retrieval mechanisms and conflict detection strategies to prevent models from confidently adopting misleading information over their own verified knowledge.