Open-Source Reproduction and Explainability Analysis of Corrective Retrieval Augmented Generation

This paper presents a fully open-source reproduction of the Corrective Retrieval Augmented Generation (CRAG) system using Wikipedia and Phi-3-mini, demonstrating comparable performance to the original while providing the first SHAP-based explainability analysis that reveals the evaluator's reliance on named entity alignment and identifies key failure modes.

The Gist

Imagine you are a brilliant but slightly forgetful student named LLM (Large Language Model). LLM is great at writing essays and answering questions, but sometimes it gets things wrong because it's making things up out of thin air (a problem called "hallucination").

To fix this, LLM has a study partner called RAG (Retrieval-Augmented Generation). Before answering a question, RAG runs to the library, grabs a few books, and hands them to LLM to help write the answer.

The Problem:
Sometimes, RAG grabs the wrong books. Maybe it grabs a book about "Apples" when the question is about "Apple the company." If LLM reads the wrong book, it gets confused and gives a wrong answer.

The Original Solution (CRAG):
The original researchers created a super-smart "Librarian" (called a Retrieval Evaluator) to check the books before LLM reads them.

• If the book looks perfect, the Librarian says, "Correct!" and LLM uses it.
• If the book is totally wrong, the Librarian says, "Incorrect!" and sends LLM to the internet (Google) to find better info.
• If the book is okay but not great, the Librarian says, "Ambiguous!" and LLM uses both the library book and the internet.

The Catch:
The original "Librarian" and the "Internet" were locked behind paywalls. You needed a credit card for Google Search and special, expensive software to run the Librarian. This meant regular researchers couldn't test or improve the system.

---

What This Paper Did: The "Open-Source" Makeover

The author, Surya, decided to rebuild this entire system using only free, open-source tools, like swapping a private, paid library for a public one.

Here is the simple breakdown of their work:

1. The Swap (Reproduction)

Surya replaced the expensive parts with free alternatives:

• The Internet: Instead of paying for Google Search, they built a clever robot that searches Wikipedia (the free online encyclopedia).
• The Brain: Instead of using a heavy, expensive AI model, they used a smaller, free, and very smart model called Phi-3.
• The Librarian: They kept the original "Librarian" (the T5 model) but made sure it could run on the new, free setup.

The Result: The new, free system worked almost exactly as well as the expensive, original one. It proved you don't need to spend a fortune to get smart AI results.

2. The Detective Work (Explainability)

This is the most interesting part. The researchers wanted to know: How does the Librarian actually decide if a book is good or bad?

They used a tool called SHAP (think of it as an X-ray machine for AI brains) to see what the Librarian was looking at.

The Big Discovery:
They expected the Librarian to be reading the meaning of the sentences (Semantic Understanding).

• Example: They thought the Librarian understood that "The President of France" and "Emmanuel Macron" mean the same thing.

But the X-ray showed something else:
The Librarian is actually just a Name Matcher. It's not really reading for meaning; it's just checking if the names in the question match the names in the book.

• If the question asks about "Henry" and the book has "Henry," the Librarian says "Correct!"
• If the question asks about "Henry" and the book is about "Mitochondria," the Librarian says "Incorrect!"

The Metaphor:
Imagine a security guard at a club. You expect him to check your ID to see if you are a good person. But instead, he just checks if your name is on a list.

• If your name is "John," he lets you in.
• If your name is "Jane," he lets you in.
• But if you are a famous actor named "Titanic" (a weird name), and the list doesn't have "Titanic," he kicks you out, even if you are a good person.

3. The Failure Modes (Where it breaks)

Because the Librarian is just a "Name Matcher," it fails in two funny ways:

1. The "Science" Problem: The Librarian was trained mostly on questions about people (like "Who is the President?"). When asked a science question like "Why is the sky blue?", there are no specific names to match. The Librarian gets confused and says, "I don't know, let's search the internet," even if the answer is simple.
2. The "Movie" Problem: If you ask about a movie like "Titanic," the Librarian gets confused because it doesn't recognize "Titanic" as a person's name in its training list. It rejects good answers just because the name didn't match its specific list.

The Takeaway

This paper is like a DIY guide for the AI world.

1. It showed that you can build a top-tier AI system without paying for expensive tools.
2. It revealed that the "smart" part of the system is actually a bit "dumb"—it's just matching names, not really understanding the world.
3. It warned us that if we want AI to be good at science or movies, we need to teach the "Librarian" to look beyond just names.

In short: The author built a free, open-source version of a smart AI helper, and then used an X-ray to show us that the helper is actually just a very literal name-checker, not a deep thinker. This helps us understand how to make the next generation of AI smarter.

Technical Summary

1. Problem Statement

Corrective Retrieval Augmented Generation (CRAG) is a framework designed to improve the robustness of Retrieval-Augmented Generation (RAG) systems. Unlike standard RAG, which blindly trusts retrieved documents, CRAG introduces a retrieval evaluator to assess document quality and trigger corrective actions (Correct, Incorrect, or Ambiguous).

However, the original CRAG implementation suffers from significant reproducibility barriers:

• It relies on proprietary components, specifically the Google Search API (paid) and closed-source LLaMA-2 fine-tuned weights.
• It uses deprecated OpenAI API calls.
• These dependencies prevent researchers from replicating results or building upon the work without substantial financial resources.
• Furthermore, the internal logic of the T5-based retrieval evaluator was a "black box," with no prior analysis on how it determines relevance.

2. Methodology

The authors addressed these issues through a two-pronged approach: Open-Source Reproduction and Explainability Analysis.

A. Open-Source Reproduction

The team rebuilt the CRAG pipeline using entirely free and open-source components, replacing the proprietary stack:

• Generator: Replaced the fine-tuned LLaMA-2-7B with Phi-3-mini-4k-instruct (a 3.8B parameter instruction-tuned model).
• Web Search: Replaced the Google Search API with a multi-stage Wikipedia API pipeline. This pipeline uses regex-based keyword extraction and four fallback strategies (direct lookup, typed suffix, search API, disambiguation) to retrieve context.
• Retrieval Evaluator: Retained the original fine-tuned T5-large checkpoint but applied it within the new open-source pipeline.
• Knowledge Refinement: Implemented the original "decompose-then-recompose" algorithm to split documents into sentence strips and filter them based on the evaluator's score.

B. Explainability Analysis (SHAP)

To understand the decision-making process of the T5 retrieval evaluator, the authors applied SHAP (SHapley Additive exPlanations):

• They analyzed token-level attributions for 9 representative samples (3 per action type: Correct, Incorrect, Ambiguous).
• They used a PartitionExplainer with a text masker to determine which tokens contributed positively or negatively to the relevance score.

C. Evaluation Setup

• Datasets:
  • PopQA: 1,385 long-tail entity-centric questions (open-domain).
  • ARC-Challenge: 1,172 science-based multiple-choice questions.
• Baselines: Compared against a "Vanilla RAG" (top-1 retrieval without correction) and the original CRAG results (where available).

3. Key Contributions

1. Fully Open-Source CRAG Pipeline: A complete reproduction of the CRAG system that matches the original performance using free alternatives (Wikipedia API + Phi-3-mini), removing the need for paid APIs or closed models.
2. First Explainability Analysis of CRAG: The first study to use SHAP to analyze the T5 retrieval evaluator, revealing its underlying mechanism.
3. Identification of Failure Modes: Discovery that the evaluator functions primarily as a Named Entity Alignment Detector rather than a semantic relevance judge, leading to systematic failures in specific domains (e.g., science, movies, religion).

4. Key Results

Performance Comparison

• PopQA: The open-source reproduction achieved 54.4% accuracy, closely matching the original system's 54.9% (despite using a different generator). This suggests the CRAG correction mechanism is the primary driver of performance, not the specific generator.
• ARC-Challenge: The system achieved 85.2% accuracy, outperforming Vanilla RAG (84.8%). The original LLaMA-2 based CRAG scored 53.7% on this dataset, highlighting that the Phi-3-mini generator's strong parametric knowledge of science significantly boosted performance.

Action Distribution & Analysis

• Correct Action: Achieved 78.1% accuracy on PopQA, proving the evaluator effectively identifies high-quality retrievals.
• Ambiguous Action: Without web search, accuracy was low (19.3%). Adding the Wikipedia pipeline improved this to 23.0%, confirming the necessity of external search for ambiguous cases.
• Domain Bias: On ARC-Challenge, the T5 evaluator classified 88.3% of questions as "Ambiguous." This indicates a training bias toward biographical entity questions (PopQA) and an inability to handle science questions effectively.

Explainability Findings (SHAP)

• Entity Alignment: The evaluator relies heavily on named entity alignment.
  • Correct: Entity name tokens (e.g., "Henry") contribute large positive SHAP values.
  • Incorrect: Mismatched entity tokens become strong negative drivers.
• Semantic Blindness: The model struggles with semantic relevance if entity names do not match. For example, it failed to recognize a factually correct document about "Titanic" because the subword "Titan" received a negative score, indicating the model is trained on specific entity distributions (biographies) rather than general facts.
• Religion Failure: For "Religion" questions, the evaluator frequently assigned a "Correct" action (high confidence), yet the overall accuracy was only 5.0%. This reveals that high retrieval confidence does not guarantee the presence of the specific answer entity in the text.

5. Significance and Future Directions

• Democratization of RAG Research: By proving that proprietary components can be replaced with free alternatives without sacrificing performance, this work lowers the barrier to entry for RAG research.
• Diagnostic Insight: The SHAP analysis provides critical insights for improving RAG systems. It suggests that current evaluators are brittle regarding domain transfer (e.g., from biographies to science) and entity types (e.g., movies vs. people).
• Future Work: The authors recommend:
  1. Fine-tuning the retrieval evaluator on diverse question types (beyond biographies) to improve domain generalization.
  2. Developing lightweight web search alternatives with higher coverage than Wikipedia.
  3. Conducting larger-scale statistical SHAP analysis to validate qualitative findings.

In conclusion, this paper successfully demystifies the CRAG framework, proving its efficacy through open-source tools while exposing its reliance on entity matching over true semantic understanding, thereby guiding future improvements in retrieval evaluation.