💬 NLP

Zero-RAG: Towards Retrieval-Augmented Generation with Zero Redundant Knowledge

Zero-RAG addresses knowledge redundancy between Large Language Models and external corpora by introducing a Mastery-Score metric to prune redundant documents and a Query Router with Noise-Tolerant Tuning to enhance internal knowledge utilization, thereby accelerating retrieval by 22% while maintaining performance.

Original authors: Qi Luo, Xiaonan Li, Junqi Dai, Shuang Cheng, Xipeng Qiu

Published 2026-01-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qi Luo, Xiaonan Li, Junqi Dai, Shuang Cheng, Xipeng Qiu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, well-read librarian (the Large Language Model or LLM) who has memorized a massive library of facts inside their own head. You also have a giant, dusty warehouse of books (the External Corpus, like Wikipedia) that you use to help the librarian answer questions.

For a long time, the standard way to work was: "No matter what the question is, go grab a few books from the warehouse, hand them to the librarian, and let them read them before answering."

The paper "Zero-RAG" argues that this old way is actually inefficient and sometimes even counterproductive. Here is the simple breakdown of why and how they fixed it:

The Problem: The "Redundant" Warehouse

The authors noticed that modern librarians (LLMs) have become so smart that they already know a huge chunk of what's in the warehouse.

The Waste: If the librarian already knows the answer perfectly, handing them a book about it is just a waste of time. It's like asking a chef who knows how to boil an egg to read a recipe book before doing it.
The Distraction: Worse, sometimes reading that extra book actually confuses the librarian. The paper found that when the librarian was given information they already knew, they sometimes got distracted and gave a worse answer than if they had just relied on their own memory.
The Cost: Indexing and searching through a massive warehouse takes a lot of energy and time. If 30% of that warehouse is just stuff the librarian already knows, you are wasting a lot of effort searching through "junk" that isn't needed.

The Solution: Zero-RAG

The team proposed a new system called Zero-RAG (Zero Redundant Knowledge). Think of it as a three-step upgrade to the library system:

1. The "Mastery Score" (The Audit)

First, they needed a way to figure out which books in the warehouse were already in the librarian's head.

How it works: They created a test called the Mastery-Score. They take a sentence from the warehouse, turn it into a quiz, and ask the librarian to answer it without looking at the book.
The Result: If the librarian gets 100% of the quiz right, that sentence gets a high "Mastery Score." It means the librarian has already "mastered" this fact.
The Action: They use this score to prune (throw away) those redundant sentences from the warehouse. They removed about 30% of the Wikipedia corpus this way, leaving only the "new" or "hard" stuff the librarian doesn't know yet.

2. The "Query Router" (The Smart Receptionist)

Now that the warehouse is smaller, they added a smart receptionist at the front desk.

How it works: When a question comes in, the receptionist asks: "Does the librarian already know this?"
The Action:
- If Yes: The receptionist says, "Skip the warehouse! Just answer from memory." This saves time and avoids the librarian getting distracted by irrelevant books.
- If No: The receptionist sends the librarian to the (now smaller) warehouse to find the specific answer.

3. "Noise-Tolerant Tuning" (The Training)

Even with a smaller warehouse and a smart receptionist, sometimes the librarian might still accidentally grab a book that isn't helpful.

The Fix: The authors trained the librarian to be "noise-tolerant." They practiced scenarios where the librarian was given a mix of helpful books and useless, distracting books.
The Goal: This taught the librarian to ignore the "noise" and stick to their own internal knowledge if the books didn't help, making the system more robust.

The Results

By using this system, the team achieved some impressive numbers:

Smaller Warehouse: They cut the size of the Wikipedia database by 30%.
Faster Speed: Because the warehouse is smaller and they skip the search for known questions, the retrieval process became 22% faster.
No Loss in Quality: Surprisingly, the librarian didn't get worse at answering questions. In fact, by removing the distracting "redundant" books, the system performed just as well as before, and sometimes better.

Summary Analogy

Imagine you are taking a test.

Old Way: You are allowed to bring a giant encyclopedia. You have to flip through pages for every single question, even the ones you know by heart. Sometimes, reading the encyclopedia makes you second-guess your own memory.
Zero-RAG Way: You first check which questions you already know. For those, you write the answer immediately. For the ones you don't know, you only look up the specific pages you need in a smaller, trimmed-down encyclopedia. You also practice ignoring pages that don't make sense.

The result? You finish the test faster, you make fewer mistakes, and you don't waste time carrying around a book you don't need.

Technical Summary: Zero-RAG

Problem Statement

Retrieval-Augmented Generation (RAG) is widely used to mitigate Large Language Model (LLM) hallucinations by supplementing internal knowledge with external corpora. However, as LLMs evolve, their internal knowledge bases expand significantly, leading to substantial knowledge redundancy between the model's parameters and the external corpus (e.g., Wikipedia).

This redundancy presents two critical challenges:

Computational Overhead: Dense retrieval indexing and search workloads scale with corpus size. Redundant knowledge unnecessarily increases the computational cost of the retrieval stage.
Performance Degradation: Contrary to the assumption that more context is always better, exploratory analysis reveals that redundant knowledge can actively hinder LLM performance. When an LLM is provided with documents containing facts it has already mastered, the irrelevant context distracts the model, causing a significant drop in accuracy (approximately 20 points in some experiments) compared to answering without retrieval.

Methodology: Zero-RAG

To address these issues, the authors propose Zero-RAG, a framework designed to prune redundant knowledge from the RAG corpus while maintaining or improving retrieval-augmented performance. The system operates through four integrated components:

1. Mastery-Score and Corpus Pruning

The core innovation is the Mastery-Score, a metric designed to quantify how well an LLM has "mastered" a specific passage, distinguishing between mere memorization and the ability to flexibly utilize knowledge after Supervised Fine-Tuning (SFT).

Calculation: For a given sentence, the system generates $n$ question-answer pairs. It then evaluates the LLM's ability to answer these questions using only its internal knowledge. The Mastery-Score is the average Exact Match (EM) score across these pairs.
Scalability: Since calculating Mastery-Scores for every sentence in a large corpus is computationally prohibitive, a Mastery-Classifier (a regression model) is trained to predict these scores.
Pruning: Sentences with high Mastery-Scores (indicating the LLM already knows the content) are pruned from the corpus, resulting in a concise, non-redundant subset ( $D_{retained}$ ).

2. Query Router

Even with a pruned corpus, some queries may not require external retrieval if the LLM can answer them confidently from internal knowledge.

Function: A binary classifier determines whether a specific query is "mastered" by the model.
Action: If the query is mastered, the router bypasses the retrieval step entirely, allowing the LLM to answer directly. This avoids the latency and noise associated with retrieving irrelevant documents for known facts.

3. Noise-Tolerant Tuning

To ensure robustness when retrieval does occur (either for unmastered queries or due to retrieval errors), the LLM undergoes Noise-Tolerant Tuning.

Training Strategy: The model is fine-tuned using three types of data samples:
1. Queries with relevant documents.
2. Queries with distracting (noise) documents.
3. Queries without any RAG context.
Objective: The loss function encourages the model to rely on internal knowledge when documents are irrelevant and to effectively utilize documents when they are relevant, thereby suppressing the negative impact of "noisy" retrieval results.

4. Inference Pipeline

The final inference process integrates these modules:

Pruning: The raw corpus is filtered using the Mastery-Score predictor.
Routing: For each query, the Query Router decides whether to retrieve.
Generation: If retrieval is skipped, the model answers internally. If retrieval is triggered, the model searches the pruned corpus and generates the answer using the Noise-Tolerant Tuned weights.

Key Contributions

First RAG-Oriented Pruning: To the authors' knowledge, Zero-RAG is the first work to explore corpus pruning specifically by removing knowledge redundancy between the corpus and the LLM, rather than just optimizing retrieval efficiency.
Novel Metrics and Modules: The introduction of the Mastery-Score to identify redundant knowledge and the Query Router to dynamically bypass retrieval for mastered questions.
Robustness Mechanism: The Noise-Tolerant Tuning strategy ensures the model remains effective even when the pruned corpus fails to retrieve the perfect document or when irrelevant documents are retrieved.

Experimental Results

The authors evaluated Zero-RAG on four fact-intensive QA datasets (HotpotQA, TriviaQA, EntityQuestions, and PopQA) using Llama3-70B and Llama3.3-70B models.

Corpus Reduction: Zero-RAG successfully pruned 30% of the Wikipedia corpus.
Efficiency Gains: The pruning resulted in a 22% acceleration in the retrieval stage (reducing average latency from ~12.28s to ~9.66s).
Performance Retention: Despite the significant reduction in corpus size, RAG performance was retained.
- On TriviaQA, removing 70% of the corpus resulted in a negligible performance drop (0.62 points).
- On HotpotQA and EntityQuestions, the performance drop at a 30% pruning ratio was minimal (less than 2 points on average).
Comparison with Baselines:
- Standard RAG (without pruning) often showed slight performance drops compared to the model answering alone on certain datasets, confirming the "distractor" effect of redundant knowledge.
- Zero-RAG with Noise-Tolerant Tuning consistently outperformed baselines, demonstrating that the proposed tuning effectively mitigates the risks of retrieval noise.
Model Size Independence: The approach was also effective on smaller models (Llama3-8B), indicating that knowledge redundancy is a pervasive issue across model scales.

Significance and Claims

The paper claims that Zero-RAG offers a paradigm shift in RAG design by treating the external corpus not just as a knowledge supplement, but as a resource that must be curated to match the LLM's current internal state.

Efficiency without Sacrifice: The primary significance is the demonstration that a smaller, curated corpus can yield faster retrieval speeds without compromising (and sometimes improving) answer accuracy.
Mitigating the "Distractor" Effect: The work highlights that uncurated RAG can be counterproductive, and that filtering out "mastered" knowledge is essential for optimal performance.
Future Direction: The authors aim to inspire further research into the design choices for pruning RAG corpora, providing a codebase, model, and pruned corpus to facilitate reproducibility and further improvement.

The authors acknowledge limitations, noting that their study is currently focused on Wiki-based databases and that the effectiveness on domain-specific or multimodal datasets remains to be explored. They also note that the system's performance is dependent on the quality of the initial dataset.