💬 NLP

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

This paper introduces RARE, a redundancy-aware retrieval evaluation framework that addresses the limitations of existing QA benchmarks in highly similar corpora by decomposing documents into atomic facts and enhancing LLM-based data generation with CRRF, ultimately revealing significant robustness gaps in current retrievers through the new RedQA benchmark across financial, legal, and patent domains.

Original authors: Hanjun Cho, Jay-Yoon Lee

Published 2026-04-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hanjun Cho, Jay-Yoon Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to find a specific fact in a library.

The Old Way (Standard Benchmarks):
Most current tests for AI search engines are like a library where every book is about a completely different topic. If you ask, "Who wrote Hamlet?", the AI finds the Shakespeare book. If you ask, "What is the capital of France?", it finds the France book. There is no confusion because the books look nothing alike. The AI gets an A+ because it's easy to pick the right book.

The Real World (The Problem):
But in the real world—like in a law firm, a bank, or a patent office—the "library" is a nightmare. Imagine a room with 1,000 books about "Batteries." They all look almost identical. They all say similar things, just with slightly different words.

Book A says: "The battery lasts 10 hours."
Book B says: "Battery life is approximately 10 hours."
Book C says: "You can expect a 10-hour runtime."

If you ask an AI, "How long does the battery last?", and it grabs Book B, it is technically correct! But in old tests, the system might say, "Wrong! The 'Gold Standard' answer was in Book A." The AI gets punished for finding the right information in a different, but identical, book. This makes the AI look bad, even though it's doing a great job.

The Solution: RARE
The authors of this paper built a new framework called RARE (Redundancy-Aware Retrieval Evaluation). Think of RARE as a new way to grade the AI that understands the "messy library" reality.

Here is how RARE works, using simple analogies:

1. The "Atomic Fact" Breakdown

Instead of looking at whole books (documents), RARE uses a magic microscope to break every document down into its smallest, indivisible Lego bricks (atomic facts).

Old way: "Here is the whole paragraph about the battery."
RARE way: "Here is the specific brick that says '10 hours'."
By looking at these tiny bricks, RARE can see that Book A, Book B, and Book C all contain the exact same brick. This stops the AI from being unfairly punished for finding the right brick in a different book.

2. The "CRRF" Judge (The Panel of Experts)

When creating these tests, the authors used AI to generate questions. But AI is lazy; it often writes boring or confusing questions.
To fix this, they invented CRRF. Imagine you are hiring a chef.

The Lazy Way: You ask one judge, "Is this dish good?" and they give a vague score of 7/10.
The CRRF Way: You ask five different experts to judge the dish separately:
1. Expert A: "Is it salty enough?"
2. Expert B: "Is the meat cooked?"
3. Expert C: "Is the presentation nice?"
4. Expert D: "Is it fresh?"
5. Expert E: "Is it spicy?"
  Then, instead of averaging their scores (which can be messy), you rank them. If Expert A says "Top 1" and Expert B says "Top 1," that dish is a winner. This method is much more stable and reliable than asking one person to do everything at once.

3. The New Test: RedQA

Using RARE, they built a new test called RedQA (Redundancy QA). They tested it on three tough areas: Finance (bank reports), Legal (laws), and Patents (inventions).

The Shocking Result:
When they ran the tests, the AI's performance crashed.

On the easy "Standard Library" tests, the AI got 66% correct.
On the new "RedQA" tests (the messy, redundant libraries), the AI dropped to 5% to 27% correct.

Why?
Because in the real world, the AI gets confused by all the similar-looking books. It grabs the wrong one, or it grabs three books that say the same thing but misses the one extra piece of info needed to answer a complex question.

The Takeaway

The paper argues that we have been lying to ourselves. We thought our AI search engines were smart because they passed easy tests with distinct books. But in the real world, where information is repetitive and messy, those same engines are failing miserably.

RARE is the new ruler that finally measures how well an AI can navigate a crowded, confusing, and repetitive library without getting lost. It tells us: "Hey, your AI isn't as smart as you thought, and here is exactly how to fix it."

1. Problem Statement

Current Retrieval-Augmented Generation (RAG) benchmarks (e.g., HotpotQA, Natural Questions) rely on corpora with distinct documents and minimal overlap. However, real-world enterprise domains (Finance, Legal, Patents) exhibit high information redundancy and strong inter-document similarity.

This mismatch creates two critical evaluation failures:

Unfair Penalization: In high-redundancy settings, retrievers that surface valid alternative evidence (semantically equivalent to the "gold" passage) are penalized because standard benchmarks assume a single canonical source.
Overestimation of Robustness: Models performing well on low-overlap benchmarks often fail in real-world scenarios where near-duplicate documents confuse retrieval systems, leading to an inability to distinguish relevant evidence from redundant noise.

Existing metrics fail to account for the fact that in redundant corpora, the same fact appears across multiple documents, making standard "single-passage" evaluation invalid.

2. Methodology: The RARE Framework

The authors propose RARE (Redundancy-Aware Retrieval Evaluation), a modular framework designed to construct realistic benchmarks by decomposing documents into atomic facts and tracking redundancy. The pipeline consists of three main stages:

A. Valid Information Selection

Atomic Decomposition: Documents are decomposed into "atomic information units" (minimal, indivisible factual claims) rather than using full passages. This allows for precise tracking of facts across documents.
Validity Filtering: An LLM filters out units that are incomplete, trivial, or metadata-only.
Multi-Criteria Ranking: Remaining units are ranked based on five criteria: Validity, Completeness, Specificity, Clarity, and Questionability.

B. Systematic Redundancy Tracking

Two-Stage Detection: To identify semantically equivalent facts across different chunks:
1. Embedding Search: A high-recall candidate set is retrieved using vector similarity (threshold $\tau=0.5$ ).
2. LLM Verification: An LLM judge confirms if the candidate is factually equivalent to the target.
Gold Label Construction: The "gold evidence set" for a question includes the originating passage plus all semantically equivalent alternatives found. This ensures retrievers are credited for finding any valid source, not just the original one.

C. Question-Answer Generation

Logical Filtering: Generated multi-hop questions undergo a "zero-tolerance" filter against five criteria (e.g., Answer Exclusion to prevent answer leakage, Information Equivalence to ensure the question requires exactly the selected evidence).
CRRF (Criterion-wise Prompting with Reciprocal Rank Fusion):
- Motivation: LLMs struggle to jointly reason over multiple quality criteria, often producing unstable confidence scores.
- Mechanism: Instead of asking the LLM to rank based on all criteria at once, RARE evaluates each criterion independently via separate prompts. The results are then fused using Reciprocal Rank Fusion (RRF).
- Benefit: This approach discards unreliable confidence magnitudes and relies on ordinal preferences, significantly stabilizing the quality of generated data.

3. Key Contributions

RARE Framework: A generalizable pipeline for constructing domain-specific RAG benchmarks that explicitly quantify and incorporate redundancy and passage similarity.
CRRF Method: A novel recipe for stabilizing LLM-based data generation by decoupling multi-criterion evaluation and using rank-based fusion, improving reliability over joint prompting.
RedQA Benchmark: A new dataset covering Finance (SEC 10-K), Legal (U.S. Code), and Patent (USPTO) domains. It contrasts sharply with General-Wiki (low overlap) in terms of similarity and redundancy metrics.
Empirical Analysis: A comprehensive quantification of the performance gap between standard benchmarks and enterprise conditions.

4. Experimental Results

The authors evaluated various retrievers (Sparse BM25, Dense Embeddings, and LLM-based Embeddings) on RedQA.

Performance Degradation in High-Overlap:
- On General-Wiki (low overlap), strong retrievers achieve 66.4% PerfRecall@10 at 4-hop depth.
- On Enterprise Corpora, performance collapses:
  - Finance: Drops to 8.5%.
  - Legal: Drops to 5.0%.
  - Patent: Drops to 27.9%.
- This reveals a massive robustness gap that standard benchmarks fail to capture.
Similarity vs. Redundancy:
- Document Similarity is the primary driver of retrieval difficulty. Higher similarity leads to monotonic performance drops (confusion among near-duplicates).
- Redundancy can sometimes be beneficial, providing alternative evidence paths that mitigate errors, though it complicates the retrieval task.
Impact of Hop Depth:
- Performance degrades sharply as hop depth increases in high-overlap corpora. The "overlap-induced evidence bottleneck" causes top- $k$ retrieval to recycle the same facts rather than finding complementary evidence.
End-to-End RAG:
- Retrieval quality remains the dominant factor in end-to-end accuracy. While parametric knowledge helps, the "RAG Gain" is substantial in enterprise domains where parametric knowledge is weak.
CRRF Efficacy:
- Ablation studies show CRRF (Separate prompts + RRF) achieves the highest NDCG@3 and stability compared to joint prompting or score-based aggregation.

5. Significance

Realism: RARE shifts the evaluation paradigm from "clean" academic datasets to "messy" real-world enterprise conditions, providing a more accurate proxy for deployment robustness.
Fair Evaluation: By using redundancy-aware gold labels, RARE prevents the unfair penalization of retrievers that find valid alternative evidence, encouraging the development of systems that handle document overlap effectively.
Methodological Advance: The introduction of CRRF offers a best-practice pattern for using LLMs in data generation, addressing the instability of multi-criterion reasoning.
Practical Insight: The findings suggest that current state-of-the-art retrievers are ill-equipped for high-similarity domains, highlighting a critical area for future research in diversity-promoting retrieval and robustness against near-duplicates.

In conclusion, the paper argues that existing benchmarks overestimate RAG capabilities in enterprise settings and provides the tools (RARE, CRRF, RedQA) to build evaluations that faithfully reflect the challenges of high-redundancy corpora.