💻 computer science

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

StratRAG is a new open-source dataset designed to benchmark the multi-hop reasoning capabilities of Retrieval-Augmented Generation (RAG) systems by evaluating retrieval strategies against complex questions embedded within noisy, distractor-filled document pools.

Original authors: Aryan Patodiya

Published 2026-04-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aryan Patodiya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are playing a high-stakes game of "Scavenger Hunt Detective."

To solve a mystery, you don't just need one clue; you need to find two specific clues that are hidden in different places, and then use them together to solve the puzzle. For example: "Who is the director of the movie that stars the actor born in the same city as the inventor of the lightbulb?"

To solve this, you first have to find the city of the inventor, then find the actor, and finally find the director. This is called "Multi-Hop Reasoning"—you have to "hop" from one piece of information to the next.

The Problem: The "Messy Library"

Right now, scientists are building AI systems (like ChatGPT) that use a technique called RAG (Retrieval-Augmented Generation). Think of RAG as giving an AI a library card so it can look up facts instead of just guessing.

However, testing these AI "librarians" is hard. Most tests are like asking, "Can you find a book about cats?" That's easy. But most real-world questions are like the scavenger hunt above. Current tests are often messy, disorganized, or don't tell you exactly where the "gold" clues are hidden, making it impossible to tell if the AI is actually a good detective or just getting lucky.

The Solution: StratRAG (The Ultimate Training Ground)

The author, Aryan Patodiya, created StratRAG. Think of StratRAG as a perfectly controlled obstacle course for AI detectives.

Instead of a messy library, StratRAG gives the AI a small, specific box containing exactly 15 envelopes.

2 envelopes contain the "Gold Clues" (the truth).
13 envelopes contain "Distractors" (information that looks important but is actually a dead end).

Because the author knows exactly which envelopes hold the truth, he can give the AI a "grade" on how good its searching skills are, without even worrying about whether the AI can write a good final answer. He is testing the searching (the retrieval), not the writing (the generation).

The Results: Who is the best detective?

The author tested three different "searching styles":

The Keyword Searcher (BM25): Like a person looking only for specific words. If the clue says "Automobile," they won't find a clue that says "Car."
The Vibe Searcher (Dense Retrieval): Like a person looking for the meaning or "vibe" of a sentence. They understand that "Car" and "Automobile" are the same thing.
The Hybrid Detective (The Winner): A person who uses both specific keywords and general "vibes." This detective was the most successful.

The "Boss Level": The Bridge Question

Even the best Hybrid Detective struggled with one specific type of puzzle: The Bridge Question.

In a Bridge Question, the two clues are connected by a "bridge" (a hidden link) that isn't mentioned in the question. It’s like being told, "Find the person who lives in the house built by the man who invented the telephone." The question doesn't mention the house or the man; it only mentions the telephone.

The AI gets stuck because it's looking for "telephone" clues, but it doesn't realize it needs to find the "man" first to cross the bridge to the "house."

What’s Next?

The author suggests that instead of just teaching AI to look for words or vibes, we should teach it using Reinforcement Learning—essentially, teaching it through "trial and error," much like training a dog with treats. If the AI finds the right clues and solves the mystery, it gets a "treat" (a mathematical reward). Eventually, the AI will learn the secret logic of how to "hop" across those bridges to solve the toughest mysteries.

Technical Summary: STRATRAG: A Multi-Hop Retrieval Evaluation Dataset for RAG Systems

1. Problem Statement

Retrieval-Augmented Generation (RAG) systems rely on a two-stage process: retrieving relevant documents and generating answers based on them. While much research focuses on the quality of the generation stage, evaluating the retrieval stage—specifically its ability to handle complex, multi-hop reasoning under noisy, realistic conditions—is underexplored.

Existing multi-hop benchmarks (like HotpotQA or MuSiQue) are designed for end-to-end Question Answering (QA) rather than isolated retrieval evaluation. They lack:

Fixed-size document pools: Making it difficult to standardize retrieval difficulty.
Controlled distractor sets: Making it hard to measure precision against topically related noise.
Guaranteed gold-document positions: Preventing direct, metric-driven benchmarking of the retriever alone.

2. Methodology

The author introduces StratRAG, a dataset restructured from HotpotQA to serve as a dedicated retrieval benchmark.

Dataset Construction:
- Source: Derived from HotpotQA (distractor setting).
- Structure: Each of the 2,200 examples consists of a query and a fixed pool of 15 documents (2 gold documents and 13 topically related distractors).
- Question Categorization: Questions are classified into three types: Bridge (complex reasoning where the connection is implicit), Comparison (contains comparative keywords), and Yes-No.
- Preprocessing: The author implemented a pipeline to identify gold documents via Unicode normalization, filter out empty/useless paragraphs, and ensure a controlled environment for evaluation.
Benchmarking Framework:
The author evaluates three retrieval strategies on the validation set:
1. BM25: A sparse, lexical retrieval baseline.
2. Dense Retrieval: Using the all-MiniLM-L6-v2 bi-encoder for semantic similarity.
3. Hybrid Fusion: A combination of BM25 and Dense scores using min-max normalization and equal weighting ( $\alpha = 0.5$ ).
Metrics: Performance is measured using standard Information Retrieval (IR) metrics: Recall@k (k=1, 2, 5), Mean Reciprocal Rank (MRR), and NDCG@5.

3. Key Contributions

StratRAG Dataset: A clean, structured, and open-source benchmark (available on Hugging Face) specifically designed for multi-hop retrieval evaluation.
Baseline Benchmarks: Comprehensive performance data for sparse, dense, and hybrid retrieval methods.
Error Analysis: Identification of "Bridge" questions as a specific failure mode for current retrieval technologies.
Research Roadmap: A formal hypothesis and framework for using Reinforcement Learning (RL) to optimize retrieval policies based on downstream answer correctness.

4. Results and Findings

Hybrid Superiority: Hybrid retrieval achieved the best performance across all metrics (Recall@2 = 0.70, MRR = 0.93, NDCG@5 = 0.85), proving that lexical precision and semantic coverage are complementary.
Dense vs. BM25: Dense retrieval consistently outperformed BM25, suggesting that semantic understanding is more critical than keyword matching for multi-hop tasks.
The "Bridge" Challenge: Bridge questions were significantly harder to retrieve (Recall@2 = 0.67) compared to Comparison (0.88) or Yes-No (0.83) questions. This is because the link between gold documents in bridge questions is often latent and not explicitly stated in the query.
Upper Bound Insight: The Hybrid Recall@5 of 0.905 suggests that current retrieval methods can provide the necessary evidence for ~90% of questions to a downstream generator.

5. Significance

StratRAG provides the research community with a tool to decouple retrieval performance from generation performance. By isolating the retriever, researchers can more accurately diagnose whether a RAG system fails because it cannot find the information or because it cannot reason with the information it has found. Furthermore, the paper sets the stage for a shift from "similarity-based retrieval" to "reasoning-based retrieval" via Reinforcement Learning.