💻 computer science

RAGtio: A Modular Framework for Systematic Evaluation of Hybrid Retrieval-Augmented Generation Pipelines

This paper introduces RAGtio, a modular, open-source framework built on Haystack and Qdrant that enables systematic, reproducible evaluation of hybrid RAG retrieval pipelines in the biomedical domain through dual assessment modes and accessible interfaces for both technical and non-technical users.

Original authors: Annamaria Defilippo, Nicola Procopio, Pietro Hiram Guzzi, Pierangelo Veltri, Patrizia Vizza

Published 2026-08-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Annamaria Defilippo, Nicola Procopio, Pietro Hiram Guzzi, Pierangelo Veltri, Patrizia Vizza

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can write essays, answer questions, and tell stories. It's like a brilliant student who has read almost every book in the world. But there's a catch: this robot is a bit forgetful about recent news, and sometimes, when it doesn't know the answer, it just makes something up to sound confident. This is a problem if you need the truth, especially in serious places like hospitals or research labs. To fix this, scientists invented a trick called "Retrieval-Augmented Generation" (or RAG for short). Think of it as giving the robot a backpack full of specific textbooks. When you ask a question, the robot doesn't just guess from its memory; it first opens the backpack, finds the right pages, reads them, and then answers. This makes the robot much more reliable. But here's the tricky part: how do you know if the robot is actually finding the right pages? If the backpack is messy, or if the robot is looking for the wrong keywords, it might grab the wrong page and give you a wrong answer. Scientists need a way to test if the "search engine" part of the robot is working perfectly before they let it talk to patients or researchers.

This is exactly what the paper "RAGtio" is all about. The authors, a team from universities in Italy, built a new, open-source tool called RAGtio to act like a rigorous "test drive" for these robot search engines. They realized that while many people are building these RAG systems for medicine and biology, they often skip the hard part: systematically checking if the search is actually good. RAGtio is a modular framework, which you can think of as a giant, customizable Lego set for testing. Instead of building a new test every time, researchers can plug in different "search strategies" (like looking for exact words vs. looking for meaning) and see which one works best on their specific pile of documents.

The tool is clever because it offers two different ways to test the system. The first way, called Mode A, is like a speed run. The system uses a smart AI to automatically invent thousands of questions based on the documents it has, then checks if the search engine can find the answers. It's fast and great for seeing how the system handles a huge volume of data. The second way, Mode B, is the "expert check." Here, a human specialist writes real, tricky questions and marks exactly which pages contain the answers. This is harder to do but gives a much more honest picture of how the system would perform in the real world, where questions might be phrased in ways the documents never used.

The researchers tested RAGtio on four different medical topics: cancer (oncology), diabetes, drugs (pharmacology), and infectious diseases. They tried out four different search methods: looking for exact word matches, looking for similar meanings, a mix of both, and a mix that gets a second opinion from a "re-ranker" to polish the results. Their findings suggest that the "hybrid" approach—combining exact word matching with meaning-based searching, and then giving the top results a quick second look—is usually the strongest performer. In their tests, this method consistently found the right information early in the list of results. For example, in their cancer tests, the system found the correct answer in the top 5 results about 73% of the time when using the automated tests, and even better when checked by human experts.

However, the paper is careful not to claim this is a magic bullet that solves everything. The authors point out that their tests were done on a relatively small number of documents (about 20 papers total across the four topics) and that the human experts who checked the answers were just one person, not a whole team. They also note that while the hybrid method worked well in their specific setup, the tool is designed so that users can swap out the search engines to see what works best for their own specific library of documents. The main takeaway isn't that they found the single "best" search engine for the whole world, but that they built a transparent, easy-to-use workshop where anyone can test and compare their own search engines without needing to be a coding wizard. By making this tool open and free, they hope to help doctors and researchers build more trustworthy AI assistants that don't just sound smart, but actually get the facts right.

Technical Summary: RAGtio

Problem Statement

While Retrieval-Augmented Generation (RAG) systems are increasingly adopted in the biomedical domain to mitigate Large Language Model (LLM) limitations such as hallucinations and outdated knowledge, the evaluation of their retrieval components remains non-standardized. Existing literature predominantly focuses on end-to-end question-answering (QA) accuracy, making it difficult to isolate the specific contribution of the retrieval stage. Furthermore, retrieval-specific metrics are often applied inconsistently, and evaluation pipelines are frequently implemented on an ad hoc basis. This lack of systematic, reproducible evaluation tools hinders the ability of non-technical users (e.g., clinicians and domain researchers) to rigorously assess retrieval quality, particularly regarding the "self-referential bias" where synthetic queries may artificially inflate performance metrics.

Methodology

The paper presents RAGtio, a modular, open-source framework designed for the systematic evaluation of hybrid RAG retrieval pipelines. The system is built on the Haystack library and utilizes Qdrant as a vector database, deployed via Docker Compose to ensure reproducibility.

System Architecture

The framework orchestrates five distinct stages:

Document Ingestion: Supports heterogeneous formats (PDF, DOCX, CSV, JSON, etc.) and processes them through a Document Cleaner and a configurable Document Splitter (supporting character, recursive, and paragraph-based chunking).
Indexing: Generates both dense embeddings (via FastEmbed ONNX-compatible sentence-transformers) and sparse representations (via BM25). These are stored in Qdrant with cosine similarity indexing.
Retrieval: Implements four switchable configurations selectable via YAML without code changes:
- Sparse: BM25 lexical retrieval.
- Dense: Semantic retrieval using sentence-transformer embeddings.
- Hybrid: A weighted Reciprocal Rank Fusion (RRF) of sparse and dense results ( $w_{dense}=0.7, w_{sparse}=0.3$ ).
- Hybrid with Re-ranking: The hybrid output is further refined by a cross-encoder re-ranker.
Answer Generation: Constructs prompts for LLMs (supporting local Ollama inference or OpenAI-compatible APIs) using retrieved chunks, with instructions to remain faithful to the context.
Evaluation: Assesses retrieval quality using four standard Information Retrieval (IR) metrics: Recall@K, Hit Rate@K, Mean Reciprocal Rank (MRR), and nDCG@K.

Dual Evaluation Modes

A core methodological contribution is the implementation of two complementary evaluation modes to address bias:

Mode A (Synthetic): Automatically generates queries from randomly sampled document chunks using an LLM. This enables rapid, large-scale assessment but is susceptible to self-referential bias.
Mode B (Manual): Accepts user-provided query-relevance pairs curated by domain experts. This yields an unbiased estimate of retrieval quality on genuinely novel queries, mitigating the bias inherent in Mode A.

Key Contributions

Modular Framework: A domain-agnostic platform covering ingestion, indexing, retrieval, generation, and retrieval-focused evaluation, distinct from prior work that often ties evaluation to specific corpora or tasks.
Configurable Retrieval Strategies: The ability to compare sparse, dense, hybrid, and re-ranked hybrid strategies within a single, controlled environment using identical indices and query sets.
Complementary Evaluation Protocol: The introduction of Mode A and Mode B allows users to examine the sensitivity of retrieval performance to query-generation bias, a critical factor in specialized domains like biomedicine.
Operational Accessibility: Unlike many research prototypes, RAGtio is provided as a fully containerized, deployable software suite with a YAML-based configuration interface and a web-based GUI. This allows non-technical users to run evaluations, inspect results, and curate relevance judgments without programming expertise.

Results

The framework was evaluated across four biomedical domains: Oncology, Diabetes, Pharmacology, and Infectious Diseases.

Quantitative Performance: Using the hybrid configuration with cross-encoder re-ranking, the system demonstrated consistent effectiveness. In Mode A (synthetic), the Oncology domain achieved an MRR of 0.657 and Recall@10 of 0.732. Pharmacology showed the strongest performance with an MRR of 0.778.
Mode B Insights: Manual evaluation (Mode B) revealed that while the system effectively surfaces at least one relevant passage immediately (Hit Rate@1 of 0.800 in Oncology), recall metrics were lower. The authors attribute this to the nature of Mode B, where expert-annotated queries often have a broader set of relevant chunks than synthetic queries, making full coverage within a limited top-K window more difficult.
Interactive Q&A: In qualitative tests, queries associated with high re-ranker confidence scores generally produced correct answers. However, the study notes that high retrieval confidence does not guarantee perfect answers, and conversely, the LLM can sometimes compensate for moderately noisy retrieval if the top-ranked passage is highly relevant.

Significance and Claims

The paper positions RAGtio not as a novel retrieval algorithm, but as a systematic evaluation infrastructure. Its primary significance lies in bridging the gap between methodological rigor and practical accessibility.

Reproducibility: By providing a containerized, configuration-driven workflow, the framework enables researchers to reproduce retrieval experiments without rebuilding complex pipelines.
Bias Mitigation: The dual-mode evaluation protocol explicitly addresses the "self-referential bias" common in RAG studies, offering a more realistic assessment of retrieval quality through expert-curated benchmarks.
Usability: The web interface lowers the barrier to entry for domain experts, allowing them to validate RAG systems on their own corpora without software development skills.

The authors modestly acknowledge limitations, including the small scale of manual annotations (single annotator), the focus on a single embedding model in the case study, and the exclusive use of the hybrid configuration for the main evaluation. They frame the work as a foundational step toward more transparent and standardized RAG evaluation, particularly in specialized fields like biomedicine.