Technical Summary: TRIAD for Automated Multi-Hop RAG Evaluation

Problem Statement

The rapid adoption of Retrieval-Augmented Generation (RAG) systems in industry has created a critical need for domain-specific evaluation datasets. While existing benchmarks like HotpotQA and MuSiQue effectively challenge RAG systems on Wikipedia-based knowledge, they cannot be directly transferred to proprietary, domain-specific settings where the underlying knowledge base is not publicly available.

Current manual creation of such datasets is labor-intensive and costly, requiring expert annotators. Conversely, existing automated generation frameworks often focus on simple single-hop queries or require extensive preprocessing, such as constructing knowledge graphs or pre-labeling documents. Furthermore, many approaches fail to include unanswerable questions, which are essential for evaluating whether a system hallucinates or correctly abstains when context is insufficient. There is a gap in tools that can automatically generate high-quality, multi-hop QA datasets with labeled contexts directly from a RAG system's existing vector database without complex preprocessing.

Methodology: The TRIAD Approach

The paper introduces TRIAD, a three-stage automated pipeline designed to generate multi-hop QA datasets with relevance-labeled context documents and unanswerable questions. The process leverages the vector database already utilized by the target RAG system.

1. Generation

This stage constructs QA pairs based on the source corpus:

• Document Selection: An initial document is selected (manually or randomly) from the RAG corpus.
• Retrieval & Bridging: The system queries the vector store for similar documents. An LLM is prompted to identify a "bridging topic" that connects these retrieved chunks. If no common topic exists across all chunks, the system iteratively samples new initial chunks until a valid configuration is found where documents share a bridging topic.
• Question Synthesis: Using the bridging topic, the document set, and specific type requirements, an LLM generates a question and its corresponding answer.
• Question Taxonomy: The approach defines specific multi-hop question types to trigger different retrieval behaviors:
  • Parallel Retrieval: Comparison (contrasting entities) and Intersection (filtering/aggregating entities).
  • Sequential Retrieval: Composition (where the answer to the first part is required to retrieve the next part).
  • Cross-cutting Categories: Binary (Yes/No) and Temporal (requiring temporal reasoning).
  • Unanswerable: Questions that cannot be answered from the provided corpus, designed to test hallucination.

2. Validation

A feedback loop ensures data quality through two mandatory checks performed by an LLM-as-a-judge:

• Quality Check: Assesses answer correctness, grounding in context, and whether the question requires multi-hop reasoning.
• Sufficient Context Check: Verifies that no single context document is sufficient to answer the question. If a single document suffices, the question is rejected as a single-hop query.
• Feedback Loop: If checks fail, specific feedback is sent to the generator to revise the question, which is then re-evaluated.

3. Extraction

This stage prepares the dataset for downstream evaluation:

• Context Augmentation: Non-relevant (distractor) documents are retrieved and labeled as irrelevant to simulate real-world retrieval noise.
• Difficulty Labeling: The number of relevant contexts is counted to assign difficulty levels (fewer relevant chunks = higher difficulty).
• Unanswerable Simulation: For unanswerable questions, the context list is modified to contain only non-relevant documents (or partially relevant ones) to test the generator's ability to abstain.

Key Contributions

1. Automated Pipeline: TRIAD provides a fully automated method to generate multi-hop QA datasets directly from a RAG system's existing vector store, eliminating the need for manual annotation or complex knowledge graph construction.
2. Structured Taxonomy: The framework explicitly categorizes questions by retrieval patterns (parallel vs. sequential) and includes unanswerable questions, enabling granular evaluation of both retriever and generator components.
3. Validation Mechanism: The inclusion of a strict validation loop with an LLM judge ensures that generated questions are truly multi-hop and high-quality, filtering out single-hop queries and hallucinated content.
4. Domain Adaptability: The approach is designed to be adaptable to any proprietary domain, moving evaluation beyond fixed public benchmarks.

Experimental Results

The authors evaluated TRIAD using the HotpotQA and MuSiQue corpora as source data, generating 478 and 481 valid questions respectively.

• Human Validation: Manual assessment of 200 generated questions showed high quality:
  • Answerability: ~90-93%
  • Correctness: ~92-98%
  • Multi-hop Reasoning: ~85-91%
  • Unambiguity: ~92%
  • The validator demonstrated high accuracy in its decisions: the true rejection rate (the proportion of rejected questions that were correctly identified as invalid) was 85% for HotpotQA and 90% for MuSiQue. Consequently, less than 15% of the rejected instances were actually valid questions. The overall acceptance rate for the generated datasets was 66-68%.
• RAG Performance Trends: The generated datasets were tested against seven distinct RAG setups (varying embedding models, retrieval chunk counts, and LLMs).
  • Trend Consistency: Despite the generated questions yielding higher absolute scores (easier retrieval) than static benchmarks, they exhibited similar performance trends across different RAG configurations. For instance, Google's Embedding-002 consistently outperformed others, and reducing retrieved chunks increased precision but decreased recall, mirroring results on HotpotQA and MuSiQue.
  • Question Types: Intersection questions achieved the highest correctness, while comparison questions scored highest on faithfulness.
• Unanswerable Questions: Models were tested on unanswerable queries with only non-relevant contexts. Gemini-2.5-Flash-Lite and Gemma-4-31B-It correctly identified unanswerable cases >99% of the time, while GPT-5-nano achieved 94%.

Significance and Claims

The paper claims that TRIAD serves as a foundational validation for a pipeline capable of generating reliable evaluation datasets for domain-specific RAG systems.

• Practical Utility: The approach addresses the industrial bottleneck of creating evaluation datasets for proprietary data, offering a "directly deployable evaluation setup" without the cost of manual annotation.
• Comparative Validity: While the authors acknowledge that the generated datasets are "weaker" (easier) than human-created benchmarks like HotpotQA, they demonstrate that the datasets preserve meaningful performance distinctions between different RAG systems. This suggests the approach is viable for comparative evaluation in specific domains.
• Limitations: The authors modestly note that results are limited to the specific models and prompts used, and the non-deterministic nature of LLMs affects full reproducibility. They also highlight that the method was tested on relatively clean, general-knowledge data, and future work is needed to validate its performance on noisy, industry-specific corpora.

In conclusion, TRIAD offers a scalable, automated solution for generating multi-hop evaluation data, enabling organizations to assess RAG systems on their own proprietary knowledge bases while maintaining the ability to compare system configurations effectively.