💬 NLP

MiRAGE: A Multiagent Framework for Generating Multimodal Multihop Question-Answer Dataset for RAG Evaluation

MiRAGE is a multiagent framework that generates verified, domain-specific, multimodal, and multi-hop question-answer datasets to address the lack of specialized benchmarks for evaluating Retrieval-Augmented Generation systems in high-stakes enterprise applications.

Original authors: Chandan Kumar Sahu, Premith Kumar Chilukuri, Matthew Hetrich

Published 2026-01-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chandan Kumar Sahu, Premith Kumar Chilukuri, Matthew Hetrich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but slightly naive, robot how to read a complex manual for a nuclear power plant. The manual isn't just words; it's full of charts, 3D diagrams, and tables. If you just ask the robot simple questions like "What is the pressure?" it might guess or make things up because it hasn't learned how to connect the dots between a picture on page 10 and a sentence on page 50.

This is the problem MiRAGE solves.

The Problem: The "One-Step" Robot

Current AI systems (called RAG, or Retrieval-Augmented Generation) are like students who are great at finding a single sentence in a book to answer a question. But in the real world—like in finance, medicine, or engineering—answers are rarely in one spot. They are scattered. You might need to look at a graph, read a regulation, and check a table all at once to get the right answer.

Existing tests for these AI systems are too easy. They use simple questions from general news or Wikipedia. They don't test if the AI can handle the messy, multi-picture, multi-step puzzles found in real corporate documents.

The Solution: A Team of Specialized Experts (MiRAGE)

The authors created MiRAGE, which stands for a Multiagent framework for RAG Evaluation. Instead of asking one AI to do everything, MiRAGE acts like a construction crew or a newsroom where different specialists work together to build a perfect test.

Here is how their "swarm" of AI agents works, using a simple analogy:

The Librarian (Data Ingestion):
Imagine a messy library where books are mixed with blueprints and charts. The Librarian agent doesn't just scan the text; it looks at the pictures and tables, describes them in words, and organizes everything into neat, logical piles. It makes sure the connection between a chart and its caption isn't lost.
The Detective (Context Building):
This agent is the "multi-hop" master. If you ask a question, the Detective doesn't just grab the first page that looks relevant. It starts with one clue, realizes it's missing information, and goes hunting for more clues. It keeps digging until it has gathered all the scattered pieces of evidence needed to solve the puzzle. It builds a "semantic context window," which is just a fancy way of saying it assembles the full story before answering.
The Expert (Persona Injection):
The system knows it's dealing with a specific field, like Finance or Biology. It puts on a "hat" (a persona) of a senior expert in that field. This ensures the questions it generates sound like they came from a real professional, not a generic robot. It asks deep, deductive questions that require real understanding.
The Fact-Checker (Adversarial Verifier):
This is the most critical part. Once the team generates a question and an answer, the Fact-Checker steps in. It acts like a skeptical editor. It looks at the answer and says, "Wait, does the document actually say this?" If the AI made up a number or connected two unrelated facts, the Fact-Checker throws the answer in the trash. This prevents "hallucinations" (making things up).
The Editor (Refinement):
Finally, an Editor agent looks at all the generated questions to make sure they aren't all the same. It removes duplicates and ensures the final test covers a wide variety of topics, just like a real exam.

What They Found

The team tested this framework on four very different types of documents:

Finance: Annual reports full of complex tables.
Regulations: Strict government rules with heavy logic.
Science: Biology papers with 3D molecular models.
Journalism: Newspaper opinion pieces.

The Results:

Harder Questions: The questions MiRAGE created were significantly harder. On average, answering them required connecting more than 2.3 different pieces of information (hops), whereas standard tests usually only require 1.
Truthful Answers: Because of the Fact-Checker, the answers were highly accurate and grounded in the actual documents.
The Visual Gap: The system is great at text, but it still struggles a bit with pure visual reasoning. The authors found that if they gave the AI a text description of an image, it worked well. But if the AI had to "see" the image directly without a description, it sometimes got confused. They call this a "frontier" that still needs work.

The Bottom Line

MiRAGE is a tool that automatically builds "Gold Standard" exams for AI systems. Instead of using easy, generic questions, it creates tough, realistic tests based on a company's own messy documents. This helps companies know if their AI is actually smart enough to handle high-stakes tasks, or if it's just guessing.

In short: MiRAGE is a team of AI specialists that builds a difficult, multi-step puzzle, solves it, checks the work, and then uses that puzzle to test if other AIs are truly ready for the real world.

Technical Summary: MiRAGE

Problem Statement

The rapid deployment of Retrieval-Augmented Generation (RAG) systems in high-stakes enterprise domains (e.g., medical diagnosis, wind energy, finance) has outpaced the development of suitable evaluation benchmarks. Existing datasets, such as Natural Questions or MS MARCO, rely on general-domain corpora and purely textual retrieval, failing to capture the complexity of specialized technical documents. Real-world knowledge in these domains is often inextricably multimodal (locked in charts, diagrams, and complex layouts) and requires multi-hop reasoning to synthesize disjoint pieces of evidence.

Current synthetic dataset generation methods (e.g., DataMorgana, SMMQG) typically employ linear strategies lacking robust feedback mechanisms, often resulting in hallucinated datasets that fail to address domain specificity, multimodality, and complex reasoning simultaneously. There is a critical need for gold-standard evaluation datasets that reflect the latent thematic structure of proprietary corpora to rigorously benchmark next-generation information retrieval systems.

Methodology: The MiRAGE Framework

MiRAGE (Multiagent Framework for RAG systems Evaluation) is a model-agnostic, multi-agent framework designed to automate the generation of verified, domain-specific, multimodal, and multi-hop Question-Answer (QA) datasets. The framework orchestrates a collaborative swarm of specialized agents through five distinct phases:

Multimodal Data Ingestion and Semantic Chunking:
- A hybrid parsing pipeline utilizes a document layout analysis engine and a Vision Language Model (VLM) "Description Agent" to generate dense textual descriptions for visual elements (tables, charts).
- A semantic chunking agent segments documents using a sliding window with an optimization function that minimizes semantic discontinuity while preventing over-fragmentation of tables and text blocks.
Identification of Expert Persona and Domain:
- An agent performs global corpus analysis using manifold learning and density-based clustering to discover latent thematic structures.
- It synthesizes a specific Expert Persona ( $P_C$ ) and Domain ( $D_C$ ) (e.g., "Financial Reporting Analyst") to condition subsequent generation agents, ensuring the output mimics expert cognitive workflows rather than generic keyword matching.
Semantic Multihop Context Building:
- A recursive retrieval loop expands an initial seed chunk into a complete semantic context ( $C_s$ ).
- A multimodal agent analyzes the current context to detect missing information, generating new search queries.
- A hybrid retrieval strategy (embedding + multimodal reranking) fetches candidates, which are verified by an agent to ensure they strictly add utility and prevent context drift. This process continues until the context is deemed complete.
Agentic QA Generation and Verification:
- A QA Generation Agent formulates complex, multi-hop questions based on the expanded context, the expert persona, and the domain.
- An Adversarial Verifier Agent fact-checks the generated answers against the source context. It evaluates Correctness (factual support) and Necessity (whether the question requires the specific context to be answered), filtering out hallucinations.
Refinement and Deduplication:
- A hierarchical clustering process groups QA pairs by semantic themes and answer redundancy.
- A refinement agent synthesizes unified QA pairs from high-similarity clusters to ensure diversity and maximize semantic coverage while minimizing redundancy.

Key Contributions

Model-Agnostic Multi-Agent Framework: MiRAGE automates the ingestion and semantic segmentation of complex multimodal documents, preserving semantic dependencies between text and visual elements.
Novel Generative Methodology: The framework utilizes recursive context expansion and persona injection to produce multi-hop QA pairs that require logical deduction across disjoint chunks, surpassing the complexity of extractive QA.
Empirical Validation: Extensive evaluation across four distinct domains (Regulations, Finance, Quantitative Biology, and Journalism) demonstrates that MiRAGE generates datasets with significantly higher reasoning complexity (average hops > 2.3) and factual faithfulness.
Ablation Insights: The study reveals that domain/persona injection, multihop context building, and QA verification are crucial for dataset quality. It further notes that while visual grounding remains a frontier, MiRAGE can be powered by LLMs if textual descriptions of images are available.

Experimental Results

The authors evaluated MiRAGE on 1,000 QA pairs generated from each of four corpora: S&P Global Annual Reports (Finance), UNECE GTRs (Regulation), Quantitative Biology papers (Science), and NYTimes Opinions (Journalism).

Reasoning Complexity: MiRAGE consistently generated multi-hop questions. The average hop count exceeded 2.3 for Finance, Regulation, and Science domains, peaking at 2.84 (Finance, Gemini 2.5 Flash). The Journalism domain showed lower complexity (~1.2), attributed to the less connected nature of the corpus.
Faithfulness and Relevance: The adversarial verifier proved effective, with faithfulness scores consistently above 0.91 for three of the four domains and relevance scores exceeding 0.81 across all experiments.
Visual Grounding: This remains a challenge. Visual grounding scores were moderate (max 0.45), indicating that while models can describe images, generating complex questions requiring precise reasoning about specific visual elements is difficult. The authors hypothesize that generated descriptions may partially render visual elements redundant, lowering grounding scores.
Ablation Findings:
- Removing the Multihop Context step caused a collapse in difficulty scores (0.85 $\to$ 0.61) and a massive increase in Jensen-Shannon divergence (4.35), indicating a loss of domain alignment.
- Removing the Verifier Agent caused the most significant drop in faithfulness (0.97 $\to$ 0.74), confirming its role in mitigating hallucinations.
- Removing Domain/Persona Injection reduced question difficulty (0.85 $\to$ 0.52), showing that without these conditions, models default to generic reasoning.
- Data Representation: The "Description Only" configuration performed comparably to the full multimodal setup, suggesting LLMs can effectively power the framework if image descriptions are available.

Significance and Claims

The paper claims that MiRAGE provides the necessary infrastructure to rigorously benchmark the next generation of information retrieval systems. By automating the creation of gold-standard evaluation datasets that reflect the latent thematic structure of proprietary corpora, MiRAGE addresses the critical gap in evaluating RAG systems for high-stakes, multimodal, and multi-hop scenarios.

The authors position MiRAGE as a solution that transcends the limitations of linear synthetic pipelines. By orchestrating a swarm of specialized agents, the framework produces datasets that exhibit deep deductive reasoning and strict factual adherence. While acknowledging that visual grounding remains a frontier and that the multi-agent architecture is computationally intensive, the paper asserts that MiRAGE establishes a robust standard for automated dataset generation, enabling organizations to stress-test RAG systems against the specific complexities of their own data.