HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions

1. Problem Statement

Multi-Hop Question Answering (MHQA) is a critical benchmark for evaluating the ability of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems to integrate information from diverse sources. However, constructing high-quality MHQA datasets faces two major bottlenecks:

1. Cost and Scalability: Manual annotation is expensive and struggles to cover diverse reasoning paths at scale.
2. Quality of Synthetic Data: Existing automated synthesis methods often produce "pseudo multi-hop" questions (answerable via a single document) or rely on extensive human guidance and structured knowledge bases (e.g., Knowledge Graphs), limiting their applicability to raw, unstructured corpora.

Current methods fail to automatically identify bridge entities across complementary contexts or retrieve truly complementary documents rather than redundant ones.

2. Methodology: The HopWeaver Framework

HopWeaver is the first fully automatic, cross-document framework designed to synthesize authentic multi-hop questions from raw corpora without human intervention. It focuses on two primary question types: Bridge Questions (connecting facts via an intermediate entity) and Comparison Questions (contrasting attributes between entities).

A. Bridge Question Synthesis Pipeline

1. Bridge Entity Identification: An LLM samples a source document ($d_s$), identifies a concentrated text segment, and extracts a "bridge entity" ($e_b$) that links disparate information. It generates an optimized query ($q$) to find complementary documents.
2. Two-Stage Coarse-to-Fine Retrieval:
   • Coarse Retrieval: Uses a modified Maximum Marginal Relevance (MMR) algorithm to select candidate documents. This balances query relevance against similarity to the source document and diversity among selected candidates to ensure the retrieved documents are complementary, not redundant.
   • Fine-Grained Reranking: A fine-tuned reranker (trained via contrastive learning on simulated success/failure signals from the synthesis process) re-scores candidates to identify the most authentic complementary document ($d_t$).
3. Question Construction:
   • Generates two sequential sub-questions: one from $d_s$ (answer is $e_b$) and one from $d_t$ (uses $e_b$ to find the final answer).
   • Fuses these into a single multi-hop question that implicitly guides reasoning from $d_s \to e_b \to d_t$ without explicitly revealing the bridge entity.
4. Validation & Iteration: The system validates the reasoning path. If the question fails constraints (e.g., "No-Shortcut" rule where a single doc answers it), it rejects the document and tries the next candidate.
5. Polishing: A dedicated module refines the question for fluency, clarity, and strict adherence to multi-hop constraints.

B. Comparison Question Synthesis Pipeline

1. Entity & Attribute Identification: Extracts a primary entity and 3-5 factual attributes (e.g., dates, numbers) suitable for comparison.
2. Filtering: Scores entities and attributes on a 1-5 scale for "concreteness" and "comparability" to ensure high-quality candidates.
3. Retrieval Strategy: The LLM chooses between Direct Recommendation (recalling a specific comparable entity) or Diversified Search (generating diverse queries to find comparable entities in the corpus).
4. Construction: Identifies a comparable entity and attribute pair across documents, generating a question that requires comparing values from distinct sources.

C. Evaluation System

HopWeaver introduces a comprehensive three-dimensional evaluation system:

1. LLM-as-Judge: Uses a diverse ensemble of LLMs (including GPT-4o, Gemma-3, etc.) with a strict scoring framework prioritizing self-consistency over human alignment to assess quality, logical sophistication, and multi-hop necessity.
2. Answerability & Difficulty: Measures performance gap between "Question-Only" (Q-Only) and "Question + Golden Docs" (Q+Docs) settings. A significant gap confirms the question requires external evidence and is not solvable by internal knowledge alone.
3. Evidence-Accessibility: Evaluates retrieval metrics (Recall@K, MAP, NDCG, Support F1) to ensure evidence is retrievable but that retrieving the complete set of evidence remains challenging, validating the multi-hop nature.

3. Key Contributions

1. First Fully Automatic Framework: HopWeaver is the first system to synthesize authentic multi-hop questions from raw corpora without manual intervention or reliance on structured knowledge graphs.
2. Novel Retrieval & Reranking: It introduces a two-stage retrieval mechanism with a simulated feedback fine-tuned reranker, effectively distinguishing complementary documents from redundant ones.
3. Comprehensive Evaluation: It proposes a rigorous, multi-dimensional evaluation system that validates the authenticity of multi-hop reasoning, outperforming traditional human-annotated benchmarks in several metrics.
4. Scalable Benchmark Generation: It demonstrates the ability to generate challenging datasets for specialized domains where annotated data is scarce.

4. Experimental Results

The framework was evaluated on the English Wikipedia corpus and compared against three human-annotated benchmarks: HotpotQA, 2WikiMultiHopQA, and MuSiQue.

• Quality: HopWeaver-synthesized questions achieved 96.4%–98.9% multi-hop validity rates and average quality scores (4.09–4.45/5.0) that matched or exceeded human-annotated datasets.
• Reasoning Authenticity: Manual evaluation confirmed that 92% of the reasoning paths were correct.
• Difficulty: LLMs showed a massive performance drop in "Q-Only" settings compared to "Q+Docs" (e.g., GPT-4o EM dropped from 94% to 69% on Comparison questions), proving the questions cannot be solved without evidence integration.
• RAG System Stress Test: When tested on advanced RAG systems (e.g., IRCoT, FLARE), HopWeaver datasets exposed significant limitations. Some advanced methods performed worse on HopWeaver data than on standard benchmarks, indicating that current RAG systems overfit to specific reasoning patterns in existing datasets and fail to generalize to the diverse structures HopWeaver generates.
• Efficiency: The fine-tuned reranker improved the success rate of finding valid question pairs by ~5% and reduced the average number of retrieval attempts by ~26% compared to zero-shot reranking.

5. Significance and Impact

• Cost-Effective Data Creation: HopWeaver offers a scalable, low-cost alternative to manual annotation, enabling the creation of massive, high-quality MHQA datasets for any domain with a raw text corpus.
• Robust Evaluation: It provides a more rigorous benchmark for RAG systems, revealing that current "state-of-the-art" models may not be robust to diverse multi-hop reasoning patterns.
• Domain Adaptability: The framework is domain-agnostic, making it particularly valuable for specialized fields (e.g., medicine, law) where high-quality multi-hop datasets are currently non-existent.
• Future Research: By automating the generation of challenging reasoning paths, it opens new avenues for training models to improve their cross-document reasoning capabilities and for developing more generalizable retrieval strategies.

In summary, HopWeaver bridges the gap between the need for complex reasoning benchmarks and the limitations of manual data creation, providing a powerful tool to advance the evaluation and training of next-generation AI reasoning systems.