Technical Summary: DocHop-QA

Problem Statement

Despite rapid advancements in Large Language Models (LLMs), current Question Answering (QA) benchmarks fail to address the core challenges of real-world scientific information seeking. Existing datasets are predominantly limited to unimodal text, short-context reasoning, and single-document settings (e.g., Wikipedia-based benchmarks like HotpotQA). They often rely on predefined hyperlinks or explicit structural anchors, which do not reflect the complexity of synthesizing fragmented evidence scattered across multiple scientific documents in heterogeneous formats (narrative text, structured tables, and visual layouts). Consequently, current models struggle with long-context demands, error accumulation across reasoning chains, and the grounding of semi-structured data, leaving a gap in evaluating true cross-document, multimodal reasoning capabilities.

Methodology

The authors introduce DocHop-QA, a large-scale benchmark comprising 11,379 instances designed to evaluate multi-hop reasoning over multimodal, multi-document scientific corpora. The dataset is constructed from publicly available PubMed Central (PMC) articles, ensuring domain relevance and realistic complexity.

Dataset Construction Pipeline

The construction follows a four-stage, LLM-driven pipeline grounded in 11 specific scientific reasoning concepts derived from human-in-the-loop analysis of PubMed articles:

1. Question Concept Proposal: 11 reasoning concepts were identified (e.g., Problem-Solution, Mechanism-Effect, Comparison) and categorized into Paragraph-Oriented (fan-hop aggregation) and Table-Oriented (chain-hop sequential inference) types.
2. Document Selection: Document pairs were selected based on topical alignment (keyword co-occurrence and semantic similarity) without relying on predefined citation links. For Table-Oriented types, citation-linked tables were utilized to model structured chain-hop reasoning.
3. Question Generation: An instruction-tuned Qwen2-Instruct model (using 4-bit LoRA) generated diverse question-answer pairs. Separate models were trained for different question categories (comparison, non-comparison, PST) to ensure semantic consistency and controllability.
4. Answer Matching: A retrieval pipeline was implemented to match questions with evidence spans.
   • Fan-hop (Paragraph-Oriented): Semantically complementary snippets are retrieved independently from multiple documents and aggregated.
   • Chain-hop (Table-Oriented): Evidence is retrieved sequentially, connecting problem/solution paragraphs with tables and referenced content across documents.

Evaluation Framework

To assess the utility of the dataset, the authors propose a task-driven evaluation framework spanning four distinct settings:

1. Generative Text Extraction: Models generate natural-language answers from text and/or image inputs.
2. Structured Generative Answering: Models generate lists of answer entity indices from labeled XML content and page images.
3. BBox Entity Index Extraction: A discriminative task predicting bounding box indices for answer spans in OCR layouts.
4. XML Entity Index Extraction: A discriminative task predicting indices directly from XML-defined entities.

Key Contributions

1. DocHop-QA Benchmark: A large-scale dataset for multi-hop QA over multimodal, multi-document scientific corpora, incorporating unstructured text, structured tables, and layout features without relying on explicit hyperlinks.
2. Scalable Generation Pipeline: A domain-agnostic, LLM-driven pipeline guided by 11 real-world scientific reasoning concepts, enabling the automated creation of diverse, challenging multi-hop QA instances without gold reasoning chains.
3. Comprehensive Evaluation: A multi-task benchmark spanning structured index prediction to generative answering, demonstrating the dataset's applicability for assessing both discriminative and generative reasoning over textual, structural, and visual information.

Experimental Results

Experiments conducted on state-of-the-art models (including InternVL2, Qwen2.5-VL, Qwen3-VL, GPT-4o, and Gemini 2.5/3) reveal significant challenges in current systems:

• Generative Performance: While models like Gemini 3 Pro and GPT-4o achieve higher BLEU and ROUGE scores with full-document ingestion, they still struggle with the complexity of synthesizing cross-document evidence. Notably, adding visual inputs (images) often degrades performance due to alignment challenges between text and layout, though full-document ingestion generally outperforms truncated text-only inputs.
• Structured Reasoning: In structured generative answering, models struggle with consistent output formatting and spatial accuracy. GPT-4o achieved the highest F1 score (16.21) in a zero-shot, text-only setting, while visual inputs generally hurt performance.
• Retrieval and Aggregation: Analysis of retrieval quality shows that while top-5 snippet recall is relatively high, complete multi-hop recall (retrieving all necessary evidence pieces simultaneously) remains low (e.g., ~0.39 for GTR-large at Top-5). This highlights the difficulty of aggregating dispersed evidence.
• Modality and Reasoning: Ablation studies confirm that textual content is the dominant modality, but visual cues contribute non-trivially. However, even with "oracle" access to gold snippets, model performance lags behind human performance, indicating that the primary bottleneck is the integration and synthesis of evidence rather than mere retrieval.

Significance and Claims

The paper positions DocHop-QA as a rigorous testbed for advancing next-generation scientific QA systems. The authors claim that the dataset:

• Captures Real-World Complexity: By integrating text, tables, and layout features across multiple documents without explicit structural anchors, it reflects the true difficulty of scientific information seeking.
• Exposes Model Limitations: Current models struggle with long-context, multi-evidence demands, particularly in synthesizing paragraph- and table-level evidence across documents.
• Facilitates Future Research: The benchmark provides a controlled environment to study evidence aggregation, cross-document synthesis, and multimodal reasoning, areas identified as fundamental yet underexplored challenges in scientific document understanding.

The authors maintain a modest stance, acknowledging limitations such as the potential for stylistic regularities in LLM-generated questions and the focus on high-level semantic interpretation rather than fine-grained numerical operations. They emphasize that DocHop-QA is designed for research purposes to improve robust QA systems, not for clinical decision-making.