🤖 AI

SciEGQA: A Dataset for Scientific Evidence-Grounded Question Answering and Reasoning

This paper introduces SciEGQA, a novel dataset featuring both a human-annotated fine-grained benchmark and a large-scale automated training set with semantically grounded evidence regions, designed to enhance the evidence localization and scientific reasoning capabilities of Vision-Language Models in document visual question answering.

Original authors: Wenhan Yu, Zhaoxi Zhang, Wang Chen, Guanqiang Qi, Weikang Li, Lei Sha, Deguo Xia, Jizhou Huang

Published 2026-03-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenhan Yu, Zhaoxi Zhang, Wang Chen, Guanqiang Qi, Weikang Li, Lei Sha, Deguo Xia, Jizhou Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery, but instead of a crime scene, your "scene" is a massive, 50-page scientific research paper filled with dense text, complex charts, and confusing diagrams.

The Problem: The "Page-Level" Detective
Currently, most AI detectives (Vision-Language Models) are trained to look at a whole page and guess the answer. It's like being asked, "Where is the suspect?" and the AI just points to the entire city block. It might be technically correct that the suspect is somewhere in that block, but it's not very helpful. The AI might get the right answer by luck, but it doesn't actually know where the proof is. This makes it hard to trust the AI or understand how it reached its conclusion.

Other AI models are trained to look at tiny, individual words (like a single letter on a page). This is too zoomed in. It's like trying to understand a movie by looking at one single pixel on the screen. You lose the story.

The Solution: SciEGQA (The "Highlighter" Detective)
The authors of this paper introduced a new dataset called SciEGQA. Think of this as a training course for AI detectives that teaches them to use a highlighter instead of just pointing or zooming in.

In SciEGQA, when an AI answers a question, it must also draw a box around the exact paragraph, chart, or sentence that proves its answer.

The Analogy: Imagine reading a textbook. Instead of just saying "The answer is on page 10," the AI says, "The answer is in the third paragraph on page 10, specifically the sentence about 'gravity'." It highlights the exact evidence.

How They Built It

The "Gold Standard" (The Benchmark): The team hired human experts to read 80 real scientific papers and manually draw these "highlight boxes" around the evidence. They created 1,600+ high-quality questions where the answer must come from the highlighted area. This is the "final exam" for the AI.
The "Practice Gym" (The Training Set): Since humans can't read millions of papers, they built a robot pipeline to generate 30,000+ practice questions automatically. They taught the robot to find chunks of text, ask questions about them, and verify the answers. This gave the AI a massive library of practice problems.

The Three Levels of Difficulty
The dataset tests the AI on three levels of complexity, like video game levels:

Level 1 (Single Page, Single Spot): "Find the answer in this one chart." (Easy)
Level 2 (Single Page, Multiple Spots): "Combine the answer from this chart and this paragraph on the same page." (Medium)
Level 3 (Multiple Pages, Multiple Spots): "Find a number in the introduction, a graph in the middle, and a conclusion at the end, then put them together." (Hard)

What They Found
When they tested the smartest AI models available today on this new "highlighter" test, the results were surprising:

The AI is bad at finding the evidence. Even the best models struggled to draw the box around the right spot. They often got the answer right by guessing, but they couldn't point to the proof.
Context matters. When the AI was given the entire document, it got confused. But when they gave it just the specific highlighted paragraph (the "evidence region"), its accuracy skyrocketed. This proves that the AI's main weakness isn't knowing the answer; it's knowing where to look.
Training works. When they taught an AI model using their new "practice gym" (the 30k dataset), the model got much better at both finding the evidence and answering the questions.

The Big Picture
This paper is essentially saying: "We can't just ask AI to give us answers anymore. In science, we need to know why and where the answer came from."

SciEGQA is a new tool that forces AI to stop guessing and start showing its work, just like a good student who underlines the key sentences in a textbook before writing an essay. It's a crucial step toward making AI trustworthy for serious scientific research.

1. Problem Statement

Scientific documents present unique challenges for Document Visual Question Answering (DocVQA) due to their complex multimodal structures (text, tables, figures, cross-references) and dense information. Existing benchmarks suffer from three primary limitations in how they represent supporting evidence:

Doc/Page-level: Annotations are too coarse (entire pages), failing to pinpoint specific evidence regions, which limits interpretability.
Token-level: Annotations are too fine-grained (individual words), fragmenting semantically coherent units and losing contextual information necessary for reasoning.
Component-level: Focuses on isolated elements (e.g., a single chart), ignoring the holistic document context required for cross-modal reasoning.

Consequently, current Vision-Language Models (VLMs) struggle to simultaneously localize precise evidence regions and perform accurate reasoning on scientific documents. There is a lack of datasets that support semantic region-level grounding (bounding boxes around coherent semantic units) combined with multi-page reasoning.

2. Methodology

The authors introduce SciEGQA, a dataset designed to bridge the gap between coarse and fine-grained annotations.

A. Dataset Design

SciEGQA defines a sample as $S = (D, Q, A, T, E)$ , where $E$ represents a set of evidence regions annotated with bounding boxes $(x_1, y_1, x_2, y_2)$ and semantic categories (text, figure, table). The dataset supports three reasoning scenarios of increasing complexity:

SPSR (Single Page Single Region): Answer derived from one region on one page.
SPMR (Single Page Multi Regions): Answer requires integrating multiple regions on the same page.
MPMR (Multi Pages Multi Regions): Answer requires synthesizing information across multiple pages and regions.

B. Data Construction

The dataset consists of two complementary components:

Human-Annotated Benchmark (Evaluation Set):
- Source: 80 scientific papers from 8 arXiv domains (cs, econ, eess, math, physics, q-bio, q-fin, stat).
- Process: 12 annotators and 24 domain experts identified semantically complete regions. A cross-annotation strategy with Intersection-over-Union (IoU) verification (>0.9) ensured high consistency.
- Scale: 1,623 high-quality QA pairs.
Automatically Constructed Training Set:
- Source: ~3,671 papers from arXiv.
- Pipeline:
  - Segmentation: Uses SAM3 (Segment Anything Model) to segment pages visually, avoiding rigid OCR-based layouts to capture coherent semantic units.
  - Filtering: Removes noise via area thresholds and semantic filtering using strong VLMs.
  - Generation: Uses advanced VLMs (GPT-5.2, Gemini 3) to generate QA pairs conditioned on the segmented regions.
  - Verification: A "Rollout" verification step re-answers questions using the generated evidence; only pairs with correct answers are retained.
- Scale: ~30,780 QA pairs.

C. Evaluation Protocol

The authors propose a Grounding–Crop–then–Answer protocol to disentangle localization from reasoning:

Grounding: The model predicts bounding boxes for evidence regions.
Crop: The predicted regions are cropped from the original images.
Answer: A fixed, strong VLM (Qwen3.5-27B) answers the question based only on the cropped regions.
Metrics: IoU (Intersection over Union) for localization precision and Answer Accuracy for reasoning capability.

3. Key Contributions

SciEGQA Dataset: The first scientific DocVQA dataset providing semantic region-level grounding (bounding boxes) that preserves semantic completeness while enabling precise spatial localization.
Novel Annotation Granularity: Introduces an intermediate granularity between page-level and token-level, specifically designed for multimodal reasoning in complex documents.
Evaluation Framework: Proposes the Grounding–Crop–then–Answer protocol to strictly separate evidence localization performance from reasoning performance.
Large-Scale Training Resource: Provides a scalable, automatically constructed training set (~30K pairs) that significantly boosts model performance when used for Supervised Fine-Tuning (SFT).

4. Experimental Results

The authors evaluated a wide range of state-of-the-art VLMs (including Qwen3/3.5 series, InternVL, Ernie, Kimi, Claude, and GPT-5.2) on the SciEGQA benchmark.

Baseline Performance: Existing models struggle significantly. The best-performing model (GPT-5.2) achieved only 38.89% Mean IoU in evidence localization. Most models scored below 40% IoU, indicating that precise evidence grounding in scientific documents remains an unsolved challenge.
Impact of Granularity: Answer accuracy improves drastically as input granularity becomes finer (Document < Page < Region). For example, Qwen3.5-27B jumped from 42.95% accuracy on full documents to 77.02% when provided with the correct cropped evidence region. This highlights that current models fail primarily due to localization errors, not just reasoning deficits.
Effectiveness of Training Data: Fine-tuning Qwen3-VL-8B on the SciEGQA training set (creating Qwen3-VL-8B-SFT) resulted in:
- Mean IoU: Increased from 21.43% to 38.66%.
- QA Accuracy: Increased from 32.04% to 49.97% (under the grounding-crop pipeline).
- This demonstrates that the automatically constructed dataset provides effective supervision for improving both localization and reasoning.

5. Significance

Benchmarking: SciEGQA sets a new, more rigorous standard for evaluating scientific document understanding, moving beyond simple page-level matching to require precise evidence grounding.
Research Direction: The results indicate that future VLMs must be optimized not just for reasoning, but specifically for visual grounding within complex, multimodal documents.
Resource: The dataset and the proposed training pipeline offer a scalable solution for generating high-quality, evidence-grounded data, which is crucial for advancing multimodal reasoning in scientific domains.
Interpretability: By requiring bounding box annotations, the dataset promotes the development of models that can "show their work," enhancing the trustworthiness of AI in scientific applications.