Technical Summary: ResearchQA

Problem Statement

Large language models (LLMs) are increasingly deployed to assist in reading and reasoning over scientific literature. However, current evaluation methods fail to adequately assess two critical properties required for scientific trust: (1) the traceability of every claim to verifiable evidence within the source document, and (2) the surfacing of complete, relevant context rather than a confident but incomplete subset. Standard evaluation metrics, such as LLM-based rubric scores or embedding similarity, often reward plausible prose or "on-topic" fabrication, failing to detect when a model invents citations or misattributes findings. Existing benchmarks like HotPotQA (Wikipedia-based) lack the dense numerical and methodological content of scientific papers, while QASPER (NLP-specific) is limited in domain scope and annotation scale.

Methodology

The authors introduce ResearchQA, a benchmark designed specifically for citation-grounded evaluation. The methodology consists of three core artifacts: a dataset, an evaluation harness, and a grader system.

1. Dataset Construction

• Corpus: Sourced from OpenAlex's open-access subset, spanning eight domains: machine learning, public health, education, environmental science, history/humanities, mathematics, psychology, and social science. The initial release contains 494 papers.
• Generation: Questions, expected answers, and evidence passages are generated by Gemini 3.1 Pro using a structured-output schema. This schema validates each row against strict constraints before writing.
• Question Taxonomy: The dataset comprises 6,211 question-answer pairs categorized into four types:
  • Lookup (Extractive): Factual questions requiring the model to locate and quote a single passage (e.g., sample size).
  • Comprehension (Abstractive): Open-ended questions requiring synthesis of themes or methodology.
  • Multi-hop: Questions requiring evidence integration from two or more distant sections (e.g., comparing a result to a baseline or computing derived quantities).
  • Adversarial: Questions containing false premises or asking about unaddressed topics, where the correct behavior is to refuse or refute with grounded evidence.
• Scoring Logic: For multi-hop questions, coverage is fractional; a model must cite at least one alternative from every required section to receive full credit.

2. Evaluation Harness

The harness routes questions through a "chat-with-paper" pipeline involving retrieval, a citation contract (requiring claims to be anchored to quoted passages), and a structured-output parser.

3. Grading System

The system employs a hybrid of deterministic and LLM-based metrics to separate citation grounding from answer quality.

• Deterministic Metrics:
  • Citation Precision: Fraction of model citations matching required sections.
  • Section Coverage: Fractional satisfaction of required sections (AND across sections, OR within alternatives).
  • Citation Accuracy: Fraction of citations that are verifiable substrings of the raw paper text. This uses a PDF-aware citation matcher that normalizes text to handle PDF extraction noise (hyphenation, ligatures, smart quotes) rather than relying on embedding similarity, which might pass fabricated-but-on-topic quotes.
  • Refusal Correctness: For adversarial rows, this measures if the model refused or provided only verifiable evidence when refuting a false premise.
• LLM-Evaluated Metrics:
  • Factual Accuracy & Completeness: Scored on a 1–5 scale using a strict, anchored rubric. Unlike standard evaluators that collapse to endpoints, this rubric defines concrete criteria for each level (e.g., "4" allows exactly one minor inaccuracy) to reduce score saturation and rubric collapse.

Key Contributions

1. ResearchQA Benchmark: A multi-domain dataset of 6,211 citation-grounded QA pairs, extending the design of QASPER to eight scientific domains with a scalable, LLM-generated annotation pipeline verified by deterministic matching.
2. Citation-Grounded Evaluation Framework: A novel evaluation harness that decouples answer quality from citation integrity. It introduces deterministic metrics (Section Coverage, Citation Accuracy, Refusal Correctness) that specifically penalize fabrication and missing evidence, which standard LLM evaluators often miss.
3. PDF-Aware Citation Matcher: A normalization pipeline designed to handle the specific noise of PDF text extraction, ensuring that verbatim quotes are correctly identified without relying on embedding similarity that could validate hallucinated content.
4. Anchored Rubric Evaluation: A methodology for LLM evaluation that uses level-anchored definitions to prevent score compression, allowing for meaningful variance in completeness scores.

Results

The authors evaluated eight leading closed- and open-weight models (including Gemini 3.1 Pro, GPT-5.4, Claude Opus 4.7, and open-weight models like GPT-OSS-120B).

• Metric Discrimination: Citation-based metrics (Section Coverage, Refusal Correctness) show substantial variance across models (e.g., a 39-point gap in Refusal Correctness), whereas LLM-evaluator scores (Factual Accuracy, Completeness) remain tightly compressed near the ceiling (e.g., 4.745–5.000). This suggests current frontier models are uniformly accurate on factual claims but vary significantly in their ability to cite verifiable evidence.
• Performance Trade-offs: No single model dominates all dimensions.
  • Gemini 3.1 Pro leads in composite quality and refusal correctness (0.870).
  • Open-weight models (e.g., Zai GLM 4.7, GPT-OSS-120B) approach the citation accuracy of top closed models while achieving 3 to 6 times lower per-example latency.
  • Claude Opus 4.7 produces the most complete answers but has the lowest citation precision, indicating a tendency to over-cite.
• Task Difficulty:
  • Multi-hop questions are the most challenging, showing the widest spread in all metrics (e.g., a 42-point gap in Section Coverage).
  • Lookup questions are largely saturated, with most models achieving high accuracy.
  • Adversarial questions reveal a clean split in model behavior regarding grounded refusal.
• Domain Variance: Performance varies by subject matter. Economics and History/Humanities show higher median performance, while Education and Machine Learning are the most challenging domains with lower medians and wider variance.

Significance

The paper argues that for scientific reading, the ability to trace claims to verifiable evidence is more critical than fluency. ResearchQA demonstrates that standard evaluation methods are insufficient for detecting citation hallucinations. By introducing deterministic citation metrics, the benchmark reveals significant differences in model behavior that are invisible to LLM evaluators.

The work posits that a strong frontier LLM paired with a deterministic verification step can effectively replace human annotators for scaling citation-grounded datasets to multiple domains at low cost. The benchmark highlights that while models are becoming factually accurate, their ability to provide verifiable, section-specific evidence—particularly in multi-hop and adversarial scenarios—remains a critical differentiator. The authors release the benchmark, evaluation harness, and evaluator prompts to facilitate further research into citation-grounded reasoning.

Limitations: The authors note that the dataset relies on LLM-generated ground truth without stratified human validation, and the evaluation uses a single Gemini model for grading (introducing potential same-family bias). Additionally, the corpus is limited to English open-access papers, excluding image-heavy layouts and closed-access literature.