A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews

Technical Summary: Webis-SR4ALL-26 Corpus

Problem Statement

Systematic reviews (SRs) are the gold standard for evidence synthesis in fields like medicine, social sciences, and education. Automating the most expensive stages of this process—retrieval (finding relevant literature) and screening (deciding which studies to include)—is a major research goal. However, existing benchmarks for these tasks suffer from two primary limitations:

1. Disciplinary Bias: Most existing datasets are heavily concentrated in the biomedical domain.
2. Scale Constraints: Current collections are relatively small, limiting their utility for large-scale machine learning training and cross-domain comparative studies.

Methodology

The authors introduce Webis-SR4ALL-26, a large-scale, cross-disciplinary corpus. The construction follows a three-stage pipeline:

1. Data Collection & Filtering:

   • Source: The authors queried the OpenAlex database for works titled "systematic review" or "systematic literature review."
   • Refinement: After deduplication, language filtering (English only), DOI verification, and the removal of "update" papers, they integrated existing benchmark datasets (e.g., CLEF TAR, CSMeD) to reach a final count of 301,871 systematic reviews spanning 27 scientific domains.
   • Diversity: While medicine remains the largest field (~64%), the corpus includes significant representation from Psychology, Social Sciences, Neuroscience, and Engineering.

2. Full-Text Acquisition & Parsing:

   • The pipeline attempted automated downloads of PDFs via OpenAlex links, successfully retrieving 72,678 full texts.
   • These documents were parsed into Markdown using PaddleOCR-VL, a vision-language model capable of handling complex scientific layouts (multi-column text, tables, etc.).

3. Structured Information Extraction (Verify-Then-Repair):

   • To extract methodological artifacts (objectives, research questions, inclusion/exclusion criteria, and search strategies), the authors used an LLM-based pipeline centered on the Qwen3-32B model.
   • Verification: To prevent hallucinations, they implemented a "grounded extraction" strategy. Every extracted value was paired with a verbatim evidence span from the text. A two-stage check (string matching and an LLM-based verifier called MiniCheck) ensured the extraction was factually supported.
   • Repair: If a field was found to be null after the primary pass, a targeted "Repair Pass" was triggered to attempt extraction again without compromising the precision of already-verified fields.

4. Query Normalization:

   • Because search strategies are written in heterogeneous, database-specific languages (e.g., Scopus vs. PubMed), the authors used an LLM to normalize them into simplified, executable Boolean expressions compatible with the OpenAlex API.

Key Contributions

• Webis-SR4ALL-26 Corpus: A massive, multi-layered dataset of over 300k reviews.
• Multi-Layered Data: The corpus provides:
  • Bibliographic Layer: Resolved OpenAlex metadata and citation links.
  • Methodological Layer: Structured text (objectives, criteria, etc.) extracted from full texts.
  • Retrieval Layer: Normalized, executable Boolean queries.
• Open Pipeline: Release of the pre-processing pipeline and code for extraction validation and retrieval demonstration.

Results

The authors conducted a proof-of-concept retrieval experiment to demonstrate the utility of the normalized query layer. They executed the normalized queries against OpenAlex and compared the results to the reviews' actual reference lists:

• Boolean Queries: Achieved a recall of 0.245 and precision of 0.014.
• Keyword Lists: Achieved a recall of 0.180 and precision of 0.016.
• Observation: While precision was low (typical for large-scale web searches), the Boolean strategies significantly outperformed simple keyword lists in terms of recall, proving the effectiveness of the normalization process.

Significance

Webis-SR4ALL-26 shifts the landscape of systematic review research from a biomedical-centric focus to a cross-disciplinary one. It serves four major research communities:

1. Information Retrieval (IR): Provides a massive benchmark for testing retrieval and screening algorithms.
2. Natural Language Processing (NLP): Offers a target for training models to extract complex methodological artifacts from scientific text.
3. Meta-Science/Sociology: Enables large-scale studies on how scientific reporting practices and search strategies evolve across different disciplines and over time.
4. Evidence Synthesis: Supports the development of tools to assist researchers in automating the most labor-intensive parts of the systematic review workflow.