SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

1. Problem Statement

Despite significant advances in Natural Language Processing (NLP) driven by large-scale open corpora, scientific language remains underrepresented in open datasets. Existing resources often lack:

• Transparency and Reproducibility: Many scientific datasets rely on proprietary pipelines or non-open-source tools, making it difficult to verify or extend the data construction process.
• Structured Accessibility: Scientific text possesses unique linguistic and structural properties (precise terminology, formal discourse, data-centered argumentation) that differ from general web text. Existing datasets often fail to preserve document structure (e.g., citations, metadata) in a standardized format suitable for complex downstream tasks like citation graph analysis.
• Data Licensing Ambiguity: The legal status of Open Access (OA) content is often unclear, hindering the creation of fully redistributable, high-quality training sets.

2. Methodology: The SciLaD Pipeline

The authors present SciLaD (Scientific Language Dataset), constructed entirely using open-source frameworks and publicly available data. The pipeline consists of three main stages:

A. Data Harvesting

• Sources: The primary source is Unpaywall (a database of 50M+ OA articles), supplemented by arXiv (LaTeX sources), PubMed Central (PMC) (JATS XML), and PLOS (XML).
• Scale: The initial harvest yielded 38.9 million entries. After filtering out non-document "components" (figures/tables) and handling download failures, 31.6 million PDF documents were secured.
• Infrastructure: Processing was performed on the JetStream 2 HPC environment using 24 parallel virtual machines.

B. Preprocessing and Normalization

To handle the heterogeneity of formats (PDF, JATS XML, LaTeX), the pipeline normalizes all data into a single structured format: TEI XML (Text Encoding Initiative).

• PDF Processing: Instead of using slow and expensive Visual Language Models (VLMs) like Nougat, the authors used Grobid, an open-source tool based on RNNs and CRFs. Grobid processes ~120 pages/second on commodity hardware, avoiding OCR errors by extracting the native text layer of born-digital PDFs.
• XML/LaTeX Conversion:
  • JATS XML: Converted to TEI XML using Pub2TEI (XSLT stylesheets).
  • LaTeX: Converted to structured XML using LaTeXML, which preserves mathematical semantics.
• Output: This stage produced 35.3 million publications in TEI XML, preserving 109 million citation nodes and 1.51 billion edges.

C. Postprocessing and Filtering

• Text Extraction: TEI XML was converted to plain text, retaining paragraphs and captions while excluding complex tables/equations to ensure text quality.
• Language Filtering: A weighted language detection score (using fasttext) filtered for English. Documents with <80% confidence were removed.
• Quality Filtering: The Gopher language quality filter was applied. Unlike web-data filters (e.g., C4), this was tuned for scientific text.
• Deduplication: MinHash deduplication (5-grams, 112 hash functions) was applied to remove near-duplicates (similarity >75%).
• Final Corpus: The English split resulted in 10.9 million documents (approx. 68.7 billion tokens).

3. Key Contributions

1. SciLaD Dataset:
   • A multilingual corpus of 35.3 million scientific articles in TEI XML.
   • A filtered English split of 10.9 million documents (68.7B tokens) for pre-training.
   • Includes rich metadata, citation graphs, and author keywords.
2. Open-Source Pipeline:
   • A fully reproducible pipeline using Grobid, Pub2TEI, LaTeXML, and Datatrove.
   • All code and data processing scripts are publicly available on GitHub.
3. SciLaD-M Models:
   • Pre-trained RoBERTa-base encoders (110M parameters) from scratch.
   • Two variants: SciLaD-M (custom) (trained with a custom tokenizer) and SciLaD-M (RoBERTa) (using the original RoBERTa tokenizer).
4. Evaluation Framework:
   • A reproducible benchmarking pipeline for encoder and encoder-decoder models across five scientific NLP tasks.

4. Experimental Results

The authors evaluated SciLaD-M models against baselines (SciBERT, BERT, RoBERTa, ModernBERT) on 11 datasets across 5 tasks.

• Overall Performance: SciLaD-M (custom) achieved an average F1 score of 74.91, outperforming SciBERT (71.19) and general-domain models.
• Named Entity Recognition (NER):
  • Achieved 97.20 F1 on BC5CDR (Chemical/Disease), surpassing all baselines.
  • Performed on par with SciBERT on JNLPBA and NCBI-disease.
• Relation Extraction (REL):
  • Outperformed all baselines on ChemProt (83.83) and SciERC (81.23), demonstrating effective capture of relational semantics.
• Text Classification (CLS):
  • Achieved the best results on CitationIntent (64.46), significantly outperforming general models.
• Dependency Parsing (DEP):
  • Both SciLaD variants outperformed baselines on the GENIA corpus (UAS and LAS), indicating improved syntactic understanding of scientific text.

Conclusion: The results validate that a model pre-trained exclusively on open-access, open-source processed data can achieve state-of-the-art performance comparable to models trained on proprietary or mixed-source data.

5. Significance and Impact

• Reproducibility & Transparency: SciLaD sets a new standard for scientific NLP by providing a fully transparent, open-source pipeline. This allows the community to audit, extend, and verify data curation processes.
• Ethical Data Sourcing: The dataset demonstrates that high-quality scientific LLMs can be built without relying on proprietary, restricted, or potentially illegally acquired data sources.
• Structured Utility: By preserving TEI XML structure and citation graphs, SciLaD enables advanced downstream tasks (e.g., bibliometric analysis, citation graph modeling) that are difficult with plain-text-only datasets.
• Community Resource: The release of the dataset, models, and evaluation pipeline lowers the barrier to entry for researchers in specialized scientific domains, fostering further innovation in Natural Scientific Language Processing (NSLP).

Limitations & Future Work:

• The current plain-text split is limited to English; future work aims to expand to other languages.
• The dataset currently focuses on discriminative tasks; future iterations will explore generative tasks (summarization, QA) and more sophisticated exploitation of the citation graph and TEI structure.
• A strictly CC-BY licensed split for unrestricted redistribution is planned for future releases.