Technical Summary: Self-Driving Datasets for Biomedical Knowledge

Problem Statement

The authors identify three critical limitations in existing manually curated biomedical repositories (e.g., TDC, ClinVar, B3DB) that hinder the development of high-quality AI models for therapeutic design:

1. Lack of Nuance: Properties like oral bioavailability or toxicity are not intrinsic functions of a molecule alone but depend heavily on experimental context (e.g., fed vs. fasted state, dosing route, disease state). Tabular databases often discard this context, forcing models to estimate marginal distributions over conditions.
2. Incompleteness: Curated databases lag behind primary literature by years and capture only a fraction of measured data. Furthermore, evidence for the same phenomenon is often reported in incompatible forms that are difficult to flatten into a single schema.
3. Accuracy Issues: Audits reveal non-trivial label errors in widely used benchmarks (e.g., 16.5% error on TDC BBB, 7.3% on TDC Oral Bioavailability). Models trained on these labels inherit a noise floor that caps achievable performance.

The primary obstacle to solving these issues is the scale of the literature: extracting structured records from tens of millions of papers requires "surgical retrieval" capabilities that neither classical information extraction nor current general-purpose deep research agents possess. Existing agents synthesize narrative reports rather than exhaustively scanning corpora for structured data instances.

Methodology: The Starling System

The paper introduces Starling, a multi-agent deep research system designed to autonomously convert the primary literature (specifically a 22.5M-paper PubMed corpus) into structured, nuanced datasets. The system operates on a tagged and indexed corpus through three coupled components:

1. Corpus Construction and Entity Tagging

• Corpus: A 22.5M-paper subset of PubMed (approx. 2.5 trillion tokens) licensed for AI usage. Text and tables are extracted into markdown.
• Entity Tagging Pipeline: A teacher-student distillation pipeline tags the corpus with 19 top-level biomedical entity types (e.g., SmallMolecule, Gene, Disease) grounded in nine reference ontologies (UMLS, DrugBank, ChEBI, etc.).
  • Teacher: gpt-oss-120b labels windows of 3 paragraphs, running separate passes for each entity type to improve precision.
  • Distillation: High-frequency entities are validated against reference ontologies using GPT-5.
  • Student: gpt-oss-20b is fine-tuned on the combined supervision and run on the full corpus.
• Retrieval Infrastructure: A hybrid sparse-dense retrieval layer supports entity-filtered semantic queries. Embeddings (Qwen3-Embedding-4B) are used to re-rank search results rather than filter the corpus, allowing for surgical filtering based on entity co-occurrence.

2. The Starling Agent Pipeline

Starling operates in two phases to transform natural language task descriptions (e.g., "find all molecules with reported oral bioavailability") into structured datasets:

• Phase 1: Query Construction (Planning):

  • A Proposer agent constructs a set of "probes," each consisting of an entity filter (CNF constraints) and a semantic query.
  • The system iteratively refines probes to maximize precision (via a Validator agent judging relevance) and recall (by identifying gaps where relevant text is excluded by current filters, guided by an Investigator agent).
  • Once probes are finalized, the Proposer induces a unified extraction schema (field names, types, constraints) by sampling papers and running an Extractor in free-form mode, then refining the schema based on Validator feedback.

• Phase 2: Extraction and Judging:

  • The Extractor (Gemma 4 31B) processes the ordered subcorpus, emitting structured records linked to supporting passages.
  • A Judge (a distilled Qwen3.5-9B model) performs a final filter, scoring extractions on five axes: task relevance, primary-label correctness, span fidelity, secondary-field accuracy, and entity attribution. Only records passing all axes are retained.

Key Contributions

1. Self-Driving Dataset Construction: A fully autonomous pipeline that requires no human intervention for schema design or paper selection, capable of processing literature at scale.
2. Nuanced, Context-Rich Data: The system captures experimental conditions (e.g., food state, dosing route) as first-class fields, preserving signal that tabular databases discard.
3. Scale and Accuracy: The system produces datasets that are larger and, according to audits, more accurate than existing manually curated benchmarks.
4. Open Release: The authors release the agent code, entity tagging models, retrieval recipes, and six specific datasets, along with an API for hosted access.

Results

The system was evaluated on six therapeutic discovery tasks: Blood-Brain Barrier (BBB) permeability, Oral Bioavailability, Acute Toxicity (LD50), Gene-Disease Associations (GDA), Protein Subcellular Localization (PSL), and Chemical Reactions.

• Scale: Starling produced approximately 6.3 million judge-filtered records.
  • Oral Bioavailability: 185K extractions (19K molecules), covering 96.1% of TDC instances and exceeding TDC size by 29x.
  • Gene-Disease Associations: 3M extractions, 6x the size of ClinVar.
  • Protein Localization: 1.9M extractions, 24x the size of UniProt (Homo sapiens).
  • Chemical Reactions: 916K extractions, discovering ~400K novel products not present in five major reference databases.
• Accuracy:
  • Frontier model rejection rates for kept extractions ranged from 0.6% to 7.7% across tasks.
  • Human audits of disagreements between Starling and manually curated benchmarks (e.g., BBB_Martins, Bioavailability_Ma) revealed that curated databases had confirmed error rates of 7.3% to 19.7%, whereas Starling's confirmed error rates were ≤1.8%.
• Downstream Utility: Fine-tuning the MiniMol predictor with Starling data improved or maintained performance on test sets compared to training on TDC data alone, particularly for molecules where structure-only models fail (e.g., active influx mechanisms for BBB).
• Cost: The extraction cost is approximately $0.001 per record, dominated by inference costs.

Significance and Claims

The paper argues that primary literature, when processed autonomously, can yield datasets that are larger, more nuanced, and more accurate than the curated databases they replace. The authors claim that the "nuance" captured—specifically the experimental context explaining within-molecule variance—enables better predictive modeling and a more accurate characterization of model failures.

The work establishes a foundation for multimodal predictive and generative models in AI-driven therapeutic design, moving beyond the "marginal distribution" limitations of current tabular data. The authors emphasize that their system is fully self-driving, capable of keeping datasets synchronized with rapidly expanding literature at a fraction of the cost of manual curation.

Note: The paper explicitly states limitations, including the inability to extract data from figures (text and tables only), the reliance on licensed corpora that cannot be redistributed, and the necessity of the "Judge" step to prevent low-quality data floods. The system does not perform evidentiary weighting or provenance adjudication beyond the claims made in the source text.