🧬 biology

TheBioCollection: Unified Pre-Training Scale LLM Corpus for Biology

The paper introduces TheBioCollection, a unified 52.6B-token pre-training corpus that consolidates diverse biological resources into a cohesive format enriched with computed properties and instruction tasks, significantly enhancing a 16B-parameter model's biological understanding and performance across multiple domains while preserving its general linguistic capabilities.

Original authors: Hyunjin Seo, Hyeon Hwang, Gyubok Lee, Jay Shin, Jimin Park, Taesoo Kim, Sanghoon Lee, Hongjoon Ahn, Sungjun Han, Sangwon Jung

Published 2026-07-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hyunjin Seo, Hyeon Hwang, Gyubok Lee, Jay Shin, Jimin Park, Taesoo Kim, Sanghoon Lee, Hongjoon Ahn, Sungjun Han, Sangwon Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine the world of biology as a massive, chaotic library where the books are scattered everywhere. Some are tiny molecular blueprints, others are giant protein instruction manuals, and some are complex maps of how cells talk to each other. The problem? These "books" are written in thousands of different, confusing languages—some are just lists of numbers, others are graphs, and many are locked in formats that a super-smart computer brain (a Large Language Model) can't actually read.

Enter THEBIOCOLLECTION, a team of digital librarians who decided to clean up this mess. They didn't just gather these scattered resources; they built a massive, unified training set containing 52.6 billion tokens (think of these as individual words or pieces of information) that turns all those messy formats into a single, readable language.

The Great Translation Job
The team took data from tiny molecules, proteins, DNA sequences, and even whole cells, and did something clever: they didn't just copy-paste the data. They used special "translator tools" to compute new facts and write them out as natural sentences.

Instead of just giving a molecule a chemical code, the system calculated its weight, how many rings it has, and how "drug-like" it is, then wrote a sentence describing all of that.
Instead of just listing a protein's sequence, they added details about its shape and where it lives in the cell.
They even created new "quiz questions" (instruction tasks) for things that were previously missing, like asking the computer to design a new protein binder or find a specific spot on a DNA strand.

The Experiment: Teaching a Brain
To see if this new library actually works, the researchers took a base computer brain called Gravity-16B-A3B (which had never seen biology data before) and fed it this new collection. They kept the brain's architecture exactly the same to ensure the results were purely because of the new data, not because they changed the brain itself.

The Results: A Double-Digit Leap
The results were surprisingly strong. After training on this new collection, the model's ability to handle biology tasks more than doubled.

Its overall score jumped from 0.223 to 0.499.
It got significantly better at everything: understanding small molecules, designing proteins, finding spots on DNA, and even figuring out how cells react to changes.
The biggest improvements happened in areas where the data was structured and calculated by tools, like finding specific DNA regulatory spots (jumping from 0.134 to 0.516) and designing protein binders (jumping from 0.234 to 0.645).

What It Didn't Do (The "No-Go" Zone)
It's important to note what this paper didn't find. The researchers explicitly tested whether adding all this biology data would make the computer brain "forget" how to speak normal human language or solve general puzzles. They compared their new model against one that only read general science articles and web text.

The result? The biology-trained model stayed almost exactly as good at general language tasks as the one that only read web text.
The paper rules out the idea that you have to sacrifice general smarts to get biological smarts. The model learned biology without losing its ability to chat or reason about the world.
The paper also suggests that simply reading science articles (like those from PubMed) helps a little, but it argues against the idea that reading text alone is enough. The structured, tool-computed data in THEBIOCOLLECTION provided a much bigger boost, especially for tasks that require precise, non-textual facts.

The Cross-Domain Magic
One of the coolest findings is that the model didn't just learn to be good at one thing; it started connecting the dots between different fields. When asked to link a protein's function to a pathway it belongs to, or a drug to the gene it affects, the model's accuracy jumped from 0.313 to 0.507. This suggests the model is learning to "think" across different biological layers, not just memorizing facts.

The Bottom Line
The paper suggests that the secret sauce for building a biology-savvy AI isn't necessarily a new, complex brain architecture. Instead, it's about having a high-quality, unified, and tool-enriched library of data. By turning scattered, messy biological data into a cohesive, readable story, the researchers showed that you can teach a general AI to understand the deep, complex language of life without breaking its ability to understand the rest of the world.

Technical Summary: THEBIOCOLLECTION

Problem Statement

The development of Large Language Models for Biology (BioLM) has created a critical demand for training corpora that enable models to genuinely understand biological systems. While the field possesses abundant resources—including molecular databases, protein repositories, genomic annotations, single-cell atlases, and pathway databases—these resources remain scattered across heterogeneous formats (tables, sequence files, graph edges, isolated schemas). Consequently, they lack a cohesive, unified corpus suitable for pre-training language models. Existing efforts have not consolidated these disparate resources into a large-scale pre-training dataset that teaches BioLMs a broad spectrum of biological knowledge spanning molecules, proteins, genomes, cells, and their cross-domain relationships. Furthermore, current corpora often lack coverage for specific capabilities such as protein binding design and genomic feature localization.

Methodology

The authors present THEBIOCOLLECTION, a 52.6B-token pre-training scale corpus designed to unify these heterogeneous biological resources into a training-ready format. The construction pipeline involves four primary stages:

Data Collection: The corpus aggregates public, commercially usable resources across five domains: small molecules, proteins, genomic sequences, cells/pathways, and cross-domain relationships. Sources include major databases (e.g., PubChem, UniProt, GENCODE, AlphaFoldDB, Human Protein Atlas) and broad scientific literature (PubMed, bioRxiv, medRxiv).
Dataset Refinement: Structured database records are transformed into self-contained natural language descriptions. Key attributes (physicochemical properties, structural information, functional annotations) are consolidated into single textual records. Symbolic identifiers (SMILES, protein/DNA/RNA sequences) are wrapped in dedicated tagging tokens (e.g., <smiles>...</smiles>) to distinguish them from surrounding text. Cross-domain knowledge graphs (e.g., Hetionet, DRKG) are utilized to link entities, creating mechanistic chains where a single record connects a protein to its binding compounds, pathways, and cellular responses.
Tool-Computed Feature Enrichment: To expose signals that exist in computational tools but not in free text, the authors run standard bioinformatics tools over source records and verbalize the outputs.
- Small Molecules: RDKit descriptors (ring systems, scaffold complexity, stereochemistry, QED, etc.) are computed and added to narratives.
- Proteins: UniProt sequences are linked to AlphaFold and PDB structures, enriching records with secondary structure, fold topology, and solvent exposure data derived via Biopython and DSSP.
- Genomic Sequences: Regulatory DNA and RNA transcripts are annotated with features like GC content, CpG statistics, and motif profiles.
- Cells: Single-cell records are converted to language capturing expression, perturbation response, spatial context, and morphology.
Instruction Construction: The corpus includes both imported instruction datasets (e.g., Mol-Instructions, ProteinLMBench) and source-derived instruction tasks generated specifically to address gaps in existing corpora. These new tasks include:
- Protein Binding: Generating binder sequences, recovering masked binding-site residues, and summarizing contact patterns based on structural interfaces.
- DNA/RNA Feature Localization: Exact span-recovery tasks for regulatory elements, splice sites, and RNA motifs, ensuring answers are programmatically checkable.

To evaluate the corpus, the authors introduce THEBIOCOLLECTION-EVAL, a matched evaluation suite covering recognition, generation, and prediction across molecular, protein, genomic, cellular, and cross-domain settings.

Experimental Setup

The authors trained the Gravity-16B-A3B architecture (a 16B parameter model) on THEBIOCOLLECTION. To isolate the effect of the corpus, the base model was held fixed, and training was conducted before the standard annealing phase. The training mixture included THEBIOCOLLECTION alongside general scientific and web text (from OLMoCR, DCLM-CC, etc.) to prevent catastrophic forgetting of general linguistic abilities.

Key Results

Training on THEBIOCOLLECTION resulted in significant improvements in biological task performance while preserving general language capabilities:

Overall Performance: The model trained on THEBIOCOLLECTION more than doubled the base model's overall score on THEBIOCOLLECTION-EVAL (from 0.223 to 0.499).
Domain-Specific Gains:
- Small Molecules: Improved from 0.223 to 0.513 (e.g., forward synthesis increased from 0.213 to 0.619).
- Proteins: Improved from 0.159 to 0.407, with notable gains in binder design (0.234 to 0.645).
- Genomic Sequences: Improved from 0.175 to 0.468, particularly in DNA regulatory/splice span localization (0.134 to 0.516).
- Cells/Pathways: Improved from 0.335 to 0.609.
Ablation Study: When compared to a model trained only on general scientific text (Text-annealing only), THEBIOCOLLECTION provided an additional +0.114 improvement in the biological overall score. The most significant gains were observed in structured, tool-derived tasks (e.g., genomic localization +0.242, binder design +0.348), whereas text-only models performed better on tasks where facts are explicitly stated in literature.
General Language Preservation: Despite the heavy weighting of biological data, the model's performance on general language benchmarks (MMLU, ARC-c, HellaSwag, etc.) remained nearly intact, with an average drop of only 0.9 points compared to the text-annealing baseline.
Cross-Domain Reasoning: The model demonstrated improved ability to chain evidence across domains (e.g., Protein function $\to$ Pathway), raising average accuracy from 0.313 to 0.507 in zero-shot settings.

Significance and Claims

The paper claims that THEBIOCOLLECTION demonstrates that carefully constructed data is a primary driver of practical biological capability in BioLMs. Key contributions include:

Unified Corpus: It provides the first large-scale, pre-training scale corpus that unifies scattered biological resources into a shared language interface, covering small molecules, proteins, genomes, cells, and pathways.
Tool-Driven Enrichment: It introduces a methodology for verbalizing tool-computed biological properties, exposing structured signals (like molecular descriptors and structural features) that are typically absent in free-text corpora.
Instruction Expansion: It constructs new instruction datasets for underrepresented capabilities, specifically protein binding and genomic feature localization, which existing public instruction datasets rarely address.
Joint Learning: The results suggest that biological capability and general linguistic ability are not mutually exclusive; they can be learned jointly without significant interference, provided the training corpus is carefully curated.
Evaluation Standard: The accompanying THEBIOCOLLECTION-EVAL suite offers a standardized, cross-domain benchmark for assessing biological reasoning, moving beyond single-domain evaluations.

The authors conclude that their work highlights the potential of data-centric approaches to advance BioLMs, showing that a unified, enriched, and instruction-rich corpus can substantially improve model performance across the biological spectrum without requiring specialized architectures or additional training tricks.