BOTANIC-0: a series of foundation models for plant genomic data

This paper introduces Botanic0, a family of plant genomic foundation models pretrained on diverse plant genomes that achieve state-of-the-art performance across various genomic prediction tasks and establish a scalable framework for advancing crop improvement and genome editing research.

The Gist

Imagine you are trying to teach a robot how to speak the language of plants. Plants don't speak English or French; they speak in a code made of four letters: A, C, G, and T. These letters form the DNA instructions that tell a plant how to grow, how to fight off bugs, and how to survive a drought.

For a long time, scientists had to read these instructions one by one, like a librarian manually checking every book in a massive library. It was slow, expensive, and often they couldn't figure out why a plant had a certain trait.

Enter Botanic-0.

What is Botanic-0?

Think of Botanic-0 as a super-intelligent plant translator that has read the entire library of plant DNA. It's a family of three "brains" (models) of different sizes:

• Botanic-0-S (Small): A quick, nimble student.
• Botanic-0-M (Medium): A well-read scholar.
• Botanic-0-L (Large): A master librarian with a near-infinite memory.

These models weren't taught by humans giving them flashcards. Instead, they were self-taught. The researchers gave them millions of pages of raw plant DNA and said, "Here is a sentence with a missing word. Guess what the missing word is."

By playing this "fill-in-the-blank" game billions of times across 43 different types of plants (from tiny mosses to giant wheat), the models learned the grammar of life. They learned that certain letter combinations usually mean "start growing here," while others mean "stop" or "make a flower."

Why Do We Need This?

The world is getting hotter, and crops are struggling. We need to breed new, tougher plants faster than ever before. Traditionally, breeding a new crop variety takes 8 years. That's too slow when pests are evolving and the climate is changing.

Botanic-0 acts like a crystal ball for plant genetics.

• The Old Way: A scientist guesses which gene might help a tomato survive heat, then spends years testing it in a field.
• The Botanic-0 Way: The AI looks at the DNA, predicts exactly which tiny change will make the tomato heat-resistant, and tells the scientist, "Try editing this specific letter." This shrinks the timeline from years to months.

How Good Is It?

The researchers tested their "brains" on a series of challenges, like identifying where genes start and stop, or predicting how strong a plant's immune system is.

• The "Zero-Shot" Test: They asked the model to guess the effect of a mutation it had never seen before. It did surprisingly well, proving it truly understood the rules of the language, not just memorized the answers.
• The "Fine-Tuning" Test: They gave the model a specific job (like predicting crop yield) and gave it a little bit of extra training. It became a top-tier expert, matching or beating other state-of-the-art models.

The "Scaling" Secret

One of the coolest findings is that bigger is better.
Imagine trying to learn a language. If you only read a few pages, you might know basic words. If you read a whole encyclopedia, you understand the nuance, the poetry, and the exceptions.
The researchers found that as they made the Botanic-0 models bigger (giving them more "brain power"), they got consistently better at predicting plant traits. There was no sign of them "hitting a wall" yet. This suggests that if we build even bigger models in the future, they could become even more powerful tools for feeding the world.

The Big Picture

Botanic-0 is like handing a farmer a GPS for the genome. Instead of driving around in circles hoping to find a good spot to plant, the AI points directly to the genetic "coordinates" that will lead to a bumper crop.

The team has released these models to the public (like giving everyone the keys to the library), hoping that scientists and farmers around the world will use them to:

1. Design better crops that can survive climate change.
2. Edit genes with precision to fix diseases.
3. Speed up the journey from the lab to the dinner table.

In short, Botanic-0 is teaching computers to speak the language of nature, so we can finally have a conversation with our crops and ask them to help us survive a changing world.

Technical Summary

1. Problem Statement

The development of resilient, high-yielding crop varieties is critical for global food security but remains a slow process (up to 8 years per variety). A major bottleneck is translating genomic associations into mechanistic understanding of how genetic variants influence gene regulation and phenotypes. While Genome-Wide Association Studies (GWAS) identify trait-associated regions, determining causal variants and designing precise genome edits requires extensive experimentation.
Existing deep learning approaches for genomics have shown promise, but plant genomics presents unique challenges due to:

• Genomic Complexity: Large genome sizes, extensive non-coding regions, structural variations, and lineage-specific regulatory architectures.
• Data Scarcity: Many plant species lack sufficient experimental annotations for supervised learning.
• Standardization Gaps: Evaluation practices for plant genomic language models (gLMs) are still evolving, making fair comparisons difficult.

The authors aim to address these issues by developing a family of plant-specific genomic foundation models capable of learning regulatory biology directly from raw DNA sequences without labeled data, thereby accelerating crop improvement research.

2. Methodology

Model Architecture and Pre-training

• Model Family: The authors introduce Botanic0, a series of three Transformer-based models (inspired by ESM-2 and adapted for DNA) with varying capacities:
  • Botanic0-S: 114M parameters.
  • Botanic0-M: 260M parameters.
  • Botanic0-L: 991M parameters (comparable to the state-of-the-art AgroNT).
• Tokenization: The models use 6-mer tokenization (grouping DNA sequences into 6-base pairs) with a context window of 6,000 bp (1,024 tokens). This differs from single-base tokenization used in some generalist models but aligns with the AgroNT approach.
• Pre-training Objective: Masked Language Modeling (MLM). The models are trained to predict masked 6-mers given the surrounding context.
• Training Data:
  • Source: 43 phylogenetically diverse plant species (nuclear DNA only) downloaded from NCBI.
  • Dataset Construction: Genomes were split into overlapping 6,100 bp windows. The dataset is dominated by intergenic regions (>85%), reflecting natural genomic composition.
  • Validation: 5 species (e.g., Oryza sativa, Hordeum vulgare) were held out completely during training to test zero-shot generalization.
• Training Pipeline:
  • Implemented using the lingua library for reproducibility and efficient streaming.
  • Optimizer: AdamW with a warmup-stable decay learning rate schedule.
  • Hardware: Trained on 8 GPUs (A100, H100, or B200) using bfloat16 precision. The largest model (Botanic0-L) took ~10 days to train.

Evaluation Strategy

The authors evaluated the models using a multi-faceted approach:

1. Zero-Shot Analysis:
   • Log-Likelihood Ratios (LLR): Measuring the model's ability to rank deleterious mutations (e.g., start codons, splice sites) in Arabidopsis thaliana.
   • Genomic Region Classification: Using embeddings to classify genomic regions (CDS, intron, intergenic, etc.) via XGBoost.
2. Probing Tasks (Frozen Weights):
   • Linear Probing & XGBoost Probing: Evaluating the quality of learned representations on five binary classification tasks (TIS, TTS, Donor, Acceptor, Conservation) from the PlantCAD benchmark and an "Enhancer Region" task from the Plant Genomic Benchmark (PGB).
   • Robustness: Analyzing sensitivity to L2 regularization strength.
3. Fine-Tuning:
   • Fine-tuning Botanic0 models on 17 PGB datasets (regression and classification) using IA³ (a parameter-efficient fine-tuning method) to adapt the models to specific downstream tasks.

3. Key Contributions

1. First Generation of Plant-Specific Foundation Models: Botanic0 represents a dedicated family of models pretrained exclusively on diverse plant genomes, filling a gap left by generalist models (like NTv2) that underperform on plant-specific tasks due to lack of domain specificity.
2. Demonstration of Scaling Laws: The paper provides empirical evidence that increasing model capacity (from 114M to 991M parameters) and training duration leads to consistent improvements in both pre-training loss and downstream task performance, suggesting the approach is scalable.
3. Comprehensive Benchmarking: The authors rigorously compare Botanic0 against state-of-the-art models (AgroNT, PlantCaduceus, NTv3, GPN) across multiple benchmarks (PGB, PlantCAD), establishing a new baseline for plant genomics.
4. Open Science: All models, code, and datasets are released on Hugging Face to support reproducible research and community benchmarking.

4. Key Results

• Pre-training Dynamics: All models showed continuous loss reduction without saturation, even for the largest model (Botanic0-L), indicating that larger models are beneficial. The validation loss on unseen species decreased over time, confirming transferability.
• Zero-Shot Performance: Botanic0-L achieved LLR scores highly correlated with PlantCAD and GPN models, outperforming AgroNT in ranking deleterious mutations.
• Embedding Quality:
  • Genomic Structure: Botanic0 embeddings successfully clustered distinct genomic regions (CDS, introns, etc.) in a 2D space. An XGBoost classifier achieved 0.641 balanced accuracy on genomic region classification, outperforming AgroNT (0.542).
  • Probing Tasks: Botanic0 models were competitive with the best existing models on standard benchmarks (TIS, TTS, Donor, Acceptor). Notably, Botanic0 models were less sensitive to L2 regularization than PlantCAD models, suggesting task-relevant information lies in a well-conditioned subspace.
• Fine-Tuning:
  • Botanic0 models achieved performance comparable to AgroNT on PGB tasks.
  • Scaling Effect: Performance on downstream tasks generally increased with model size and training steps, though the relationship was less monotonic than pre-training loss.
  • Task Specificity: The benefits of pre-training were most pronounced on complex, data-limited tasks. For tasks relying on simple sequence statistics (e.g., promoter strength), smaller models trained from scratch performed similarly, suggesting pre-training is most valuable for complex regulatory prediction.

5. Significance and Future Directions

• Accelerating Crop Improvement: Botanic0 provides a practical tool for researchers to predict the effects of genetic variants, prioritize candidates for editing, and understand regulatory mechanisms without extensive wet-lab experimentation.
• Domain Specificity vs. Scale: The results highlight that while generalist models trained on massive multi-organism datasets (like NTv3) are powerful, targeted training on domain-specific data (43 plant genomes) can yield superior or comparable results with significantly less data, emphasizing the importance of curation.
• Future Roadmap:
  • Architecture: The authors suggest moving toward single-base tokenization and architectures with Reverse-Complement (RC) equivariance (built-in symmetry for DNA strands) to improve performance.
  • Data Strategy: Future models may benefit from oversampling functional genomic regions to balance the heavy intergenic content.
  • Multimodality: The ultimate goal is to evolve Botanic into multimodal foundation models that integrate transcriptomics, proteomics, epigenetics, and environmental data to predict complex phenotypes (yield, stress tolerance) rather than just local regulatory signals.

In conclusion, Botanic0 establishes a robust foundation for plant genomics AI, demonstrating that large-scale, plant-specific pre-training is a viable and effective strategy for decoding the "genomic language" of crops.