Genomic language models improve cross-species gene expression prediction and accurately capture regulatory variant effects in Brachypodium mutant lines

This study demonstrates that deep learning sequence-to-expression models leveraging context-aware embeddings from the PlantCaduceus genomic language model, augmented with chromatin accessibility data, significantly outperform existing state-of-the-art methods in predicting both cross-species gene expression and the regulatory effects of single-nucleotide mutations in *Brachypodium*.

The Gist

Imagine you have a massive library of instruction manuals for building plants. These manuals are written in a four-letter code (A, C, G, T) that tells the plant's cells exactly how much of a specific protein to make. This is gene expression.

For a long time, scientists have tried to build a "translator" that can read these DNA manuals and predict exactly how much protein will be made. The problem? The manuals are written in a complex language with hidden rules, spacing tricks, and context clues that simple translators miss.

This paper introduces a new, super-smart translator called EMPRES. Here is how it works, explained simply:

1. The Old Way: Reading Letter-by-Letter

Previous models (like the one called PhytoExpr) treated DNA like a simple list of letters. They looked at an "A" and said, "Okay, that's an A." They didn't understand that an "A" next to a "T" might mean something totally different than an "A" next to a "G."

The Analogy: Imagine trying to understand a sentence by only looking at the individual letters without knowing how they are grouped into words or sentences. You might know the letters "C-A-T" are there, but you wouldn't know if it's a pet, a vehicle, or a type of hat.

2. The New Way: The "Genomic Language Model"

The authors used a tool called PlantCaduceus. Think of this as a model that has read every plant genome in existence and learned the "grammar" of DNA. It understands that certain DNA patterns are like "words" and that the distance between them matters.

The Analogy: Instead of just seeing the letters "C-A-T," this new model sees the concept of a cat. It understands the context. It knows that if you change one letter in a specific spot, it might turn a "cat" into a "bat," changing the whole meaning of the sentence.

3. Adding a "Weather Report" (Chromatin Accessibility)

DNA doesn't exist in a vacuum; it's wrapped up in a ball of yarn (chromatin). Sometimes the yarn is tight (the instructions are hidden), and sometimes it's loose (the instructions are easy to read).

The new model also looks at a "weather report" called chromatin accessibility. It asks, "Is the DNA open for business right now?" By combining the "grammar" of the DNA with the "weather" of the cell, the model gets a much clearer picture.

4. The Big Test: The "SIEVE" Experiment

To prove their new translator works, the scientists didn't just use computer simulations. They built a real-life test lab using a grass called Brachypodium.

• The Setup: They created 796 different mutant plants. Each mutant had a tiny, single-letter typo in its DNA manual (like changing a "C" to a "T").
• The Challenge: They asked the models: "If we make this tiny typo, how will the plant's protein production change?"
• The Result:
  • The old models (PhytoExpr) were like guessing games. They could tell you that a "cat" is different from a "dog," but they failed to predict the difference between a "cat" and a "bat" (a single letter change).
  • The new EMPRES model was a detective. It successfully predicted that specific single-letter typos would cause specific changes in protein production.

Why This Matters

This is a huge leap forward for plant science and farming.

• Precision Breeding: Imagine being a farmer who wants to grow a drought-resistant corn. Instead of waiting years to see if a mutation works, you could use this model to "simulate" the mutation on a computer first. If the model says, "Yes, changing this one letter will make the plant drink less water," you can go straight to growing that specific plant.
• Understanding Evolution: It helps us understand how tiny changes in DNA over millions of years created the vast diversity of plants we see today.

The Bottom Line

The authors built a new AI that doesn't just memorize DNA letters; it understands the language of plants. It can predict how plants will behave just by reading their DNA code, and it's accurate enough to spot the effects of a single typo. It's like upgrading from a dictionary to a fluent speaker who can translate the future of crop improvement.

Technical Summary

1. Problem Statement

Predicting gene expression levels directly from cis-regulatory DNA sequences (promoters and terminators) is a fundamental challenge in plant genomics. Current state-of-the-art (SOTA) Sequence-to-Expression (S2E) models, such as PhytoExpr, rely on one-hot encoding of DNA sequences. This approach treats nucleotides as independent entities, failing to capture:

• Biochemical properties and evolutionary context.
• Sequential order and higher-order dependencies (motif grammar, spacing, orientation).
• Long-range regulatory interactions.

Furthermore, while S2E models can predict expression differences between genes, they struggle to predict the effects of single-nucleotide variants (allelic variation) within a gene. Validating these models in planta (in whole plants) at single-base resolution has been a critical gap in the field.

2. Methodology

A. Data and Feature Engineering

• Dataset: The authors utilized a dataset of ~589,000 genes across 17 angiosperm species (from the PhytoExpr dataset), covering a 150-million-year evolutionary timescale.
• Input Sequences: 10,000 bp regions centered on the Transcription Start Site (TSS) and Transcription Termination Site (TTS) (5kb upstream/downstream).
• Feature Representation (The Core Innovation): Instead of one-hot encoding, the authors replaced raw sequences with context-aware embeddings from pre-trained Genomic Language Models (gLMs):
  • PlantCaduceus: A gLM pre-trained on 16 angiosperm genomes to generate sequence embeddings.
  • a2z: A model trained to predict chromatin accessibility and DNA methylation from sequence.
• Embedding Strategy: Regulatory sequences were divided into 20 overlapping windows. Embeddings were extracted from the penultimate layers of PlantCaduceus (384 dimensions) and a2z (925 dimensions), along with a2z's predicted chromatin accessibility scores.

B. Model Architecture: EMPRES

The authors developed EMPRES (Embedding-based Prediction of Expression from Sequence), a custom deep learning framework:

• Architecture: A dual-branch 1D Convolutional Neural Network (CNN).
  • Branch 1: Processes TSS features.
  • Branch 2: Processes TTS features.
  • Fusion: Outputs are concatenated and passed through fully connected layers to predict log10(1+TPM).
• Model Variants: Four specific configurations were tested:
  1. EMPRES 1: PlantCaduceus embeddings only.
  2. EMPRES 2: PlantCaduceus embeddings + a2z chromatin accessibility predictions.
  3. EMPRES 3: PlantCaduceus embeddings + a2z embeddings.
  4. EMPRES 4: a2z embeddings only.
• Training: Hyperparameters were optimized using Optuna. Models were trained using 5-fold cross-validation (CV) with gene-family-aware splits to ensure generalization to unseen gene families.

C. Experimental Validation (The SIEVE Population)

To rigorously test variant effect prediction, the authors utilized a novel SIEVE (Selection of mutations by In Silico and Experimental Variant Effects) population in Brachypodium distachyon:

• Generation: 796 lines (769 mutants, 27 controls) generated via sodium azide mutagenesis and self-fertilized for 5 generations.
• Data: Whole-genome sequencing (WGS) identified single-nucleotide variants; RNA-seq quantified gene expression.
• Validation Metrics:
  • Between-gene: Correlation between predicted and observed mean expression across control lines.
  • Within-gene (Allelic): Correlation between predicted and observed expression deviations in mutant lines compared to the gene-specific control mean. This tests the model's ability to detect single-base mutation effects.

3. Key Results

A. Cross-Species Prediction Accuracy

• Superior Performance: EMPRES models significantly outperformed the SOTA benchmark PhytoExpr.
  • EMPRES 1 & 2: Achieved a Pearson correlation ($R$) of 0.82 (vs. PhytoExpr's 0.74).
  • Variance Explained: EMPRES 1/2 explained 67% of the variance ($R^2$), compared to 54% for PhytoExpr.
• Feature Importance: Models using PlantCaduceus embeddings (EMPRES 1-3) consistently outperformed those using only a2z (EMPRES 4), indicating that gLM embeddings capture richer regulatory information than chromatin accessibility predictions alone.
• Generalization: The models maintained high accuracy across all 17 species, regardless of genome size.

B. Variant Effect Prediction (In Planta Validation)

This was the most significant finding, demonstrating the models' ability to predict the impact of specific mutations:

• Between-Gene Prediction: EMPRES 1 and 2 achieved a regression coefficient ($\beta$) of 0.78, significantly outperforming PhytoExpr ($\beta \approx 0.57$).
• Within-Gene (Allelic) Prediction:
  • EMPRES 2 achieved a regression coefficient of $\beta = 0.38$.
  • PhytoExpr showed only a weak association ($\beta = 0.06$).
  • Significance: The results confirm that EMPRES models can capture the directional effect of single-nucleotide mutations on gene expression, a task where previous models failed.
• Noise Handling: While the $R^2$ for within-gene predictions was low (due to high biological noise in mutant lines), the regression coefficient ($\beta$) provided a robust measure of the true genetic signal captured by the models.

4. Key Contributions

1. Paradigm Shift in S2E Modeling: Demonstrated that replacing one-hot encoding with pre-trained genomic language model embeddings (PlantCaduceus) significantly improves cross-species gene expression prediction.
2. First In Planta Validation of Variant Effects: Successfully validated S2E models on a large-scale, purpose-built mutant population (SIEVE), proving that these models can predict the functional impact of single-base mutations in whole plants.
3. Benchmarking: Established a new benchmark for plant regulatory genomics, showing that current SOTA models (PhytoExpr) are insufficient for capturing allelic variation, while gLM-based approaches offer a viable path forward.
4. Integration of Chromatin Data: Showed that while chromatin accessibility features (a2z) are useful, the primary driver of performance is the context-aware sequence representation from gLMs.

5. Significance and Future Directions

• Precision Breeding: The ability to predict how specific regulatory mutations alter gene expression in planta is a critical step toward precision breeding and crop improvement, allowing for the design of regulatory variants with desired phenotypic outcomes.
• Scalability: The approach prioritizes cross-species generalizability without requiring species-specific epigenomic data for every new species, making it scalable for non-model crops.
• Future Work: The authors suggest that while the current models capture a significant signal, there is still an accuracy gap between gene-level and allele-level predictions. Future improvements could involve:
  • Knowledge Distillation: Training smaller "student" models to approximate the performance of the large EMPRES ensembles to reduce computational costs.
  • Contrastive Learning: Fine-tuning models on allele-specific expression data to further refine variant effect predictions.
  • Advanced Attribution: Using perturbation-based methods (in silico mutagenesis) to interpret how specific sequence changes drive predictions.

In conclusion, this paper establishes that Genomic Language Models are a superior feature representation for plant regulatory genomics, enabling accurate cross-species expression prediction and, crucially, the detection of regulatory variant effects that were previously undetectable by standard deep learning approaches.