A Long-Context Generative Foundation Model Deciphers RNA Design Principles

The paper introduces EVA, a long-context generative foundation model trained on over 114 million full-length RNA sequences that leverages a Mixture-of-Experts architecture to achieve state-of-the-art performance in deciphering RNA design principles and enabling versatile, controllable sequence engineering across diverse RNA classes.

The Gist

Imagine you are trying to teach a robot to write poetry. But instead of English or French, the robot needs to learn RNA, the complex molecular language cells use to build life, fight diseases, and run our bodies.

For a long time, scientists had a hard time teaching robots this language. The existing "teachers" (AI models) were like students who only memorized short sentences. They couldn't understand the full story of a long RNA molecule, and they often made up nonsense when asked to write new ones.

Enter EVA (Evolutionary Versatile Architect). Think of EVA as a super-genius librarian who has read every book in the universe's library of life.

Here is the story of how EVA works, explained simply:

1. The Massive Library (The Data)

To learn a language, you need to read a lot of books. Previous AI models tried to learn RNA by reading a few thousand short snippets. EVA, however, was trained on 114 million full-length RNA sequences.

• The Analogy: If previous models were like someone who learned English by reading 100 text messages, EVA is someone who has read the entire Library of Congress, including every novel, scientific paper, and history book ever written.
• The Result: EVA didn't just memorize words; it learned the grammar of life. It understands how RNA folds, how it behaves in humans versus bacteria, and how it evolves over millions of years.

2. The Super-Long Memory (The Context Window)

One of the biggest problems with old AI models is their short attention span. They could only "see" about 1,000 letters of an RNA sequence at a time. If the sequence was longer, they would forget the beginning by the time they reached the end.

• The Analogy: Imagine trying to solve a 10,000-piece puzzle, but you can only hold 1,000 pieces in your hands at once. You'd lose the picture. EVA has a memory span of 8,192 letters. It can look at the entire puzzle at once, understanding how the beginning connects to the end.
• The "Needle in a Haystack" Test: The researchers tested this by hiding a tiny, specific sequence (a needle) inside a massive block of random letters (a haystack). EVA could find the needle every time, proving it remembers long-distance connections perfectly.

3. The Master Chef with a Recipe Book (Controllable Design)

EVA isn't just a reader; it's a creator. But unlike a chaotic artist who throws paint at a wall, EVA is a master chef who can follow specific instructions.

• The Analogy: Imagine a chef who can cook any dish. If you say, "Make me a pizza," they do. If you say, "Make me a pizza, but use Italian ingredients and make it spicy," they do that too.
• How it works: You can tell EVA: "Design a tRNA (a tiny delivery truck for cells) for a human," or "Design an aptamer (a molecular magnet) that grabs a specific virus." EVA understands these instructions and generates brand-new, working RNA sequences that have never existed before, but still follow the rules of biology.

4. The Crystal Ball (Predicting Mutations)

EVA is so good at understanding the "grammar" of life that it can predict what happens if you change a single letter in an RNA sequence.

• The Analogy: If you change a word in a sentence, the meaning might change. EVA can look at a sentence and say, "If you change this letter, the sentence will become nonsense," or "If you change this letter, the sentence will become even stronger."
• Why it matters: This helps scientists figure out which mutations in a virus might make it dangerous, or which changes in a gene might cause a disease, without having to run expensive lab experiments first.

5. The Real-World Magic (What EVA Can Do)

The paper shows EVA doing some incredible things:

• Vaccine Design: It helped redesign the "instructions" for mRNA and circular RNA vaccines to make them more stable and effective, like tuning an engine to run smoother.
• CRISPR Guides: It designed better "scouts" for CRISPR (the gene-editing scissors), helping them find the right spot to cut DNA more accurately.
• New Drugs: It created new RNA shapes (aptamers) that can stick to specific targets, acting like tiny, custom-made drugs.

The Big Picture

Before EVA, designing RNA was like trying to build a house by guessing where the bricks go. With EVA, we now have a blueprint and a master builder who understands the physics of the bricks, the history of the neighborhood, and the needs of the people living there.

EVA is open-source, meaning the "recipe" is free for everyone to use. This is a huge leap forward, giving scientists a powerful new tool to cure diseases, engineer new medicines, and perhaps one day, design life itself.

In short: EVA is the first AI that truly "speaks" the full language of life, allowing us to write new chapters in the story of biology.

Technical Summary

1. Problem Statement

Programmable design of RNA sequences with defined functions is a central challenge in biology. While recent advances in foundation models have shown promise, existing RNA generative models face three critical limitations:

• Lack of Controllability: Most models are encoder-only (masked language modeling) designed for representation learning rather than generative design, lacking explicit control over context and constraints.
• Short Context Windows: Existing models typically operate with context lengths (e.g., 1,024 tokens) insufficient to model full-length transcripts, preventing the capture of long-range dependencies essential for complex RNA structures.
• Data Heterogeneity and Scale: Training data often mixes RNA and DNA or lacks phylogenetic diversity, leading to models that fail to generalize across the full evolutionary manifold of RNA. Furthermore, RNA sequences exhibit lower conservation and more imbalanced cluster sizes than proteins, causing standard training to overfit on abundant families (like rRNA) while neglecting rare functional classes.

2. Methodology: The EVA Framework

The authors introduce EVA (Evolutionary Versatile Architect), a long-context generative foundation model designed to overcome these limitations.

A. Data: OpenRNA v1

• Scale: A curated dataset of 114 million full-length RNA sequences (231.3 billion nucleotides).
• Diversity: Spans the full tree of life (prokaryotes and eukaryotes) and covers 15 distinct RNA categories (including mRNA, tRNA, rRNA, circRNA, lncRNA, viral RNA, etc.).
• Curation: Sequences were deduplicated, filtered for low complexity, and annotated with RNA type and taxonomic lineage (7-level Greengenes hierarchy).
• Sampling Strategy: To address the heavy-tailed distribution of RNA families, the authors implemented an evolutionary conservation-based sampling strategy. They re-weighted sampling probabilities inversely proportional to the square root of cluster size ($P \propto 1/\sqrt{N}$), ensuring rare functional families are adequately represented during training.

B. Model Architecture

• Backbone: A decoder-only Transformer with 1.4 billion parameters.
• Mixture-of-Experts (MoE): Utilizes a sparse MoE architecture (8 experts per layer, top-2 routing) to improve parameter efficiency and performance, a first for RNA foundation models.
• Context Window: An extended 8,192-token context window, allowing the model to process full-length transcripts and capture long-range structural dependencies.
• Training Objectives: A hybrid approach combining:
  • Causal Language Modeling (CLM): For autoregressive sequence generation.
  • Generalized Language Modeling (GLM): For masked infilling (fill-in-the-middle), enabling targeted region redesign.
• Two-Stage Curriculum:
  1. Pre-training: Conditioned only on RNA-type tags to learn universal RNA grammar and structural motifs without being overwhelmed by species-specific noise.
  2. Mid-training: Introduces taxonomic lineage tags to refine the latent space for species-specific adaptability, preventing overfitting while enabling controllable generation.

C. Conditioning Mechanism

The model accepts structured conditioning tokens at the input:

• RNA Type: (e.g., <rna_tRNA>, <rna_mRNA>) to steer generation toward specific structural classes.
• Taxonomic Lineage: (e.g., d__eukaryota;p__chordata;s__homo_sapiens) to enforce organism-specific constraints (e.g., codon usage bias).

3. Key Contributions

1. First Long-Context RNA Foundation Model: EVA is the first model to successfully scale to 1.4B parameters with an 8,192-token context, enabling the modeling of full-length transcripts.
2. Unified Design Framework: It unifies diverse tasks (mutation prediction, de novo design, and targeted redesign) within a single architecture without task-specific fine-tuning for the base model.
3. OpenRNA v1 Dataset: The release of a massive, high-quality, modality-specific RNA dataset (114M sequences) and the associated OpenRNA v1 training corpus.
4. Interpretability: Demonstrates that the model learns biologically meaningful representations (e.g., separating 5'UTR, CDS, and 3'UTR features) using Sparse Autoencoders (SAEs).
5. Cross-Modal Capability: Shows that an RNA-only trained model can predict fitness for DNA gene regions and protein sequences, suggesting a shared evolutionary manifold.

4. Key Results

A. Performance Benchmarks

• Fitness Prediction: EVA achieves state-of-the-art zero-shot performance in predicting mutational effects.
  • ncRNA: Mean Spearman correlation of 0.40 across ribozymes, aptamers, and tRNAs.
  • mRNA: Mean Spearman correlation of 0.31, outperforming dedicated codon optimization models.
  • Gene Essentiality: Outperforms the 7B-parameter DNA model (Evo 7B) on eukaryotic benchmarks and the 1B DNA model (Evo2) on prokaryotic benchmarks, despite being trained solely on RNA.
  • Protein: Achieves a correlation of 0.46 on protein DMS benchmarks, matching specialized protein language models.

B. Generative Capabilities

• Structural Fidelity: In de novo tRNA design, generated sequences maintain a mean TM-score of 0.74 (high structural similarity to natural tRNAs) while having low sequence identity (~0.45) to training data, proving the model learns fold constraints rather than memorizing sequences.
• Controllability: Conditioning on RNA type and species reduces KL divergence by >13-fold (vs. GenerRNA) for sequence/structure distributions.
• Long-Range Recall: "Needle-in-a-haystack" tests confirm EVA can reliably detect perturbations at arbitrary positions within the full 8,192-token context.

C. Applications

• RNA Aptamers & CRISPR: Fine-tuned on small datasets (n=30 for aptamers, n=40+ for omegaRNAs), EVA successfully designed novel sequences with improved fluorescence and editing efficiency.
• Vaccine Design:
  • mRNA: Optimized codon usage for SARS-CoV-2, VZV, RABV, and HIV vaccines, improving both Minimum Free Energy (MFE) and Codon Adaptation Index (CAI) compared to baselines.
  • circRNA: Successfully redesigned IRES elements for circular RNA vaccines, reducing inhibitory long-range interactions by up to 58% and increasing ribosome accessibility by 71%.

D. Interpretability

• Sparse Autoencoders (SAEs): Analysis revealed that individual neurons in EVA correspond to specific biological features (e.g., IRES, ORF, 5'UTR, 3'UTR).
• Hierarchical Organization: Features cluster by biological domain (eukaryotic vs. prokaryotic) and RNA type, indicating the model has learned the hierarchical structure of RNA biology from sequence alone.

5. Significance

EVA represents a paradigm shift in RNA engineering. By combining long-context modeling, MoE scalability, and structured conditioning, it moves beyond simple sequence generation to controllable, evolution-consistent design.

• Scientific Impact: It provides evidence that RNA sequences encode a unified "grammar" that governs structure, function, and evolutionary fitness across species and modalities (RNA/DNA/Protein).
• Practical Impact: The model offers a powerful tool for designing next-generation therapeutics (mRNA/circRNA vaccines, CRISPR guides, aptamers) with reduced immunogenicity and improved stability.
• Open Science: The full release of the model, dataset (OpenRNA v1), and training code democratizes access to large-scale RNA modeling, accelerating the field of generative biology.

The work concludes that scaling generative models on evolutionary RNA diversity yields a unified framework capable of both high-accuracy prediction and controllable design, paving the way for the systematic engineering of complex RNA systems.