Designing mRNA coding sequence via multimodal reverse translation language modeling with Pro2RNA

This paper introduces Pro2RNA, a multimodal reverse-translation language model that integrates protein, taxonomy, and RNA representations to generate host-adapted, species-specific mRNA coding sequences, outperforming existing optimization methods in benchmark evaluations.

The Gist

The Big Idea: Translating "Protein Blueprints" into "Natural-Sounding Instructions"

Imagine you have a master architect's blueprint for a beautiful house (this is the Protein). You want to build this house, but you need to hire a construction crew. The problem is, you have crews from different countries: one from Japan, one from Brazil, and one from Germany.

If you hand the blueprint to the Japanese crew, they need instructions written in Japanese. If you give them the blueprint written in German, they might understand the idea of the house, but they won't know how to build it efficiently. They might get confused, work too slowly, or even build a wobbly, unsafe structure.

In biology, this is exactly the problem scientists face. They have a protein they want to make (the blueprint), but they need to write the mRNA instructions (the construction manual) in a way that a specific host cell (like a bacteria or a human cell) can read and follow perfectly.

Pro2RNA is a new AI tool that acts as a super-translator. It takes a protein blueprint and writes a brand-new mRNA instruction manual that sounds exactly like it was written by a native speaker of that specific host's "language."

---

The Problem: Why Old Methods Failed

For a long time, scientists tried to optimize these instructions using a simple rule: "Use the most popular words."

• The Old Way: Imagine a construction crew that loves using the word "Hammer." The old method would replace every single tool instruction with "Hammer," even if the job required a "Screwdriver."
• The Result: The crew gets confused. They try to hammer screws. The house gets built, but it's shaky, the workers are exhausted, and the final product is ugly. In biology, this leads to proteins that don't fold correctly or cells that get tired and stop working.

Scientists realized that nature doesn't just use the "most popular" words; it uses a rhythm. Sometimes, it pauses. Sometimes, it uses a "rare" word to slow down the worker so they can focus on a tricky part of the build.

The Solution: Pro2RNA (The "Polyglot" AI)

The researchers built Pro2RNA, which is like a multilingual, culturally aware AI translator. It doesn't just translate words; it understands the culture and style of the host organism.

Here is how it works, using our construction analogy:

1. The Three Brains (The Architecture)

Pro2RNA combines three different "brains" to do its job:

• The Protein Brain (ESM2): This brain looks at the blueprint (the protein) and understands the shape and structure of the house. It knows what needs to be built.
• The Culture Brain (SciBERT): This brain reads the "ID card" of the construction crew (the host organism, e.g., E. coli or Human). It knows the specific dialect, slang, and rhythm that this specific crew prefers. It understands that "E. coli" likes to pause at certain spots, while "Human cells" prefer a different flow.
• The Writer Brain (mRNA-GPT): This is the actual scribe. It takes the understanding from the other two brains and writes the new instruction manual, word by word (codon by codon).

2. The "Border-Species" Training

The AI was trained on a massive library of millions of real instruction manuals from many different species.

• The Analogy: Instead of just teaching the AI how to speak "English," they taught it how to speak "English, French, Spanish, and Mandarin" all at once.
• The Benefit: Because it learned from so many different "languages," it can now guess how to speak to a new species it has never met before. It's like a polyglot who can figure out how to talk to a new tribe by recognizing patterns from tribes they already know.

Why This is a Game-Changer

The paper shows that Pro2RNA beats the old "most popular word" methods in two major ways:

1. It Sounds Natural: The instructions it writes look and feel like they belong to that specific host. If you compare the AI's instructions to the host's natural DNA, they match much better than the old methods.
2. It Avoids "Over-Optimization":
   • The Trap: Old methods tried to make the instructions perfectly efficient, using only the "fastest" words. This is like telling a construction crew to run at full speed 24/7. They burn out, make mistakes, and the building collapses.
   • The Pro2RNA Way: Pro2RNA knows that sometimes you need to slow down. It intentionally includes some "slower" words to create a natural rhythm. This helps the protein fold correctly and prevents the cell from getting overwhelmed.

The Real-World Impact

Think of Pro2RNA as the ultimate custom-fit suit maker.

• Old Method: Buying a suit off the rack and pinning it to fit. It works, but it's uncomfortable and looks weird.
• Pro2RNA: Measuring the person, understanding their style, and sewing a suit from scratch that fits perfectly and feels like a second skin.

Why does this matter?

• Vaccines: It can help design better mRNA vaccines that work faster and last longer in the human body.
• Medicine: It can help produce life-saving drugs (like insulin or antibodies) in bacteria or yeast factories much more efficiently.
• Synthetic Biology: It allows scientists to design entirely new biological systems that work smoothly inside living cells.

In a Nutshell

Pro2RNA is a smart AI that learns the unique "language" and "rhythm" of different living things. Instead of forcing a protein to speak a generic language, it writes a custom instruction manual that the host cell loves to read, resulting in better, safer, and more effective biological products.

Technical Summary

1. Problem Statement

The design of mRNA coding sequences (CDS) is critical for mRNA vaccines, therapeutics, and synthetic biology. While the genetic code is degenerate (multiple codons encode the same amino acid), organisms exhibit distinct codon usage biases driven by evolutionary pressure, tRNA abundance, and translational regulation.

• Limitations of Traditional Methods: Conventional optimization relies on replacing rare codons with "optimal" ones based on frequency tables. This approach often ignores global sequence context, potentially disrupting ribosome pausing, co-translational protein folding, and introducing negative cis-regulatory elements, leading to misfolded proteins or reduced yield.
• Limitations of Existing AI Models: While Large Language Models (LLMs) have advanced protein and RNA modeling, existing tools often lack species-specific adaptability or fail to integrate host taxonomy information effectively during the reverse translation process (generating mRNA from a protein sequence).

2. Methodology: The Pro2RNA Framework

Pro2RNA is a multimodal, reverse-translation language model designed to generate host-adapted mRNA coding sequences from protein sequences and host taxonomy information.

Architecture

The model employs an encoder-decoder architecture integrating three distinct pretrained language models:

1. Taxonomy Encoder (SciBERT): Encodes host organism taxonomy (e.g., domain, phylum, genus, species) as natural language prompts. This captures semantic and hierarchical biological context.
2. Protein Encoder (ESM2): Encodes the target protein sequence using a transformer-based protein language model to capture structural, functional, and evolutionary residue-level representations.
3. Generative RNA Decoder (mRNA-GPT): A GPT-style autoregressive model trained on coding mRNA sequences to generate codons token-by-token. (Note: GenerRNA was tested but found unsuitable as it is trained on non-coding RNA).

Training Strategy

• Multimodal Fusion: The embeddings from SciBERT (taxonomy) and ESM2 (protein) are concatenated and projected via a Multi-Layer Perceptron (MLP) to create a unified conditioning vector.
• Parameter-Efficient Fine-Tuning (PEFT): The backbone parameters of SciBERT, ESM2, and the RNA decoder are frozen. Only LoRA (Low-Rank Adaptation) adapters and the fusion MLP layers are trained. This allows for efficient adaptation to new hosts without full retraining.
• Training Objective: Autoregressive next-token prediction (cross-entropy loss) on codon sequences, conditioned on the fused protein-taxonomy representation.
• Datasets:
  • Pro2RNA-bacteria: Trained on ~1 million mRNA-protein pairs from 261 bacterial species across 6 families.
  • Pro2RNA-eukaryote: Trained on ~0.55 million pairs from 13 representative eukaryotic species (including human, mouse, yeast, plants).

3. Key Contributions

1. Multimodal Reverse Translation: First framework to explicitly integrate host taxonomy (via natural language processing) and protein sequence (via protein language models) to guide mRNA generation.
2. Species-Aware Generalization: Demonstrated that training on "border species" (multiple related organisms) significantly improves generalization to unseen species compared to single-species models.
3. Biologically Plausible Optimization: Unlike traditional tools that aggressively maximize codon adaptation (often leading to high negative cis-regulatory elements), Pro2RNA learns a context-dependent strategy that balances codon usage with regulatory integrity.
4. Modular and Efficient: The LoRA-based architecture allows rapid adaptation to new organisms with minimal computational cost.

4. Key Results

Ablation Studies

• The full architecture (ESM2 + SciBERT + mRNA-GPT) achieved the highest naturalness score (0.5506), significantly outperforming variants without taxonomy encoding (0.5301) or using non-coding RNA decoders (GenerRNA: 0.1264).
• Including SciBERT provided complementary taxonomy-aware context, proving that biological text semantics are crucial for species-specific generation.

Performance on Eukaryotes

• Naturalness: Pro2RNA-eukaryote achieved higher naturalness scores than single-species models (e.g., Pro2RNA-human) on both human and unseen species like Candida albicans.
• Benchmarking: Outperformed CodonTransformer and commercial tools (Twist Bioscience, IDT, Genewiz) across four model organisms (S. cerevisiae, A. thaliana, M. musculus, H. sapiens) in:
  • Codon Similarity Score: Higher identity to native sequences.
  • Codon Similarity Index (CSI): Better alignment with host codon weights.
  • Dynamic Time Warping (DTW): Closer match to native translational dynamics (%MinMax profiles).

Performance on Bacteria

• Pro2RNA-bacteria achieved a naturalness score of 0.5866 on held-out test data.
• It outperformed existing methods in codon similarity and CSI for E. coli, demonstrating successful capture of bacterial coding rules.

Heterologous Expression & Regulatory Elements

• Balanced Optimization: Pro2RNA generated sequences with intermediate CSI values (avoiding extreme over-optimization) and significantly reduced negative cis-regulatory elements compared to commercial pipelines (e.g., Genewiz) and GEMORNA.
• Biological Insight: By avoiding "maximal" codon bias, Pro2RNA implicitly captures the evolutionary trade-off where slight mismatches between codon usage and tRNA supply can prevent mRNA secondary structure issues and aid co-translational folding.
• Comparison with GEMORNA: While GEMORNA produced sequences with near-perfect CSI (~1.0) but artificial signatures, Pro2RNA produced sequences with CSI distributions closer to native host genes, suggesting higher naturalness.

5. Significance

• Paradigm Shift: Moves mRNA design from static, rule-based codon optimization to data-driven, context-aware generative modeling.
• Practical Utility: Provides a flexible framework for designing mRNA for diverse hosts (bacteria to humans) without retraining from scratch, crucial for rapid vaccine and therapeutic development.
• Biological Fidelity: By prioritizing regulatory integrity over raw codon frequency, Pro2RNA generates sequences more likely to result in functional, correctly folded proteins in real biological systems, addressing a major bottleneck in synthetic biology and gene therapy.

The code for Pro2RNA is available at: https://github.com/ZHymLumine/Pro2RNA/.