Technical Summary: GENERator: A Long-Context Generative Genomic Foundation Model

Problem Statement

Despite the rapid expansion of genomic datasets, the interpretation and rational engineering of sequence function remain significant challenges. While Large Language Models (LLMs) have advanced biological sequence analysis, existing genomic foundation models face three primary limitations:

1. Restricted Training Scope: Many models are trained on narrow domains (e.g., human genomes, bacteriophages, or specific viral sequences), limiting their applicability to the broader eukaryotic landscape characterized by complex gene architectures.
2. Limited Generative Flexibility: Early models predominantly utilized Masked Language Modeling (MLM) objectives, which excel at representation learning but are not naturally optimized for open-ended sequence generation or functional design.
3. Computational Inefficiency: Achieving long-context modeling often requires prohibitive computational costs. Strategies like State Space Models (SSMs) with single-nucleotide tokenization or massive parameter counts (e.g., 40B+ parameters) create barriers to training, inference, and downstream experimentation.

Methodology

Model Architecture and Training

GENERator is a generative genomic foundation model built on an autoregressive Transformer decoder architecture.

• Data Curation: The model is pre-trained on 386 billion nucleotides of eukaryotic DNA curated from the RefSeq database. Crucially, the authors employ a functional sequence training strategy, extracting gene-centric functional regions (including coding regions, introns, UTRs, and proximal regulatory segments) rather than training on indiscriminate genomic sequences. This approach avoids overwhelming the model with non-functional repetitive content.
• Tokenization Strategy: To balance sequence resolution with effective context coverage, GENERator adopts a 6-mer tokenization strategy. This reduces the effective token sequence length, enabling the modeling of substantially longer genomic contexts (up to 98k nucleotides) within realistic computational budgets. The authors systematically compared tokenizers (1-mer to 8-mer, BPE) and found that 6-mer tokenization offers the optimal trade-off between preserving local sequence grammar and enabling long-range context under a fixed token budget.
• Architecture: The model utilizes a decoder-only Transformer (based on the LLaMA framework) with parameter scales of 1B and 3B. It is trained using an autoregressive next-token prediction objective.

Evaluation Framework

The paper evaluates GENERator across three dimensions:

1. Intrinsic/Zero-Shot Capabilities: Unsupervised embedding clustering, sequence recovery (a unified, tokenization-agnostic metric for generative fidelity), and alignment-free variant effect prediction (VEP).
2. Task-Specific Fine-Tuning: Performance on established benchmarks (NT tasks, Genomic Benchmarks) and newly proposed "Gener" tasks (gene and taxonomic classification).
3. Generative Applications:
   • Central Dogma Validation: Generating protein-coding DNA sequences that translate into structurally plausible proteins.
   • Cis-Regulatory Element (CRE) Design: A prompt-guided framework to design enhancers and silencers with targeted activity profiles, validated via high-throughput UMI-STARR-seq assays.

Key Contributions

1. Functional Sequence Training Paradigm: The authors demonstrate that concentrating pre-training on gene-centric functional loci yields superior downstream performance compared to training on all genomic sequences, even though the latter results in lower pre-training loss. This suggests that functional regions contain the primary "semantics" of DNA, while non-functional regions introduce noise that can bias the model toward trivial statistical regularities.
2. Tokenization for Causal Generation: The study establishes that 6-mer tokenization is superior to both single-nucleotide and Byte Pair Encoding (BPE) tokenization for autoregressive genomic language models. While BPE performs well in MLM, its hierarchical vocabulary creates ambiguity in next-token prediction, leading to poor generative performance.
3. Context vs. Resolution Trade-off: The paper argues that increasing nominal context length via SSMs with single-nucleotide tokenization does not yield proportional gains in generative accuracy. Instead, transformer-based models with coarser tokenization (k-mers) more effectively convert extended context into biologically informative representations.
4. Alignment-Free Variant Effect Prediction: GENERator supports VEP in a zero-shot setting without Multiple Sequence Alignments (MSA), achieving competitive performance against MSA-based methods (like GPN-MSA and CADD) while remaining applicable to non-model organisms.

Results

• Representation Learning: Unsupervised embedding analysis reveals that GENERator organizes genomic sequences in latent space according to deep evolutionary (phylogenetic) relationships, with clear clustering across six eukaryotic taxonomic groups.
• Generative Fidelity: In sequence recovery benchmarks, GENERator-1B outperforms Evo2-1B, and GENERator-3B approaches the performance of Evo2-7B. Crucially, GENERator achieves this with substantially improved computational efficiency, exhibiting tens-fold faster generation speeds compared to Evo2.
• Benchmark Performance: Through fine-tuning, GENERator achieves leading performance across NT tasks, Genomic Benchmarks, and taxonomic classification tasks, outperforming models like Enformer, DNABERT-2, and HyenaDNA.
• Central Dogma: Generated DNA sequences successfully translate into proteins with low perplexity (as scored by ProGen2) and high structural similarity (TM-score > 0.8) to natural proteins, despite low sequence identity (< 0.3). This indicates the model captures protein-coding principles rather than memorizing sequences.
• CRE Design: Using a prompt-guided framework (, , ), GENERator designed synthetic enhancers and silencers. Experimental validation showed that the highest-activity synthetic enhancers exceeded the strongest natural genomic sequences by 35% (housekeeping) and demonstrated a two-fold increase (developmental). The model also successfully generated ultra-low-activity silencers.

Significance and Claims

The paper positions GENERator as an efficient and biologically grounded foundation for genomic interpretation and programmable sequence design. Its significance lies in demonstrating that biologically informed design choices (functional data curation, appropriate tokenization, and alignment with biological structure) can yield models that are both scientifically robust and computationally accessible, without relying on extreme parameter scales (e.g., 40B+).

The authors claim that GENERator moves beyond descriptive modeling toward targeted, experimentally grounded sequence design. By enabling the generation of regulatory elements with activity profiles that surpass natural genomic sequences, the model offers a practical tool for synthetic biology and gene therapy. Furthermore, the work highlights that for biological sequence models, scaling alone is not a reliable proxy for utility; instead, aligning model architecture and training objectives with the specific noise and structural characteristics of biological data is paramount.

The authors explicitly note limitations: the current model is trained exclusively on eukaryotic genomes and does not yet incorporate prokaryotic or viral domains. They propose a future "modular" framework where specialized models (e.g., for prokaryotes) could be developed alongside GENERator to address these distinct biological contexts.