Technical Summary: GENOME-FACTORY

Problem Statement

Despite the transformative potential of Genomic Foundation Models (GFMs) in tasks such as epigenetic prediction, regulatory element discovery, and species classification, their adoption in life sciences remains limited. A fundamental gap exists between domain expertise and technical implementation: engineers often lack biological context, while biologists lack the expertise to manage large-scale models. Furthermore, existing general-purpose language model frameworks (e.g., LLaMA-Factory) fail to address the unique requirements of genomics, including:

• Specialized Data Handling: The need for domain-specific formats (FASTA), integration with repositories like NCBI, and specific preprocessing (e.g., GC content normalization).
• Lack of Standardization: Genomic models are built in heterogeneous environments with diverse dependencies, making unified deployment difficult.
• Task Divergence: Biological tasks (e.g., predicting variant effects or enhancer activity) differ significantly from language objectives (instruction tuning or text generation), requiring custom loss functions and adaptations.
• Evaluation and Interpretability: Standard text-based benchmarks are insufficient, and biological interpretability—a core requirement for scientists—is not a focus of existing fine-tuning frameworks.

Consequently, the field lacks a unified, user-friendly platform to support the full pipeline of tuning, deploying, and interpreting genomic models.

Methodology: The GENOME-FACTORY Framework

The authors introduce GENOME-FACTORY, the first integrated Python library designed to streamline the end-to-end workflow for genomic foundation models. The framework is modular, consisting of six core components:

1. Genome Collector: Automates data acquisition from public repositories (e.g., NCBI) and applies task-specific preprocessing. It supports region-based dataset construction (e.g., histone modification, enhancers, promoters) and enforces rigorous quality control, including sequence-length filtering, GC content normalization, and the removal of ambiguous nucleotides (N) and compositional outliers.
2. Model Loader: Leverages the Hugging Face Transformers ecosystem to support a diverse range of discriminative models (HyenaDNA, DNABERT-2, Caduceus, Nucleotide Transformer) and generative models (EVO, GenomeOcean). It handles the automatic loading of corresponding tokenizers and model architectures.
3. Model Trainer: Manages the fine-tuning process for both classification and regression tasks. It supports three strategies:
   • Full-parameter fine-tuning: Updates all model weights.
   • Low-Rank Adaptation (LoRA): Freezes the base model and trains low-rank matrices in attention or feed-forward layers.
   • Adapter Tuning: Inserts lightweight neural modules (e.g., MLPs) after the frozen base model.
     The trainer incorporates performance optimizations such as mixed precision (FP16/BF16), Flash Attention, gradient accumulation, and distributed data parallel training.
4. Inference Engine: Provides a unified interface for two key tasks: extracting sequence embeddings (final hidden states) for downstream analysis and generating novel DNA sequences from user prompts.
5. Benchmarker: Facilitates rigorous evaluation by integrating two existing open-source benchmarks (Genome Understanding Evaluation (GUE) and Genomic Benchmarks) and offering a plugin system for custom, domain-specific evaluation tasks. It automates metric calculation (e.g., MCC, F1 score, MSE).
6. Biological Interpreter: An open-source tool based on a sparse auto-encoder (SAE). It disentangles model embeddings into sparse latent units and aligns them with external biological readouts (e.g., motif presence, sequence length) via regression. This aims to decode internal representations to provide mechanistic insights.

The framework is accessible via a zero-code Command-Line Interface (CLI) using YAML configuration files and an interactive, code-free Web-based User Interface (WebUI) built with Gradio.

Key Contributions

1. Unified Workflow: GENOME-FACTORY is the first framework to integrate data collection, model tuning, inference, benchmarking, and interpretability into a single, modular Python library.
2. Model Agnosticism: It unifies diverse genomic models under a single interface, enabling seamless comparison and selection of the most suitable model for specific tasks.
3. Biological Interpretability: It introduces an open-source Biological Interpreter using sparse auto-encoders to decode GFM representations, moving beyond black-box predictions to link latent features with interpretable genomic properties.
4. Accessibility: By providing CLI and WebUI interfaces, the library lowers the technical barrier for non-expert users while maintaining flexibility for advanced developers.

Empirical Results

The authors validate GENOME-FACTORY across three dimensions:

1. Fine-tuning Compatibility: Experiments on six models (HyenaDNA, DNABERT-2, Caduceus, Nucleotide Transformer, EVO, GenomeOcean) using three fine-tuning strategies (Full, LoRA, Adapter) on the GUE benchmark show that Adapter tuning offers the highest efficiency (lowest memory usage and highest throughput), while LoRA provides a strong balance of performance and efficiency. For example, on DNABERT-2, the Adapter method used only 0.2M trainable parameters and 2.09GB memory compared to 117M parameters and 7.30GB for full fine-tuning.
2. Benchmarking Performance: The framework successfully benchmarked models across GUE and Genomic Benchmarks. Results indicate that GenomeOcean-100M achieved the strongest performance under full fine-tuning on GUE, while DNABERT-2 performed best on Genomic Benchmarks. Parameter-efficient methods (LoRA) often matched full fine-tuning performance (e.g., GenomeOcean-100M on Genomic Benchmarks: 68.26 vs. 66.28). The study also compared these against traditional baselines (SVM, Random Forest, CNN, PWM, HMM), demonstrating the competitive edge of foundation models.
3. Biological Interpretation: Using the Biological Interpreter on DNABERT-2, the authors demonstrated that specific SAE latent dimensions correlate strongly with sequence length. Furthermore, a logistic regression probe on SAE activations achieved 0.7881 accuracy in distinguishing CTCF-bound sequences from background, and zeroing specific latent dimensions significantly reduced classification accuracy, confirming that the model captures biologically meaningful regulatory features.

Significance and Claims

The paper claims that GENOME-FACTORY addresses a critical infrastructure gap in genomics by providing a unified, practical platform that bridges the divide between computational engineers and biologists. By standardizing the workflow from data acquisition to biological interpretation, the library accelerates the adoption of GFMs in real-world applications such as personalized medicine, evolutionary biology, and therapeutic design.

The authors position the tool as an enabler for "democratized access" to advanced genomic modeling. They explicitly note that while the Biological Interpreter is a significant step toward mechanistic interpretability, it should be viewed as an initial tool for hypothesis generation rather than a fully validated discovery engine. Additionally, the paper acknowledges potential dual-use biosecurity risks (e.g., generating pathogenic sequences) and suggests that future versions should incorporate software-level guardrails, such as sequence screening and risk-aware generation filters.

Ultimately, the work demonstrates that a unified framework can effectively support the tuning, deployment, and interpretation of diverse genomic foundation models, making these powerful tools more accessible to the broader research community.