OmniGene-4: A Unified Bio-Language MoE Model with Router-Level Interpretability

Technical Summary: OmniGene-4

Problem Statement
Current multi-modal large language models (LLMs) that integrate natural language with biological sequences (DNA, proteins, structural alphabets) face a critical interpretability gap. Specifically, it remains unclear how these models resolve sequence-grounded biological questions that rely on residue-level patterns rather than simple literature recall. The core challenge lies in understanding the internal mechanics of Mixture-of-Experts (MoE) architectures in this domain: do the gating mechanisms (routers) or the expert networks themselves drive the sequence-aware reasoning?

Methodology
The authors introduce OmniGene-4, a unified bio-language foundation model built upon the Gemma-4-26B-A4B architecture. The model features a dense MoE structure with 128 experts per layer and top-8 routing. The methodology focuses on two primary pillars:

1. Router-Level Interpretability: Instead of treating the router as a black box, the authors "hook" every router across eight distinct task families to perform a discrete router-state decomposition. This allows for the analysis of how expert selection evolves during training and inference.
2. Dual-Head and Multi-Modal Extension:
   • Dual-Head Architecture: The model incorporates auxiliary heads for per-residue classification, specifically targeting 3Di (3-state secondary structure) and DSSP (8-state secondary structure) prediction.
   • OmniGene-4-MM: To test robustness under modality scaling, the authors extend the model to include four vision modalities: chemical-structure images, medical/pathology imagery, and charts. This is achieved via a vision tower and a three-stage LoRA (Low-Rank Adaptation) pipeline, trained efficiently in approximately 1.5 GPU-days.

Key Contributions

• First Router-Level Decomposition for Biological MoEs: The study provides the first granular analysis of expert differentiation in a biological MoE, revealing that 96% of cross-task expert differentiation is established during continued pretraining (CPT), while supervised fine-tuning (SFT) accounts for only 4%, primarily reshaping the middle and output layers.
• Mechanism of Sequence-Grounded Decisions: The analysis reveals that within the protein-homology task family, per-pair routing divergence is negligible (<0.04) compared to cross-task divergence (0.23). This leads to the conclusion that the router's role is to select the appropriate modality, while the actual sequence-grounded computation occurs within the selected experts, not at the gating stage.
• Efficient Multi-Modal Scaling: The OmniGene-4-MM variant demonstrates that adding complex visual modalities preserves core sequence capabilities while acquiring chemist-readable structure understanding, all with a compute cost roughly four orders of magnitude lower than recent specialized bio-MoE models.

Results
The model demonstrates state-of-the-art performance across several benchmarks:

• Remote Homology: Achieves 82.60% accuracy, outperforming ESM-2 3B, MMseqs2, and DIAMOND by 28–31 percentage points.
• Standard Homology: Achieves 99.40% accuracy.
• General Biological Knowledge (BixBench): Scores 93.66%.
• Structural Prediction: The dual-head architecture achieves 78.6% on 3Di and 100% on DSSP classification.
• Multi-Modal Performance (OmniGene-4-MM):
  • Preserves homology capabilities (85% standard, 69.5% remote).
  • Achieves 96% accuracy on Vis-CheBI20 functional-group captioning, demonstrating robust chemical structure understanding.

Significance
The paper claims to characterize the fundamental mechanisms by which multi-modal bio-foundation models acquire, route, and preserve sequence-aware capabilities. By demonstrating that sequence-grounded reasoning is an emergent property of expert computation rather than router selection, and by showing that high-performance multi-modal integration can be achieved with minimal compute overhead, this work establishes a foundational framework for the next generation of scientific large language models.