PhageBench: Can LLMs Understand Raw Bacteriophage Genomes?

1. Problem Statement

While Large Language Models (LLMs) have demonstrated proficiency in understanding biological texts, their ability to directly interpret raw nucleotide sequences (the "language of life") without specific pre-training on genomic data remains largely unexplored.

• The Gap: Specialized Genomic Foundation Models (GFMs) exist but require massive, resource-intensive pre-training on DNA corpora. General-purpose LLMs, which excel at reasoning, have not been rigorously tested on raw genomic data.
• The Challenge: Bacteriophages (viruses infecting bacteria) represent the "dark matter" of the biosphere. Most phage sequences in metagenomic data are unannotated. Accurately annotating them is critical for antibiotic alternatives (phage therapy) and ecosystem regulation, but current methods struggle with sequences that have low homology to known references.
• Core Question: Can general-purpose LLMs, without domain-specific fine-tuning, perform biological reasoning on raw DNA sequences to identify, quality-check, and annotate phage genomes?

2. Methodology: PhageBench

The authors introduce PhageBench, the first benchmark designed to evaluate LLMs on raw phage genome understanding by mirroring the workflow of bioinformatics experts.

Dataset Construction

• Source: High-quality sequences from authoritative databases (PhageScope, NCBI RefSeq).
• Scale: 5,600 high-quality samples.
• Constraints: Sequences range from 0.5 kb to 50 kb (mean ~28 kb) to fit modern LLM context windows.
• Design Principles:
  1. Workflow Alignment: Covers the full annotation pipeline.
  2. Ground Truth: Derived from verified databases.
  3. Balance: Strict class balancing to prevent statistical bias.
  4. Format: Standardized Multiple-Choice Questions (MCQs) with randomized option ordering.

Task Categories (3 Stages, 5 Tasks)

1. Screening:
   • Task 1: Phage Contig Identification: Binary classification to distinguish phage sequences from non-phage DNA (bacteria, plasmids, eukaryotes, etc.).
2. Quality Control:
   • Task 2: Contamination Detection: Identifying host DNA fragments erroneously assembled into phage contigs.
   • Task 3: Completeness Estimation: Classifying genome assembly quality (Complete, High/Medium/Low Quality) based on structural integrity (e.g., terminal repeats).
3. Phenotype Annotation:
   • Task 4: Lifestyle Classification: Distinguishing between Virulent (Lytic) and Temperate (Lysogenic) lifestyles based on the presence of integration machinery (e.g., integrases).
   • Task 5: Host Prediction: Predicting the bacterial host family/genus/order based on sequence signatures (e.g., codon usage bias).

Evaluation Protocol

• Models: Eight advanced LLMs were evaluated, including open-weight (LLaMA-4, Qwen3) and closed-weight (GPT-4o, GPT-5.2, Gemini-3-flash, Claude-sonnet-4.5) models. Both reasoning and non-reasoning variants were tested.
• Setting: Zero-shot Chain-of-Thought (CoT) prompting without in-context examples to test inherent reasoning capabilities.
• Metric: Classification accuracy.

3. Key Contributions

1. PhageBench Benchmark: A comprehensive dataset and evaluation framework for raw genomic reasoning, moving beyond text-based biology tasks.
2. Novel Evaluation Paradigm: The first study to test general-purpose LLMs on authentic whole-genome analysis tasks without relying on sequence homology (BLAST-like) but rather on syntactic and statistical inference.
3. Systematic Analysis: A detailed breakdown of model performance across different genomic tasks, sequence lengths, and reasoning strategies.

4. Key Results

• Overall Performance: General-purpose reasoning models significantly outperform random baselines.

  • Top Performer: Gemini-3-flash achieved the highest average accuracy (56.33%), followed by GPT-5.2 (50.95%).
  • Reasoning vs. Non-Reasoning: Reasoning-enhanced models consistently outperformed non-reasoning counterparts, suggesting that intermediate logical steps aid genomic comprehension.
  • Closed vs. Open: Closed-weight models generally maintained a performance advantage over open-weight models.

• Task-Specific Findings:

  • High Success (Identification & Host Prediction): Models achieved ~70% accuracy in identifying phages and ~62% in host prediction. This indicates LLMs can effectively learn statistical genomic signatures (e.g., GC content, codon bias) and treat DNA as a structured language with local syntax.
  • Moderate Success (Completeness): Models exceeded random baselines (up to 43% for GPT-5.2) but struggled with global structural dependencies required to verify genome closure (e.g., matching 5' and 3' ends).
  • Low Success (Contamination & Lifestyle):
    • Contamination: Performance was near random for low contamination ratios (12.5%), as models failed to distinguish subtle host signals mixed with phage DNA.
    • Lifestyle: Models struggled to locate sparse functional markers (e.g., integrase genes) within long sequences, often failing to differentiate lytic vs. temperate phages.

• Sequence Length & Reasoning:

  • Length Invariance: Phage identification remained stable across sequence lengths (0.5kb–50kb).
  • Long-Range Dependency: Performance on completeness estimation degraded as sequence length increased, highlighting a bottleneck in maintaining global coherence.
  • CoT Impact: Activating Chain-of-Thought generally improved performance, though some models (e.g., Gemini-3-flash on host prediction) performed better with intuitive alignment than explicit reasoning chains.

• Failure Modes:

  • Hallucination: Models often generated biologically plausible reasoning that contradicted the input sequence (e.g., claiming to see terminal repeats that weren't there).
  • Knowledge Overriding Evidence: In lifestyle classification, models sometimes relied on memorized associations (e.g., "PhiX174 is virulent") rather than analyzing the specific input sequence, leading to errors when the input was a different temperate phage.

5. Significance and Future Directions

• Scientific Impact: The study proves that general-purpose LLMs possess latent capabilities for genomic analysis, offering a potential pathway to accelerate the annotation of "phage dark matter" without training specialized GFMs.
• Limitations Identified: Current models lack the ability to ground abstract biological knowledge in precise sequence evidence and struggle with long-range structural dependencies.
• Future Recommendations:
  1. Multi-modal Alignment: Fine-tuning models on paired raw genomes and functional annotations to bridge the gap between sequence and phenotype.
  2. Tool-Augmented Reasoning: Integrating external tools (e.g., for exact sequence matching or structural verification) to allow LLMs to "check" their hypotheses against rigorous computational results.
  3. Optimized Tokenization: Moving away from standard BPE (optimized for natural language) toward k-mer or single-base tokenization to better align with biological logic.

In conclusion, PhageBench establishes that while LLMs are promising tools for high-level genomic screening, significant advancements in reasoning architectures and grounding mechanisms are required before they can reliably replace specialized bioinformatics tools for complex, fine-grained genomic analysis.