GFMBench-API: A Standardized Interface for Benchmarking Genomic Foundation Models

The paper introduces GFMBench-API, a modular Python interface that standardizes the evaluation of Genomic Foundation Models by decoupling model-specific processing from task-specific data and metrics to enable reproducible and consistent benchmarking.

The Gist

Imagine the world of Genomic Foundation Models (GFMs) as a bustling, high-tech kitchen where chefs (scientists) are trying to create the perfect dish (a model that understands DNA).

Right now, the kitchen is a bit chaotic. Every chef has their own unique set of measuring cups, their own way of chopping vegetables, and their own recipe for tasting the food. If Chef A wants to see if their soup is better than Chef B's, they can't just taste them side-by-side. Chef A has to translate Chef B's soup into their own measuring system, which often leads to mistakes, confusion, and arguments about who actually made the better dish.

GFMBench-API is the solution to this chaos. Think of it as a universal "Tasting Station" and "Standardized Recipe Card" system for the entire kitchen.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Glue Code" Mess

Before this tool, if a scientist wanted to test a new AI model, they had to write a lot of custom "glue code."

• The Analogy: Imagine trying to plug a European hair dryer into an American wall socket. You need a messy, custom-made adapter just to get it to work. If you want to test a different hair dryer, you have to build a new adapter.
• The Reality: Scientists were wasting huge amounts of time building these adapters (code) just to make different AI models talk to the same test data. This meant no one could fairly compare models because everyone was testing them in slightly different ways.

2. The Solution: The "Universal Adapter" (GFMBench-API)

The authors built GFMBench-API, which acts like a universal power strip for genomic AI.

• How it works: It sits in the middle. On one side, it connects to the AI model (the hair dryer). On the other side, it connects to the test tasks (the wall socket).
• The Magic: The model doesn't need to know what the test is. The test doesn't need to know what the model is. They just plug into the API. The API handles all the messy translation (like converting DNA sequences into numbers the model understands) automatically.

3. The "Menu" of Tasks

The API comes with a standardized menu of challenges, just like a restaurant has a set menu.

• Supervised Tasks (The "Cooking Class"): These are tasks where the model is given a textbook and a quiz. For example, "Here is a DNA sequence; tell me if it's a promoter (a switch that turns genes on)." The model learns, takes the test, and gets a grade.
• Zero-Shot Tasks (The "Blind Taste Test"): These are harder. The model hasn't been trained on this specific quiz. It has to look at a new DNA sequence and guess, "Is this harmful?" based purely on what it already knows about DNA.
• Variant Tasks: Imagine a sentence where you change one letter. "The cat sat" becomes "The bat sat." The API checks if the AI understands that this tiny change changes the meaning.

4. The "Scorecard" (Metrics)

In the old days, one scientist might grade a model out of 100, while another used a scale of 1 to 10, and a third used a "thumbs up/down" system. You couldn't compare the scores.

• The Fix: GFMBench-API uses a single, strict grading rubric. Whether you are testing a small model or a giant one, the API calculates the score using the exact same math. This ensures that if Model A gets a 90 and Model B gets a 85, we know for a fact that Model A is better, not just that they were graded differently.

5. Why This Matters

• Fairness: It stops scientists from "cooking the books" by tweaking their tests to make their model look good.
• Speed: Instead of spending months building a testing pipeline, a scientist can now plug their model in and get results in hours.
• Progress: Because everyone is using the same ruler, we can actually see how fast the field is improving. We can finally say, "Yes, our new model is truly smarter than the old one."

In a Nutshell

GFMBench-API is the standardized testing ground that finally allows the scientific community to stop arguing about how to test AI models and start focusing on building better ones. It turns a chaotic kitchen into a well-oiled machine where the best chefs (models) can finally be recognized for their true talent.

Technical Summary

1. Problem Statement

The rapid emergence of Genomic Foundation Models (GFMs) has transformed the ability to model biological sequences, yet the field suffers from a lack of cohesive infrastructure for systematic evaluation. Current benchmarking practices are characterized by:

• Fragmentation: Researchers rely on custom-made, non-interoperable pipelines to bridge specific model architectures with diverse biological tasks.
• Inconsistency: Benchmarks (e.g., GUE, BEND, Nucleotide Transformer) use non-standardized task formulations, preprocessing steps, and metric implementations.
• Technical Debt: Significant engineering effort is wasted on "glue code" to patch incompatible data formats and reimplement methods (e.g., embedding distances) rather than focusing on core analysis.
• Bias: Variations in how models handle conditional dependencies (e.g., autoregressive vs. masked language modeling) lead to biased results depending on the specific task formulation.

These issues prevent meaningful cross-study comparisons and hinder the reproducible assessment of model progress.

2. Methodology: GFMBench-API

The authors introduce GFMBench-API, a high-level, modular Python library designed to serve as a universal middleware for the genomic machine learning ecosystem. Its core design principles include:

• Decoupling Architecture: The API separates model development (tokenizer, inference logic, training strategies) from task execution (data streams, metrics). This allows any GFM to be evaluated across diverse benchmarks without custom integration code.
• Unified Interface: It provides a standardized API for connecting model backbones with genomic tasks. Users initialize a task, and all interactions proceed through a shared interface.
• Hierarchical Task Abstraction: The system uses a class hierarchy to group tasks by paradigm:
  • Supervised Tasks: Includes multi-class classification (e.g., promoter prediction) and variant effect prediction (using paired reference/alternate sequences).
  • Zero-Shot Tasks: Evaluates pre-trained models without fine-tuning, utilizing metrics like log-likelihood ratios (LLR) and embedding similarity (Cosine, L2 distance).
• Inference-Driven Metric Compatibility: The API defines standardized inference methods (e.g., sequence-to-sequence, masked token prediction). Tasks automatically detect which methods a model implements and compute only the compatible metrics, ensuring correctness without requiring model-specific adaptations.
• Conditional Input Support: The framework supports auxiliary metadata (e.g., tissue type) as conditional inputs, allowing for context-aware evaluation while maintaining model agnosticism.

3. Key Contributions

1. Standardized Middleware: A universal API that decouples model logic from task execution, enabling "plug-and-play" evaluation of any GFM across diverse benchmarks.
2. Unified Protocol: Establishes a consistent protocol for handling genomic inputs/outputs, including reference/alternate allele handling and conditional logic, ensuring disparate models are tested against identical data streams.
3. Mathematical Consistency: Centralizes metric implementation to prevent "metric drift," ensuring that results are reproducible and mathematically consistent across different studies.
4. Comprehensive Benchmark Suite: Implements a diverse set of tasks derived from established resources (GUE, VariantBenchmarks, BEND, ClinVar, LRB), covering:
   • Supervised single-sequence tasks (Promoter, Splice site, TF binding).
   • Supervised variant effect tasks (Coding/Non-coding pathogenicity, QTLs).
   • Zero-shot variant effect tasks (SNVs, Indels, Disease/Expression association).
   • Novel open benchmarks (e.g., ClinVar-based Indel pathogenicity).

4. Results (Case Study)

The authors validated the framework by evaluating five prominent GFMs: DNA-BERT, DNABERT-2, NTv3, Caduceus-Ps, and Evo 2.

• Workflow: The evaluation was conducted via a "one-click" workflow, decoupling model implementation from task logic.
• Training Strategies:
  • Full Fine-tuning: Applied to DNA-BERT, DNABERT-2, NTv3, and Caduceus-Ps (3 epochs).
  • Linear Probing: Applied to Evo 2 (frozen weights, trained linear head).
• Performance Highlights:
  • Supervised Tasks: DNABERT-2 and NTv3 generally outperformed others in promoter and splice site prediction (e.g., DNABERT-2 achieved ~0.92 Accuracy on Splice Site). However, performance varied significantly on variant effect tasks, with NTv3 showing strong results on Non-Coding Pathogenicity (0.98 Accuracy) while others struggled.
  • Zero-Shot Embedding: Evo 2 demonstrated superior performance in embedding-based metrics (e.g., 0.87 AUROC on BEND Disease variants), significantly outperforming smaller models.
  • Zero-Shot Likelihood: Evo 2 (autoregressive) achieved high LLR scores (0.92 AUROC on SongLab ClinVar), while masked models (like NTv3) showed varying performance depending on the task.
  • Indel Handling: The framework successfully evaluated models on non-aligned Indel tasks (ClinVar Indel), where Evo 2 achieved 0.94 AUROC, demonstrating the API's capability to handle length-altering variants.

5. Significance

• Lowering Barriers: GFMBench-API significantly reduces the engineering overhead required to evaluate new genomic models, allowing researchers to focus on model architecture and biology rather than data pipeline integration.
• Reproducibility: By standardizing data splits, preprocessing, and metric calculation, the framework ensures that performance claims are validated through a rigorous, consistent framework.
• Future-Proofing: The modular design allows for the easy addition of new tasks and metrics. While currently focused on haploid DNA, the architecture is designed to extend to diploid and multi-context analyses.
• Community Standard: It aims to become the de facto standard for GFM evaluation, fostering a more systematic understanding of progress in genomic AI and enabling fair comparisons across the rapidly evolving landscape of foundation models.

The code is implemented in Python and will be publicly released upon acceptance, supporting academic use.