pre-miRBench: a multispecies benchmark and reference model for animal precursor microRNA prediction

Technical Summary of pre-miRBench: A Multispecies Benchmark and Reference Model for Animal Precursor microRNA Prediction

Problem Statement

The prediction of precursor microRNAs (pre-miRNAs) is critical for prioritizing genomic hairpins for annotation, particularly when expression data is unavailable or when genome assemblies are the sole resource. However, direct comparison of existing pre-miRNA predictors is hindered by methodological inconsistencies across studies. Previous works utilize different annotation sources (e.g., varying versions of miRBase), disparate negative set constructions (ranging from random genomic fragments to hand-selected pseudo-hairpins), and inconsistent train-test partitions. Crucially, many benchmarks fail to simultaneously evaluate generalization across both species and miRNA families. Furthermore, random record-level splits often inflate performance estimates by allowing homologous sequences to appear in both training and test sets, while imbalanced class ratios and non-representative negative sets can lead to optimistic or non-reproducible metrics.

Methodology

The authors introduce pre-miRBench, a standardized benchmark designed to address these inconsistencies through a rigorous, reproducible framework.

1. Dataset Construction:

• Species and Annotations: The benchmark covers 71 animal species derived from curated MirGeneDB 3.0 annotations. Positive examples consist of 7,056 200-nt genomic windows centered on annotated pre-miRNA loci.
• Negative Set Generation: Negative candidates were sampled from the same genomes. To ensure difficulty, the authors employed "hard-negative mining." Candidates were filtered to retain only those with predicted hairpin structures (MFE ≤ −10 kcal/mol, specific stem-loop constraints) that were not annotated as pre-miRNAs. A four-round random-forest ensemble was trained per species to identify "hard" negatives—genomic windows that mimic precursor hairpins but are likely non-functional.
• Class Balance: A fixed 1:10 positive-to-negative ratio was maintained across all partitions to ensure consistent class prevalence.
• Duplicate Removal: Exact duplicates were removed at the model-input level (normalized to 100 nt) after biological split assignment to prevent data leakage.

2. Experimental Design (Split Strategy):
The benchmark employs a cross-factor split design to evaluate generalization across two axes: species representation and miRNA family representation. This results in four distinct test sets:

• Test 1: Known species / Known family (standard performance).
• Test 2: Known species / Held-out family (generalization to unseen miRNA families).
• Test 3: Held-out species / Known family (generalization to unseen species).
• Test 4: Held-out species / Held-out family (joint generalization).
• Held-out Species: Gallus gallus (chicken) and Drosophila melanogaster (fruit fly) were excluded from training.

3. Model Evaluation:

• Retraining: Six published neural predictors (DeepMir, deepMiRGene, dnnPreMiR, mirDNN, miRe2e, and MuStARD) were retrained on the pre-miRBench partitions using their native architectures and preprocessing pipelines to ensure a fair comparison.
• Reference Model: A new model, the pre-miRBench model, was developed using Agentomics, an autonomous LLM-agent system. This ensemble consists of three neural networks: one PairMessageCNN and two SpeciesGraphGRU networks. The architecture incorporates sequence, secondary structure (RNAfold), minimum free energy (MFE), and species embeddings.
• Metrics: Performance was primarily evaluated using Average Precision (AP) to address class imbalance, supplemented by AUROC, accuracy, precision, recall, F1 score, and Matthews Correlation Coefficient (MCC) at a fixed threshold of 0.5.

Key Contributions

1. Standardized Benchmark: pre-miRBench provides the first unified dataset and split protocol that simultaneously tests generalization across species and miRNA families, utilizing curated MirGeneDB annotations and hard-negative mining.
2. Reproducible Baselines: The authors provide retrained weights and evaluation code for six state-of-the-art predictors under identical conditions, eliminating variability caused by different training data or splits.
3. Agentic Model Development: The paper demonstrates the utility of Agentomics in generating a competitive reference model without manual intervention, resulting in an ensemble that serves as a new baseline for the field.
4. Open Data: All source code, datasets, trained weights, and record-level predictions are publicly available via GitHub and Zenodo.

Results

• Overall Performance: The pre-miRBench model achieved the highest mean Average Precision (AP) across all four test sets (0.9809), outperforming the next best retrained model, dnnPreMiR (0.9719).
• Test Set Specifics:
  • The pre-miRBench model led in Test 1 (Known/Known), Test 2 (Known/Held-out Family), and Test 3 (Held-out Species/Known Family).
  • On the most challenging Test 4 (Held-out Species/Held-out Family), dnnPreMiR achieved the highest point estimate (AP 0.9677), narrowly surpassing the pre-miRBench model (AP 0.9659).
  • The difference between the two top models on Test 4 was statistically non-significant, with a paired bootstrap 95% confidence interval of −0.0445 to 0.0341.
• Binary Classification: At a fixed threshold of 0.5, the pre-miRBench model demonstrated superior balance between precision and recall, achieving the highest accuracy (0.988), recall (0.913), F1 score (0.933), and MCC (0.927) on the joint holdout test. In contrast, dnnPreMiR showed higher precision (0.980) but significantly lower recall (0.710).
• Species Heterogeneity: Performance varied between the two held-out species; six of the seven predictors performed worse on D. melanogaster than on G. gallus, highlighting species-specific challenges.

Significance and Claims

The paper claims that pre-miRBench establishes a common evaluation framework that resolves the difficulty of comparing pre-miRNA predictors due to disparate dataset constructions. By fixing the data partitions and class ratios, the study reveals that model rankings are highly dependent on the specific generalization task (e.g., unseen species vs. unseen families).

The authors assert that the pre-miRBench model provides a reproducible reference model for future comparisons. While the autonomous development via Agentomics produced a top-tier performer, the paper modestly notes that the joint holdout results (Test 4) show that the leading models are statistically comparable. The benchmark is designed not to declare a single "winner" but to provide a rigorous, transparent environment where future predictors can be evaluated against a fixed standard, specifically addressing the barriers of negative-set construction, class balance, and cross-species evaluation identified in previous literature. The work emphasizes that the true value lies in the standardized protocol and the availability of the retrained models and data artifacts for the community.