An Improved Dataset for Predicting Mammal Infecting Viruses from Genetic Sequence Information

This paper introduces a standardized, nearly doubled dataset of mammal-infecting viral pathogens with refined host labels to demonstrate that machine learning models achieve significantly better predictive performance for broader taxonomic ranks and when training and test sets share closer phylogenetic relationships, while highlighting the current limitations of generalizing these models to completely novel viral families.

The Gist

Imagine you are a detective trying to predict which viruses might jump from animals to humans and cause a pandemic. You have a massive library of genetic "fingerprint" cards for thousands of viruses, but you need a way to sort the dangerous ones from the harmless ones quickly.

This paper is about building a better, more reliable library and a better set of rules for that sorting process.

Here is the breakdown of what the researchers did, using some everyday analogies:

1. The Problem: The "Messy Library"

Before this study, scientists tried to build computer programs (Machine Learning models) to predict viral threats. But it was like trying to compare two chefs' cooking skills when:

• Chef A used a recipe book from 1990 with missing pages.
• Chef B used a book from 2020 with different ingredients.
• They were judged by different judges using different scoring systems.

Because everyone used different data and different ways to test their models, no one could agree on which computer program was actually the best. It was a "apples-to-oranges" comparison.

2. The Solution: A "Refined and Reorganized" Library

The authors (Tyler Reddy, Austin Schneider, and their team at Los Alamos National Laboratory) decided to clean up the library. They took the previous best dataset and:

• Double-checked the facts: They read the latest scientific news to update which viruses actually infect humans. They found some viruses were mislabeled and fixed them.
• Added new categories: Instead of just asking "Does this infect humans?", they added two new questions: "Does this infect primates (like monkeys)?" and "Does this infect mammals (like dogs, cows, or humans)?"
  • Analogy: Think of it like a security checkpoint. It's easier to spot a threat if you first ask, "Is this person a mammal?" (broad category) before asking, "Is this person a human?" (specific category).
• Doubled the data: They nearly doubled the number of verified records, giving the computer programs more examples to learn from.

3. The Big Discovery: "Mixing the Deck"

The most surprising finding was about how they shuffled the data.

Imagine you are teaching a student to recognize different types of cars.

• The Old Way: You showed the student 100 pictures of Ford Mustangs to study, and then tested them on 50 pictures of Ford Mustangs they hadn't seen before. They would ace the test, but only because they memorized the Mustangs, not because they learned what makes a "car."
• The New Way: The researchers mixed the deck. They made sure the "study" pile and the "test" pile had a similar mix of all car brands (Ford, Toyota, BMW, etc.).

The Result: When they mixed the data so the training and testing sets were more similar (a concept called reducing "phylogenetic distance"), the computer models got much better at predicting human infections.

• Old Score: 66% accuracy (like a C- student).
• New Score: 78% accuracy (a solid B+ student).

4. The Hierarchy of Success

The study found a clear pattern in how well the models worked based on how specific the question was:

1. Mammals (Broad): The models were best at this (85% accuracy). It's like predicting "Will this animal have fur?"—it's easier to spot the general pattern.
2. Humans (Specific): The models were good, but not perfect (78% accuracy).
3. Primates (Middle): Similar to humans (77% accuracy).

The Takeaway: It might be smarter to build a two-step security system. First, use a model to screen for viruses that infect any mammal. Then, take those suspicious ones and run them through a second, more specific model to see if they infect humans.

5. The "Peptide" Trap

The researchers tried adding a specific type of genetic feature called "peptide kmers" (think of these as tiny, specific word fragments in the virus's DNA language).

• The Surprise: Adding these tiny fragments actually hurt the model's performance when testing on new, unseen viruses.
• Why? It's like teaching a student to recognize a specific brand of car by memorizing the shape of a specific bolt. If the new car has a different bolt, the student gets confused. The models were "overfitting"—memorizing the training data too closely instead of learning the general rules.

6. The Hard Truth: The "Alien" Problem

The paper ends with a sobering thought. When they tested the models on viruses from families they had never seen before (like testing a student on a completely new alien language), the models failed. They performed no better than random guessing (50/50).

Why? Because viruses might not all share a single "common ancestor" like humans or dogs do. They might have evolved from different places entirely. If they don't share a family tree, it's incredibly hard for a computer to predict how a brand-new, alien virus will behave just by looking at its genetic code.

Summary

This paper is a major step forward in creating a standardized, high-quality library for studying viral threats. It proves that:

1. Better data organization (mixing the deck) leads to better predictions.
2. Broader categories (Mammals) are easier to predict than specific ones (Humans).
3. Less is sometimes more: Adding too many tiny details (peptides) can confuse the model.
4. The ultimate challenge: Predicting completely new viruses remains very difficult because viruses might not follow the same evolutionary rules as other life forms.

The authors have shared their cleaned-up library and the code they used, so other scientists can stop arguing about whose data is better and start working together to build better pandemic warning systems.

Technical Summary

1. Problem Statement

The prediction of viral spillover events (zoonosis) is critical for pandemic preparedness. While machine learning (ML) models have been developed to predict human-infecting viruses from genomic sequences, progress is hindered by:

• Lack of Standardization: Previous studies use disparate datasets, data splitting schemes, features, and performance metrics, making direct model comparison impossible.
• Data Quality and Leakage: Existing datasets often suffer from data leakage (e.g., identical viral isolates in both training and test sets) or reliance on incomplete genomes.
• Taxonomic Bias: Models are often trained on specific species (e.g., humans) rather than broader taxonomic groups, potentially limiting generalizability.
• Phylogenetic Distance: Training and test sets often have high phylogenetic distance (different viral families), leading to poor out-of-sample performance that may be mistaken for model failure rather than a fundamental limitation of the data.

2. Methodology

Dataset Curation and Improvement

The authors refined the dataset previously curated by Mollentze et al. (which contained 861 training and 758 holdout records) through the following steps:

• Literature Review: Manually updated host labels based on the latest literature (as of July 2025).
• New Labels: Introduced three target labels: Human, Primate, and Mammal. This allows for hierarchical prediction, hypothesizing that broader taxonomic ranks are more predictable.
• Quality Filtering:
  • Removed duplicate accession numbers present in both train and test sets.
  • Excluded "partial" genomes (incomplete sequences).
  • Removed genomes lacking coding sequences divisible by 3 (required for amino acid feature calculation).
• Rebalancing (Shuffling): The original dataset had uneven viral family representation between train and test sets. The authors randomly shuffled the data while preserving the total counts of human/primate/mammal infecting viruses. This reduced the Kullback-Leibler divergence (relative entropy) of viral family representation between train and test sets from 3.00 to 0.08, ensuring the test set contained viral families similar to those in the training set.

Machine Learning Workflow

• Models Evaluated: Eight ML estimators were tested: Random Forest, Extra Trees, Gradient Boosted Trees (XGBoost, LightGBM with Boost and DART algorithms), and Support Vector Machines (SVM) with linear, polynomial, and RBF kernels.
• Features:
  • Genomic features similar to previous work (nucleotide composition, etc.).
  • Peptide k-mers: Translated peptide sequences were added as features to test their utility.
• Validation: Models were trained on the training set and evaluated on the test set using ROC AUC. Performance was averaged across 8 estimators and 10 random seeds. Hyperparameters were optimized using the Tree-structured Parzen Estimator (TPE) algorithm via Optuna.
• Ablation Studies: The authors tested scenarios where:
  • Peptide k-mers were included vs. excluded.
  • Training and test sets were split such that no viral families overlapped (high phylogenetic distance, relative entropy > 24) to test true generalization.

3. Key Contributions

1. Standardized, Curated Dataset: Released a refined dataset with roughly double the number of curated host-virus records compared to previous benchmarks, including updated human, primate, and mammal infection labels.
2. Demonstration of Data Split Impact: Provided empirical evidence that reducing phylogenetic distance between training and test sets (via rebalancing) significantly improves predictive performance, whereas high phylogenetic distance renders models no better than random chance.
3. Taxonomic Hierarchy Insights: Showed that prediction accuracy increases with broader taxonomic categories (Mammal > Primate > Human).
4. Feature Analysis: Identified that peptide k-mer features can be detrimental to model performance in out-of-sample scenarios, likely due to overfitting on family-specific signals that do not generalize across the viral tree of life.
5. Open Source Reproducibility: Released all code, data, and workflows via GitHub and Zenodo to enable standardized benchmarking.

4. Key Results

• Impact of Rebalancing:

  • On the original (unbalanced) dataset split, the average ROC AUC for human infection prediction was 0.663 ± 0.070.
  • On the rebalanced dataset (reduced phylogenetic distance), the average ROC AUC improved to 0.784 ± 0.013.
  • This improvement is attributed to the reduced phylogenetic distance (relative entropy drop from 3.00 to 0.08), not necessarily a superior algorithm.

• Taxonomic Performance:

  • Mammal Infection: Highest predictability with an ROC AUC of 0.850 ± 0.020.
  • Primate Infection: ROC AUC of 0.774 ± 0.015.
  • Human Infection: ROC AUC of 0.784 ± 0.013.
  • Conclusion: Broader taxonomic labels yield more reliable predictions, suggesting a two-stage screening approach (Mammal filter $\to$ Human filter) may be effective.

• Generalization Limits (The "No Common Ancestor" Problem):

  • When the dataset was split such that no viral families overlapped between training and testing (simulating a truly novel virus), the models performed at random chance (ROC AUC 0.50 ± 0.08 with k-mers; 0.50 ± 0.04 without).
  • This suggests that current ML models rely heavily on shared evolutionary signals within viral families and struggle to generalize to completely novel viral lineages.

• Feature Ablation:

  • On the original split, removing peptide k-mers improved performance (0.668 vs 0.571 ROC AUC), indicating k-mers caused overfitting.
  • On the rebalanced dataset, k-mers had a negligible effect, suggesting their utility is context-dependent on the data distribution.

5. Significance and Future Directions

• Benchmarking Standard: This work establishes a necessary baseline for comparing future ML models for viral host prediction, moving away from heterogeneous, non-comparable studies.
• Strategic Screening: The results support a hierarchical prediction strategy: first screening for mammalian infectivity (high confidence), then refining for primate/human infectivity.
• Fundamental Limitations: The paper highlights a critical biological constraint: because viruses likely do not share a common ancestor, predicting the host range of a completely novel virus (one from a family not seen in training) remains an unsolved challenge for current sequence-based ML.
• Future Work: The authors suggest the need for specialized challenge datasets (e.g., viruses with single nucleotide changes altering host range) and the development of curated, community-maintained datasets similar to those in computer vision (COCO) or general ML (MLPerf). They also note the potential for integrating protein language models (like ESM-1b) but warn of data leakage risks if training data overlaps with viral protein embeddings.